Wise began playing fiddle at age 15, working locally in the Jacksonville area. He joined the Jubilee Hillbillies in 1938, then began playing with Bill Monroe's Blue Grass Boys in 1942, including dates at the Grand Ole Opry. He worked with Monroe through 1948, then played with Clyde Moody in 1948-49. He also played with the York Brothers, Flatt & Scruggs, and Connie B. Gay.
In 1954, Wise became a member of Hank Snow's Rainbow Ranch Boys, again appearing at the Grand Ole Opry; he remained with the group until 1970. Alongside this he worked as a session musician with Mac Wiseman and Red Allen, among others. Wise returned to Florida in 1984 and went into semi-retirement, though he continued to tour and record occasionally, such as with the Bass Mountain Boys in 1992.
He joined producers Randall Franks and Alan Autry for the In the Heat of the Night cast's CD Christmas Time's a Comin' performing "Christmas Time's a Comin'" with the cast on the CD released on Sonlite and MGM/UA for one of the most popular Christmas releases of 1991 and 1992 with Southern retailers. Franks occasionally joined Wise performing twin fiddle with him on his shows.
- Chubby Wise and the Rainbow Ranch Boys (Starday Records, 1961)
- The Tennessee Fiddler (Starday, 1961)
- Chubby Plays Bluegrass (Stoneway Records, 1970)
- Chubby Plays Bob Wills (Stoneway, 1970)
- Grassy Fiddle (Stoneway, 1975)
- Chubby Plays Hank Williams (Stoneway, 1977)
- Chubby Wise in Nashville (Pinecastle Records, 1994)
- An American Original (Pinecastle, 1995)