His brother Matt and sister Kitty performed an act with him by the time he was ten years old, but the family quit the stage about 1930. Another brother, Ed, became a stage manager and agent in New York.
McHugh debuted on Broadway in The Fall Guy in 1925. Warner Brothers hired him as a contract player in 1930. McHugh played everything from lead actor to sidekick and would often provide comedy relief. He appeared in more than 150 films and television productions and worked with almost every star at Warner Bros. He was a close friend of James Cagney and appeared in more Cagney movies than any other actor. He appeared with Cagney in eleven films between 1932 and 1953. Their friendship lasted until McHugh's death.
From 1954-1956, he starred in Hotel For Pets. By the 1950s, his film career had begun to decline, as evinced by his smaller role in the 1959 film, Career.
Cast as Father Timothy O'Dowd in the 1944 Bing Crosby film, Going My Way, McHugh later played William Jennings Depew in the 1962 episode "Keep an Eye on Santa Claus" in the ABC television series, Going My Way, starring Gene Kelly, and loosely based on the earlier film.
From 1964 to 1965, McHugh played the role of Willie Walters, a live-in handyman in the 27-episode ABC sitcom, The Bing Crosby Show, reunited once again on screen with Bing Crosby and co-starring Beverly Garland. McHugh's television appearance was as Charlie Wingate in the episode, "The Fix-It Man", on CBS's Lancer western series, with Andrew Duggan. McHugh played a handyman in that role too.
McHugh was married to Dorothy Spencer. He had three children and two grandchildren.