According to Jim

Daddy Dearest (5x19)

Air date: Mar 07, 2006

Jim believes that Bill (played by Joseph Bologna), the father of one of Rubys and Gracies classmates is actually his real father that abandoned Jim when he was a child.