According to Jim

Daddy Dearest (5x19)

: 07, 2006

Jim believes that Bill (played by Joseph Bologna), the father of one of Rubys and Gracies classmates is actually his real father that abandoned Jim when he was a child.