According to Jim

Daddy Dearest (5x19)

:

Jim believes that Bill (played by Joseph Bologna), the father of one of Rubys and Gracies classmates is actually his real father that abandoned Jim when he was a child.