Gifford's father, Rudy Winston, was considered a "good man to know" in the wild and woolly Chicago of the 1930s, '40s, and '50s. His list of friends and acquaintances included everyone from the mayor of Chicago and actress Dorothy Lamour to characters with monikers like Willie "the Hero" Nero and Arnold "Suitcase Solly" Banks. To his son, however, he was practically a stranger. Rudy Winston died when Barry Gifford was 12 years old, and Phantom Father is Gifford's reconstruction of a time and a man he never really knew.

Rudy Winston was born in Vienna and moved to the United States when he was 8 years old. As an adult, he ran a combination liquor store and drugstore that doubled as a drop-off point for stolen goods, drugs, and other contraband. Although his family was based in Chicago, he had places in Miami, New York, and Acapulco, and it was widely rumored that he performed the occasional hit for his gangster friends. Even before his death in 1958, Rudy had been something of a phantom in his son's life. According to Gifford's grandmother, his father didn't even speak to him until he was 5 years old; then Rudy walked out on the family when Barry was 8. Much of the material here that deals with Rudy Winston first appeared in Gifford's earlier book, A Good Man to Know. What sets The Phantom Father apart are the unsentimental portraits of family members, family friends, and the various odd characters who passed through Barry Gifford's childhood.