Eugene was born in Lincoln County, Mississippi to parents Thornton Edgar "T.E." Robinson and Narsis Bailey Robinson. He married Betty Jean Vaughn in June 1956. Their children are Edna, Ernest Eugene, James, Joyce, Brenda and Barbara Jean. Shirley Vaughn is Eugene's stepdaughter, and he is also the father of daughters Burnadette Garner and Julia Ann Porter.