Do you have any more information on your Williams family?
According to an Alabama death certificate, Rufus Edward Williams was born in Randolph County, Georgia in 1852. His father was Benjamin Williams born in Morgan (not sure if it was Morgan County, Georgia or Morgan, Calhoun County, Georgia) Benjamin's wife was Maria (last name unknown) also from Morgan.
Rufus Edward Williams sons were
Benjamin Joseph Anon William Rufus Robert Gillad Henry Chester
It was said that Rufus Edward William was orphaned before his 18th birthday which would mean that both parents would have died prior to 1870.
I'm curious to see if there is a connection to the Williams family you described since the use of the first names is similar and the connection to Morgan. Also, regarding the middle name of John, could it have been Gillad rather than Gilliam? Do you know where any of your Williams family members are in the 1850 -1870 census?