I wonder if your aunt got up into NC on that trip.The GNIS website shows a Wadfords Store (historical) in Nash Co, NC.
There is a Waford Spring in Mono Co, CA; and a Waford Mountain in Cherokee Co, OK.
If Annie? was a widow in 1870, it would seem likely her husband may have died in the Civil War.Have you checked records?
Civil War Soldiers and Sailors website does not show anyone, either side, named Whadford, but there are 6 named Wadford from AL.And, one is named William H. Wadford.Maybe your William Henry was named for his Daddy??
William H. Wadford, Confederate, 33 Reg, AL Inf.
Here is the site, you can look at the others, too.
It looks like your are a thorough researcher, so you might want to check out the 6 Wadfords listed and see where their units enlisted the men.You might be able to weed out a few of them by location alone.
I think you're on the right track about Annie? being related to either James or Mollie Hubbard.Annie? is shown "at home" that would indicate "kin" to me.