I believe I might have a connection to your James Phipps of Wythe Co, VA. Either his son or a Jasper Phipps's son, Zarr Phipps married Didaima "Daimy" Baker, daughter of Rev. Andrew & Elizabeth Avant Baker of Jonesville, VA. The couple moved from the Clay Co, KY area to Knox (presently Bell) Co, KY where Zarr disappeared from view fairly early on. I assume he died. They had two sons, Andrew "Ham" Hamilton Phipps & Henry Franklin Phipps.
If any of these names sound familiar to you, I would appreciate any information. I'm planning to go back to Wythe Co in the very near future to do some more research.