Congratulations on such good detective work! You have certainly helped Judy with at least the James family, if not the elusive family of Victoria Wedgeworth!
In the 1880 Texas Census there are 110 William/William F. James, and five Victoria James. It doesn't seem like any of them are the ones we are looking for. A possibility is Victoria James, age 25, born AL, sister-in-law in Newton County Texas.
I did find William F. James in the 1910 Census Runnels County Texas, age 49, born TX, second marriage. So this confirms your post about the William Foster James we are looking for. It also offers us a clue - William James is with his parents on the 1870 Census Austin County Texas. Austin County is right next door to Washington County, and there was a Wedgeworth family in 1850 Census Washington County.
1850 Census Washington County Texas; A. Wedgeworth born 1805, wife Ann born 1810, James born 1835, Alexander born 1838, Harriette born 1840, Mary E. born 1842 and Abram born 1847. I beleive that "A" is Alford Wedgeworth, son of James Wedgeworth, Junior and Jincey Walker. However, the big problem is that not even one member of this family is ever found again. Where could they be? What happened to them? Are they inadvertedly listed under the wrong name? Could Victoria be a daughter of theirs born after this Census? Washington County is next door to Austin County where William F. James was listed.
Where do we go from here? Judy, and any one else out there, can you help us find this family?