Took a look at this family and I concur with what you have written, but I wonder if the Jane was Isaac's sister, rather than a second wife.There is a Jane Walters of similar age living with them in 1850 & 1860.
I also concur that William was most likely a brother.Found where he died 2/7/1875 Madison Co., born ~1801 Granger Co. VA [Orange Co. VA], father Ged [Geo] Walters, mother Nancy Walters. - (No orig doc available, so have to interpret the possibilities from their transcription.)
Found in Orange County Marriages George Walters married Nancy Harvey 9/30/1800, bond by John Harvey.
I think this George & Nancy were most likely the parents of Isaac, William, & Jane.
Also found a marriage for John Harvey and Lucy Estes 4/19/1778 - possible parents for Nancy Harvey.
I suggest you contact the Historical Societies of both Madison and Orange Counties.They will have more knowledge of what records survive.Also Madison Co is working on a project to identify graves in small family cemeteries, etc.They may have found something.
Based on census info and old maps it looks like Isaac lived on the west side of the Rapidan River,NW of the town of Orange, so the family coming from Orange County makes sense.Lots of Taliaferro's in that area also.