While looking for the Mary Bryant Suthard family of Alexandria, VA, I came across these Suthards on pg 115 of the 1860 census of Alexandria. (found in genealogy special collections at Kate Waller Library in Alexandria) Could your Sylvester have moved the short distance from Prince William to Alexandria?
SUTHARD, John Hlaborer45 m Eliza49 f Delaney11 m Mary6 f William3 m
SUTHARD, Thomaslaborer40 m
SOUTHARD (pulled up original handwritten copy and it is SUTHARD, believed to be Mary Bryant Suthard),: SUTHARD, Mary A26 f John10 m Sarah7f William6 male Charles2 male Bryant, John J57 ship carpenter (Mary's widowed father)