Jo, My James Sweat, with a young son, Allen/Allan, was in Marlboro County in 1850.I have been unable to find him on any earlier census, or later census.His young son, Allen, appears as James Sweat on a Georgia 1860 census and his wife and children are followed from there on.
My James Sweat (father of Allen/Allan), doesn't appear to be connected to yours.Since I do not know his father's name, I do not know if the father might be connected.If the father of my James had a brother named James (uncle to my James), then that James may be the one you are looking for.Unfortunately, I have not had the time or health to pursue looking for an 1840 Marlboro County census to continue trying to track my James - I also do not know his father's name.
I wish you luck - I only wish I could be more helpful. . .
P.S.My James Sweat last appears on the Marlboro Co. census 1850 - then he and his wife Lavia(?) disappear.His two other sons appear on the 1860 census, and Allen/Allen (James A.) appears on a Georgia census.Where James Sweat, the father went, I do not know. . . .