Pretty much what I figured. With a child that young you can assume that he was not to old for service in the Confederate Army during the War between the States.
A John Carlton from the Spartanburg District enlisted as a private in Company E, Holcombe's Legion, South Carolina Infantry and was killed in action on 8/30/64 at the Battle of Second Mannasas, Virginia.
A J.T. Carlton enlisted as a private in Company K, 27th Regiment, South Carolina Infantry. This was a Charleston Regiment but men came from all parts of the state to enlist. My ancestor came the upcountry (Pickens District) to enlist in the 27th South Carolina Infantry.
I have no more information on these men but there is a good chance that one is the person you are seeking. Service records of Confederate Soldiers can be obtained from the National Achives in Washington, DC. The web site explaining how is http://www.nara.gov/genealogy/civilwar.htmlhttp://www.nara.gov/genealogy/civilwar.html
If he survived the war there is a good chance that he went to Texas or another western state after the war so don't restrict your search to the 1870 South Carolina census. If you have other questions about the war there is a Civil War forum at http://genforum.genealogy.com/civilwar/http://genforum.genealogy.com/civilwar/