I beleive that he was married to Mary Whitworth based on the following:
Transcript of George Moore's will 1798 Prince Edward Co. mentions daughter Elizabeth Malone and her children James Daniel Cole and Polly Whittier Smith
Elizabeth Malone's will 1829 Clarke Co. Georgia children James D. Cole and Mary W. Smith (nee Cole) by her husband Robert Smith
Josiah Cole's will 1795 Oglethorpe Co. Georgia daughter Mary Whitworth Smith, son Daniel Cole
marriage 1783 Caswell Co. North Carolina Mary Whitworth Cole and Robert Smith
marriage 1810 Amelia Co. Virginia Joseph Smith and Judith Townes
Joseph Smith's Bible see Sumner Co. Tennessee genweb page "Joseph Smith son of Robert and Mary W. Smith" Among their many children are Mary Whitworth Cole Smith and Joseph Washington Moore Smith. Also children with middle names from Judith Smith's line: Townes, Booker, Marot.
1850 census Joseph Smith said he was born in Georgia.
Aneita Mitchell Allen (descendant of Lucy Monroe Smith Mitchell, daughter of Joseph and Judith Smith)