I found this census that could be a match but I can't find any other records to match. Certainly has the GA - AL twist to it. There are step children so this is not his first marriage.
Household Record1880 United States Census
NameRelation Marital Status Gender Race Age Birthplace Occupation Father's Birthplace Mother's Birthplace Joseph M. ADAMSSelfMMaleW50GAFarmerVAKY c1830 Sarah A. ADAMSWifeMFemaleW53SCKeeping HouseSCSC c1827 Edward L. ADAMSSonSMaleW7ALGAAL c1873 Emily B. ADAMSDauSFemaleW6ALGAAL c1874 Mary J. ADAMSDauSFemaleW2ALGAAL c1878 James M. ADAMSSonSMaleW2ALGAAL c1878 Martha MCCULLOUGHSDauSFemaleW18GAAt HomeSCSC c1862 Sarah MCCULLOUGHSDauSFemaleW16GAWorks On FarmSCSC c1864
Source Information: Census Place North Carolina, Floyd, Georgia Family History Library Film1254146 NA Film NumberT9-0146 Page Number246A
This could be your J M ADAMS and his grandmother, Jane may have stepped in for a while to look after the children.
Husband Samuel Adams Birth:01 FEB 1784Of, , Prince Edward, Virginia Marriage:1803Ruckersville,Elbert, , , Alabama Death:06 NOV 1861Midway, Bullock, Alabama Burial:Baptist Churchyard, Midway, Bullock, Alabama
Father:James Adams Mother:Jane Cunningham
Wife Martha Ann Thornton Birth:1789Of, , Prince Edward, Virginia Marriage:1803Ruckersville,Elbert, , , Alabama Death:NOV 1862Salem, Russell, Alabama