On page 8 (online!)the 1840 Gilmer Co. Ga. census shows consequtively, Riley Allen, Robt Allen, Nancy Allen and Aaron Pinson.17 names before on page 7 is William Allen, the only other Allen listed in that census.William and Riley are listed in their 20's and Robert in his 30's and his wife Lydia in her 20's.Nancy Allen and Aaron and Mary Pinson in their 50's.Nancy Allen has two sons in their late teens in her house.Judge Aaron Pinson was Lydia Burch Allen's (Robert's wife) uncle.
In 1830 Rabun Co. Robert Allen is listed in his 20's and wife Lydia in her teens.They are right next door to Joseph and Mariah Heddon Eller.Mariah was Lydia's first cousin.7 doors away are Lydia's aunt and uncle, Josiah H. and Jane Pinson Carter.3 more doors away are another first cousin of Lydia's, Margaret Pinson Kelly (Mrs. John L. Kelly).The Kelly's are living next door to William Allen.Both he and his wife are shown in their 40's.Stat's don't rule out William's wife being the Nancy Allen shown in 1840 Gilmer Co. Ga.Margaret Kelly is the daughter of Judge Aaron Pinson who lived next to the Allens in 1840 Gilmer Co.
In 1836 Rabun Co. "Rial" Allen marries Zepporah Norton.I'd guess this is the Riley Allen shown in 1840 Gilmer Ga. by Robert and Lydia.Significantly, Judge Aaron Norton was the coadministrator of Gideon Nortons estate in Macon Co. before he moved to Gilmer Co. Ga.Gideon had been Joseph Pinson's (Judge Aaron's father) neighbor in Pendleton S.C.This gets more complex...I won't confuse, but you see all the relationships developing here.
I think this Allen line was in Macon (formerly Haywood) N.C. with Aaron Pinson et. al. before moving to Rabun Co.I think they are also related to the Denton and Chastain lines (these 3 families intermarried often and were also connected with Nicholson, Brown and Pinson lines etc.).
It's quite likely that your line did originate in Va. then moved to the Carolinas.