Joseph and Anna Allen 1800's Caney River, Yancey Co NC

I am seeking info about Joseph and Anna Allen. Joseph appears in the following census records for Yancey NC: 1840, 1850, 1860, 1870. In 1850, my gg grandfather James Wilson age 15 is living with him and Anna.George D Wilson, Thomas Wilson and Paul Wilson are neighbors. In 1860, James, wife Lucinda and daughter Mary are living next door to him; James' full name "Jamerson" is recorded. In 1870 James and family live near him. By 1880 James and Thomas and families have moved to Gilmer Co Ga. He is now James R. Wilson and Thomas is Thomas A Wilson.
I have a theory that James Wilson is the grandson of Joseph and Anna Allen and the son of George D.Wilson. Possibly Thomas and Paul are brothers to George D. Also. in 1860 John Wilson (age 66)and Cecilia(60)live next doot to Thomas. Any info would be appreciated.