I am a descendant of Jesse Agee, through his daughter Locca T. Agee Railey.
We believe that the Jesse Agee in the 1799 Franklin Co., VA tax list to be the son of Anthony Agee, son of Mathieu.
Jesse R. Agee married Elizabeth Childress, d/o Jacob Childress & Mary Railey in Franklin Co. 3 Mar 1788.Surety was Philip Raley, probably Elizabeth's uncle.
Jesse was paying taxes in Allen Co., KY in 1815.On 6 Jan 1816, he received a patent for 200 acres on Bays Fork Creek in Allen Co. (Allen Co. DB C, p. 437.)(There are additional land transactions as well, in Allen Co.)
As stated in your post, Jesse is listed in the 1820 Allen Co. census.He died in 1820 or 1821.He is believed to be buried in the Agee Cemetery in Allen Co.
Jesse's heirs are listed in an 1824 deed and 1825 deed:
"... Jesse Agee deceased, departed this life having Polly, Sally, Jane, Ann, Susan, Patsey, Betsey, Lockey, Samuel, and Jesse heirs and representatives..."
As can be seen from the above quote, Jesse & Elizabeth had the following children (Dates and other information from several sources, including census records, marriage records, tombstones, and deeds):
Mary (Polly):m/1Maxey Keys, 4 June 1810, in VA;
m/2Adam Wray, 22 Mar 1813, in VA
Sarah (Sally):mEzekiel Watson, 5 Mar 1811, in VA
Jane:mJohn Martin, 1829, in Allen Co., KY
Anna:b. 1797, d. 14 Oct 1853; m William Pruitt, 23 July 1814,in Warren Co., KY
Susan*: b. 18 Jan 1798; never married,lived w/ sister Locca
Patsey (Martha): b, 1800; lived w/ sister Anna
Elizabwth Nancy (Betsey):m. Silas Moody, 1821,in Allen Co., KY
Locca Tabithab. 4 Oct. 1804; d. 21 Sept. 1889;m. John William Railey,in Allen Co., KY; buried Agee Cemetery, in Allen Co., KY
Samuelb. 23 Feb 1807;d. 25 Jan 1855;m. Sarah Ann Moody,in Allen Co., KY
Jesse R., Jr.b.1808;m Elizabeth Satterfield,in Warren Co., KY
*NOTE: In answer to the question "What is the Raleigh connection to the Agee's?":Susan lived with her sister Locca T. Agee Railey, w/o John William Railey, grandson of Philip Railey, who was surety to the marriage of Jesse Agee & Elizabeth Childress.That's the Agee-Railey connection.