After Authur Dillard's death, the 1850 census for Jackson Co FL has the dillards living with my great-great Grandfather William Yarbrough and his wife Sarah W Dillard-Yarbrough:
91YarboroughWilliam43MPhysicianS.C. Sarah W.24FGa. Gibson A.10Mdo William H.8Mdo George6Mdo Sarah J.4FFla. John B.2Mdo Jesse C.4/12Mdo DillardPolly60FS.C. Frances L.19FGa. JulianElizabeth30FGa. (Widowed) Frances J.11 Fdo (dau of Elizabeth Dillard-Julian) DillardJoseph32MLaborerdo TyusP. G.30MSchool TeacherGa. The 1860 census does not show Polly Dillard. The daughters Re/married in the late 1860s and the Yarbrough's moved to Bainbridge, Decatur Co. GA.