Your love story is very pretty but the 1851, 1861, and 1871 censuses do not agree with it,
In 1851 Caton and Ann were alone,No children anywhere.
On 1861 census Elizabeth Ann is listed as 9 years old.This seems to indicate that Elizabeth is the child of Caton and Ann Varty.Also on the census were Harvey, Ezra, and Sarah--all younger than Elizabeth.
On the 1871 census were Sarah, Ida Eva, and Lucinda.
On the 1881 census was with Matilda and the following children were listed: Emma age 8, Sharlot [Charlotte] age 6; George; with the twins Albert and Ellen.
I think that with your infomation that Caton could have been married before Anna Varty but had no children.