I spent time looking and time lining. It does seem unlikely that the family could have been in New Hampshire in the late 1700s - 1800s. All we have to go on is the 1860 census for Lydia Rix Barber, which is ditto for under NH, as her husband, Erastus Barber was born in NH. That could have been an error.
As for them being in Grafton VT. What I found in one history of Grafton, NY which lists Rufus in a group of early settlers from Grafton, VT. This also could be true, or just an error.
Erastus Barber and his father, Erastus Sr were in Orford, Grafton County NH in the 1800 census. There were two Rix's in Grafton County, James and Nathaniel. I do not know if they were related to Rufus, but I thought it might be a lead to Rufus when I first posted about 2 years ago but I think, now, not. No direct evidence of Rufus.
It would seem more likely that he came out of the army in the Albany area, lived in Stephentown near by and went to Grafton about 1796, per town histories, perhaps from VT, but likely not.
Like you, I cannot find any record of Rufus after 1820 Census, so, like you, think it likely he is living in another household.
I am still looking for the note on Rufus wife Lydia as to she is thought to be a Lewis. I believe it was in one of the town histories, earlier than Springwater, but have not located it. As I said, it was a comment, a clue, but not definitive.
I found an email from you from about two years ago, thanks. It has been that long then since I am starting to pick this up again.
Again, if you want me to email any jpgs from my husbands great aunt or the timeline I put together with some notes from the town histories, let me know.