Gilbert, Don't know if this will help or not, but found your Rufus in the 1920 census. I also found info on his father, don't know if you have that, but will give you everything I found. Some of it was hard to read that's why there is a ? here and there.
July 13, 1870 Beaufort County, Long Acre Township James Woolard 35 bn NC Ellen Woolard 31 bn NC Patsey Woolard 5 mths bn NC Living with them as farm labor Rufus Woolard 20 bn NCthis would be Rufus' father Nathan Woolard 18 bn NC
June 22, 1900 Beaufort County, Long Acre Township Rufus Woolard 50 bn NC Nancy Woolard 4? bn NC William Woolard Aug 1? bn NC Rufus E. Woolard ? 1890 10yrs bn NC