In the Buchanan county 1860 census there were a rather large number of errors in age entries especially when concerning the Baldwins I am finding. What I am guessing is that neighbors answered questions in some cases. This Stephen and Phebe I believe are the same ones living near Jacob and Elisha in Russell county in 1850.
In 1860 Buchanan county we find that Elisha is home #187, Stephen is home #189, Susannah (believed to be Jacob and Stephen's sister) is married to Isaiah Stephens and they are at home#197. Jacob is a few homes away at #215.
By 1870 we find Stephen and Phebe with children Rebecca, Solomon, Cintha, Mary A., Ann, and Nancy. They are listed at home #160 in the Sandlick District. Jacob with wife Orpha and children are at home #4 in Garden district. Right next door to them is another Stephen (Jacob's son)who was married within the year.
In 1880 we find Stephen Baldwin (and family) listed as Stephen Balding and still in the Sand Lick district of Buchanan county Va. He and Pheby are listed with children Rebecca, Solomon, Sinthia, Mary A., Sarah A.,Pheby A.,Reeny M., and William D. Jacob and family are still located in the Garden district along with SD Baldwin (Jacob's son Stephen) and wife Hannah. Apparently Hannah was a second wife and was married to a Taylor as there are two stepdaughters Sarah and Marietta Taylor on the copy of the census record I have in my files. In 1880 we also have the first appearance of another Baldwin family on the census, this family was John and Celia Baldwin and children. A marriage between William E. Owens (21) and Sarah Anna Baldwin (age 18) on March 14, 1886 in Buchanan county VA at Stephen Baldwins home. The groom was the child of Eliah and Sarah Owens. The bride was the child of Stephen and Phebe Baldwin. The minister was BJ Street. This tells us that they were still in Buchanan county in 1886. Based on all this I don't think this Stephen left Buchanan county once he settled there. I'm open to bright ideas. (or not so bright) :-)I'd be happy to send you what files I have written up in word form if you like.