I've been doing some digging to try to find the ancestors of a Francis Walker, who appears in both the 1850 and 1860 Censuses for the Howard District, Anne Arundel County (now Howard County, Maryland).In each census Francis is living with another family.
In 1850 he's a 9 year old in the home of Basil Johnson, who I believe is his maternal grandfather. Basil married two women named Blowers, first Elizabeth, and then a year later, presumably after her death, Mary Ann, presumably Elizabeth's sister. Mary Ann, then, would have been Francis' grandmother, but was deceased by 1850.
In 1860 Francis is a 21 year old in the home of George Thompson, whose wife's maiden name was Mahala Blowers. I believe that Mahala Blowers Thompson would have been Francis' great aunt, the younger sister of his grandmother Mary Ann Blowers Johnson. The neighborhood in thoses censuses is thick with Johnsons and Thompsons, but no other Walkers, so unless we learn where Francis' father came from, the Walker line may be the hardest to trace. The Blowers connection may be more promising, especially if some proofs can be found for the above relationships, which are merely logical conjecture at the moment.
There are a number of other Blowers, Johnsons and Thompsons in the area in the various censuses, but not enough to permit connecting the people up.If others have more information that can help sort out these families, please post!