The closest story I could come through is the following:
Quebec census 1861 Chateauguay Andrew Tolmie blacksmith 42 Scotland 1819 Agnes Kirkwood 42 Scotland 1819 Elizabeth 13 Scotland 1848 Agnes10 Nova Scotia, Can 1851 Andrew 8 Nova Scotia, Can 1853 James 5 LC (lower Canada)1856 From this census we know that Andrew Tolmie and Agnes left Scotland between 1848 and 1851. Immigration data is not available for those years in Canada yet.
June 1854 Passage from Canada to US : Sydney, Nova Scotia to Boston Ship:Lch W L Richardson Intend to immigrate to US Andrew Tolmie 34 Agnes 34 Elizabeth6 Agnes 3 Andrew Infant male Jane McKenzie 24 (servant?) She is not mentioned in 1861.
For some reason they could not immigrate to US in 1854 as we find this family in the 1861 Quebec census in Chateauguay where we can see James born Lower Canada (Upper Canada was Ontario prov.). I have not been able to find James birthdate.
Andrew born in 1853 Nova Scotia, married Matilda ? from West Virginia as we can find him in the 1900 US Census in Ohio Belmont, Bellaire All this info is from Ancestry.com I could not find any couple by the name of Andrew Tolmie and Agnes married in Scotland on any of the census in Scotland in or before 1841. Elizabeth is born 1848 so they escape Scotland census between 1841 and 1851. I could not find any death for Andrew or Agnes in Canada either. Hope this will help you, I could not find more. Nycole