I found this in the 1881 census whilst looking for Jarmans from Ramsgate who had moved elsewhere..
Household: Name Relation Marital Status Gender Age Birthplace Occupation Disability Richd. H. JARMAN Head M Male 55 Ramsgate, Kent, England Boot Maker Mary A. JARMAN Wife M Female 53 Broadamston, Devon, England Dress Maker Maria JARMAN Daur U Female 25 Hull, York, England Dressmaker Arthur JARMAN Son U Male 21 New South Wales Plumber Emma JARMAN Daur U Female 16 New South Wales Charles JARMAN Son Male 14 New South Wales Assistant Grocer William JARMAN Son Male 11 New South Wales Scholar
Source Information: Dwelling 141 Freeman St Census Place Great Grimsby, Lincoln, England
What caught my eye was the single child born in Hull. Clearly in 1856 this family spent some time in Hull, It is not impossible that this has a connection with Stephen given that there are only two Jarman households in Hull and of the two only Stephen comes from Ramsgate. So perhaps there might be a connection- but could they be your family? did they ever come back to the uk? Or are the names just a coincidence? Do you have documented deaths in Australia for Richard and Mary?