My second great grandfather was Robert Duthie, born 11 Nov 1842 at Capo,Stracathro parish, Angus to John Duthie and Mary Duncan Duthie. He married Jane Milne on 1 Dec 1865 and then went to Ireland as a land steward. He returned some time after the 1901 census and retired in Montrose. He died 1 Dec 1920 and is buried at Rosehill Cemetery in Montrose. I do not know about David Duthie born about 1830, because I never found a register entry for his birth and he was not with the family in the 1841 census. The children of John and Mary Duncan that I know about are Margaret (c1828- ),Charles (1832- ),John (1834- ),Jane (1836-1861),Andrew (1838-1870, died at sea off Flamborough Head and was buried at Grimsby),Alexander (184-1863), Robert (1842-1920), and James (1846-1919, a paint merchant in Glasgow).
I think my family sounds very much like yours. I could not find Charles after the 1851 census, or John after 1865. I do not know for sure where Robert was in Ireland, but family tradition says it was County Clare.