I suspect I cannot be much help, although I'd love a connexion. Argyle as a middle name might well suggest a romantic parent, but Argyll is a very large county and 'Scotland' as a birthplace is a bit too vague.
My Argyll ancestors had nine children of whom four went to Ontario, but all nine were born too early to include your James, the youngest was born in 1796. One brother was a James, born 1784, went to Aldborough and died in 1860. I only know the names of two of his children, one of whom was Angus born in 1817, who also had a son James - but all of these people skip around your 1805.
Is there any chance that anyone might have written a history of the area that might let us home in a bit closer on where the immigrants came from?