1904 death registration indicates he was born in the United States and was 88 years of age at death in Grantham Township, Lincoln County, Ontario (which would never be considered part of the Greater Toronto area).
1901 census has him living with his children in Grantham Township, born 15 October 1816, aged 84 years in the US.
Grantham Township, Lincoln County, Ontario
James, Robert, widower, aged 74, born United States, father b. England, mother b. Ireland, Church of England, farmer (1901 census gives his ethnicity as Welsh)
James, George, aged 28, single, born Ontario
James, Joseph, aged 25, single, born Ontario
James, Eva, aged 23, divorced, born Ontario
James, William, aged 20, single, born Ontario
James, Minnie(?), granddaughter, aged 14, born Ontario
James, Leslie, grandson, aged 11, born Ontario
In the 1881 census, Robert's wife is noted as Frances, aged 57, born in Ireland. Additional children are Jane, aged 37, and Edward, aged 28, and Frances (daughter), aged 25, and Mary, aged 22.1871 Census has wife's name as Frances. Another daughter, Elizabeth, aged 28, is noted as well.
William John's marriage record in 1877 names his parents as Robert & Barbara, but other children name their parents as Robert & Frances, so perhaps Robert was married twice.William John, born in Pennsylvania, married Mary Catherine James, both of Clinton Township, Lincoln County. She was the daughter of Joseph & Maria James. April 17, 1877.
Makes me wonder if there are two Robert James in the same area, given that the one who died in 1904 had a son named William who was considerably younger than the one who married in 1877. Or if William John's father never came to Canada at all.