According to his Pennsylvania death certificate (now available on Ancestry.com and free to Pennsylvania residents) James died Dec 2, 1910 in Phillipsburg. He was born April 4, 1846 in Newton, St Cyrus, Devnonshire, England to John Passmore and Maria Babb. You are very lucky in that the death certificate gave the exact place of death. Quite often they would just list the state or country where they were born.
You should be able to find him in the English census records. I suggest getting his obituary. Check with the Centre County Library or the Centre County Genealogical Society to see if they can help you find it. His obit may have additional data such as siblings.
I believe James immigrated around 1868 (according to the 1900 census) which would make sense since he is in the 1880 census of Centre County.