I don't know if someone changed the spelling of the last name or if the census takers messed up, but in the 1860 Illinois census, Rock Island county,is the William Patterson family with son James W.aged 2+ birth year 1857 or 1858.
Right under the Patterson's are listed the Essex families of Benjamin and James J. Essex, and nearby is Simeon Essex.
William Patterson is 40 years old, born in Indiana. His wife Virginia Patterson is 25 years old born in Virginia.
I also found James Patterson as a 22 year old in the 1880 Dakota Territory census, union county, which makes sense if your grandfather was born in Dakota in 1881.
It sure looks like the people you are looking for. Hope that helps.