Oliver appears in the 1860 as living in Binghamton. Olliver (yes Olliver) appears in the 1850 census living in Vestal which is also in Broome County, NY. Also living in Vestal at that time and listed on the previous page is a Joseph Chalker 66 and Sarah 55. Joseph was born in Connecticut and Sarah in Vermont.
The 1880 census lists a Putman Chalker 56 who lives in Union Twp. His wife is Louis 53 and daughter Dale 26. This is the only Putman Chalker to appear anywhere in the census. His parents are listed as born in Connecticut. I believe it is Oliver. Oliver appears in the 1892 NY census and Louisana is a widow in the 1900 census.
There were a number of other Chalkers in Broome County who were born in Pennsylvania. They may have been Oliver's brothers and Joseph may have been their father. Sarah may or may not have been their mother.
A local historical or genealogical organization may be able to help you find something on the family.