According to the research notes of W. Herbert Wood circa 1930 within the LeRoy Family file at the Historical Society of Newburgh Bay and the Hudson Highlands -
Nathaniel Woolsey was the son of Richard Woolsey by his first wife whose identity is unknown; she died in 1797; Richard married a second time to Elizabeth (unknown surname) who survived him and died in 1851 aged more than 90 years.She was living with the John S (or L.) Mackey family in Marlborough at the time of the 1850 census.
Richard Woolsey was born in Bedford (NY?); died 1827; he was the son of a William Woolsey.
Hope this helps.Certainly, "The History of the Town of Marlborough, Ulster Co., NY" written by C.M. Woolsey would probably have more on this family.