There was a split in the original Gravesend VS property during that era by inheritance. Though the genealogy provided is not correct, the division of the property is detailed in Rosalie Fellows Bailey's book, "Dutch Houses and Families in Southern New York and Northern New Jersey" (Dover Press).
It seems conceivable to me that the one branch inherited the private cemetery while the other branch made other arrangements.
I am speculating with this, but one can easily imagine how the disposition of the property might have generated the folk lore. Only the one branch inherited the ancient VS house.
The property was recombined by Thomas Hicks and his VS wife, then commenced to leave the family altogether and finally when their estate was settled.