This story is a tradition among the descendants of John "Esarey" (b.1744). The name of the brother is not always part of the story. I've heard Joseph as the other brother, whose family uses the "Esrey" spelling). John was nine years older than Joseph, but they married about the same time. I've never seen evidence of a brother James except for an unproven tradition among those who spell "Essary." More information about the brother James in your story, or where the story came from, would be high value.