Brian, I'm having an "OMG" moment as I read your posting. I've been researching the possible origins of James Olds/Oles of North Ashford, CT for years, and what you propose makes absolute sense. I need to research this for a solid conclusion. However, It is very noteworthy that James has a son George (b. June 24, 1750)with his first wife, Elizabeth Fuller. She died and he then married Sarah Kingsley.
I'm going to reach out to family researcher Dan Olds, and we can have him weigh in as well. If this George is James' father, it is a MAJOR breakthrough. I hope that my post here is not to quick - I emphasize that we need to research what records exist.