That's very interesting information. I would venture a guess that your William(?)Swann was indeed the son of William B. Swann and Sarah Taylor. From the information that you've provided it appears that William B. Swann and Sarah Taylor were having an affair which was undoubtedly a factor in the divorce of William B. and his first wife, Martha J. Seargent.
You mentioned that there are conflicting dates of birth pertaining to your William Swann. Did he live into the 1900's? If so, it's quite possible that his death certificate would provide a reliable (hopefully) source of information. Also, the 1900 census should provide the month and date of his birth, assuming that he was still alive.