John, I just received James' pension records from the National Archives. I'm still looking through the material, but have found information that he was married to Charlotte Milligan first. Several witnesses state that she died in March 1864. No idea what happened to Charlotte's young children. He then married Rachel Noice in 1866.
There is even a description of James' appearance. Will continue to sift through material.