The domestic life of Tom Sayers was tangled and miserable. I should know, because I've just had a book* published on Tom's life, the main focus being on his great fight with John Heenan, the Benicia Boy.
The story of his marriage, his children, his later cohabitation and so on is just too complex to go into here in detail, but I've ploughed through Census records, also records of Births, Marriages and Deaths to be sure I've got it right. I'll just make one or two brief points here.
He cohabited with, and later married, Sarah Henderson, not Sarah Powell. The name Powell, however, makes a mysterious appearance later in his story: I *think* the woman he lived with after his marriage broke up may have been named Powell, though she called herself Charlotte Sayers.
As for children, the Aldridge Sayers children were clearly not his, and it seems unlikely that Tom had any offspring other than Tom and Sarah, both of whom were born to Tom senior and Sarah senior before they were actually married.
Information on Tom's marriage etc. is to be found in chapters 3, 7 and 17 of my book. Be warned, though, that I don't take the story beyond 1868 (three years after Tom senior's death) when a court decided that Tom's estate should go to the Aldridge Sayers children.
*'The Lion and the Eagle' (ISBN 978-1-899807-67-3) by Iain Manson, is published by SportsBooks in paperback at £14.99.