I've heard back from my cousins and they've given me a little more information. The earliest item they could find was the marriage licence for James Toogood and Ann Clark in 1790 which gave his age as 30 and hers as 19. He had been in Arundel (spelt Arundle in the Burpham register) for 2 years and she had lived there for many years. This must be the same James that your mother found, as his age almost correlates with her estimation of his date of birth, but you can never rely on the ages quoted!
My cousins also searched for James's baptism, but to no avail. They did come across an entry at Steyning, which is also in Sussex, for Thomas and Ann Toogood who had a daughter Elizabeth in 1755 and another daughter Ann was born at Upper Beeding in 1757. Do you know of these people?
I now have a name for the father of the illegitimate Robert. The name on his marriage certificate is given as Edward Nagby, farmer, but apparently the family has always suspected that this was a smokescreen and that a local aristocrat was the real father (I believe that Robert's mother Elizabeth worked in his household, but don't quote me on that). My cousins tried searching for Nagby but found no trace of him.