I found this in search of my Kizer line from McNairy County. I think your James might be connected, but am unsure how exactly (from the records I have, there is no James born in 1824...however, I don't have a complete listing of some of the lines). However, the other side of my dad's family are Williams. I have a Mary "Polly" A. Williams b. abt 1846. I have tracked her and her parents to McNairy Co. in the 1870 census. I am not sure if this is your Mary (Polly) Ann, but it sounds probable. The one I have listed...her parents are Lewis and Susan. If you want more info about this, let me know. My Polly lists Mississippi as her birth state in the census info.
My Kizer/Williams line occurs several generations later, but it is interesting that there are several different connections along the way (even in my own lists, I have seen several connections).