I note that Mary was 39 in the 1860 census. Then she was 47 in the 1870 census. Both of these census were in Holmes county, Mississippi.
Now you you are suggesting that 10 years later she is 42 and that the family has moved to Wise county, Texas. And that there are two additional children that have appeared that should have been listed in the previous 1870 census. Also note the James is no longer born in Mississippi but now is born in Texas.
There are some similarities, but too many discrepancies. I think they are different families and that those in the 1880 census are different folks.