What's the source you refer to regarding James' birth place?If it is the Banns Register, that is giving you his current residence, not his birth place, so you'd need to look further afield.Try the parishes around Llanfihangel Abercowin, to at least three deep on all sides, and compare any baptisms against the burials to make sure that you only consider those who survived infancy.
If the 1851 Census is the source, and he died after 1861, try finding him and Jane in that Census and see what it says.
His occupation may give you some leads.If he had a trade, rather than being an Ag lab or a Miner, try to get a look at Apprenticeship records.
Also, try for any surviving Settlement records which should confirm his birth place and may even mention his father's name, especially if the whole family was re-settled when James was a boy.
The Poor Law Union Workhouse records may also give you some information, too.Most families needed "Parish Relief" at some stage and if he lived to be elderly and/or infirm, he may have ended up in the Workhouse.