Just to be clear, where is James listed as a farm servant?
I wasn't sure if you are referring to the 1901 Census that I sent the url for. If it was at another location, then probably its a different man.
Bright is the "next parish" to Ballykinler, where Thomas came from. I believe Thomas died in 1910; the farm is very small and so it wouldn't necessarily have been inconsistent to describe oneself as a "labourer|" as opposed to a "farmer"
These were poor tenant farming families (i.e. they couldnèt travel too far!)and McGivern isn't a "frequently" occuring name, so you may be on to something. (Raholp is further towards Saul)