It would be hard (impossible?) to isolate these factors. The age of the father does seem to correlate with mutations. There might be some accumulated effects that contribute.
However, it is more likely that yours is a case of small sample sizes and/or multiple surname origins. It is best to compare each person to the modal (or ancestral) haplotype of the group than to one another. See: