Hi Catherine, Well, not sure if this helps, but some info that I have is as follows: a) The "Records-Secretary of the Province of Carolina (1694-1705)" mention an act the naturalization of a James Dufay, son of James Dufay and Judith Dognier (wife)born at Bologne, France
b)The New Dictionary of American Family Names (1973) shows Diffy and Diffay, the origins being "one who comes from Defay (communal grazing land)in France.
c) British Familiy Names by the Rev. Henry Barber (1903) lists Diffey under the same heading as Deaves or Dives, meaning coming from a location in Normandy (Fr. Devis)
There are also references in other books about other possible origins of Diffey. For example:
d) The Family Names of Ireland lists Devilly, Diffley and Devally, being the anglicized forms of O'Duibhghiolla, the name of 2 distinct septs of the southern Ui Fiachrach and of the Siol Anmchadha, both of County Galway. Diffley and Deffely are more usual in the adjoining County Roscommon.
e) Mrs Thelma Martineau of Little Rock, Arkansas in November 1968 wrote a paper that proposed that Diffee was a Gaelic name coming from the name Dubhsithe (pronounced Diph-See), perhaps derived from Dubhshith, who in 1164 was the Lector at the Cathedral on the Isle of Iona off the west coast of Scotland.
Well, I hope this has been of some use to you.
For your info. my family originated from Corfe Castle, not too far from you, and migrated to Australia in the 1840's.