There is no reason to believe that James Jolliff married Elizabeth Norris more than once.Her father, William Norris (d. 1806 Montgomery Co., KY), lived in Washington Co., PA, from about 1774.I don't know his origins.William's presumed son Charles Norris lived in an illegal settlement on the Ohio side of the Ohio River during the American Revolution, where James Jolliff also lived.
From information that Anne Legg has posted on the web, which came down through her husband's family and which dates from before 1920, James Jolliff was born in New Jersey, the son of Richard Jolliff, who died while a prisoner at Quebec.Essex County, New Jersey, probate records do show that Richard Jolliff, a soldier, died in 1759.
Thus the James Jolliff of Norfolk Co., VA, who died in 1775 leaving sons Richard and James is not the father of James Jolliff (d. 1821) of KY.This latter Richard was a Loyalist during the Revolution and went to Nova Scotia afterwards.The James was probably one of the James who died in Norfolk County in the 1780s or 1790s.
It seems that James Jolliff's mother may have remarried to someone who moved to Washington Co., PA, in the 1770s.James was in a list of Washington Co. militia in 1781, and several of the people there (the Willson family) were from New Jersey.I suspect he had some relationship to the family of Abner Wilson, who went to Bourbon Co., KY, from Washington Co., PA.Abner was his age.James probably named his son Abner for Abner Wilson.
I haven't been able to locate any other Jolliffs in early New Jersey.
William Jolliffe (d. 1770) was a Quaker who first appears in Bucks Co., PA, was married to someone said to be from New York City, so I doubt that he was related to the Norfolk Jolliffes.Most likely he came from England. It is possible he was related to Richard Jolliff, but there is no evidence.