The Doreys listed themselves as French in 1881 census and as coming from Quebec. I suspect their name was Doré but I haven't been able to get further than Ganonoque.
The records for Leeds County (Ganonoque) are in Brockville. If you live in the are the Brockville Museum has a great genealogical library and archive where they can help you locate what you are looking for. I can't remember the address and I am living in France now, but it is easy to find. Good luck.