I checked 1871 and 1881 Census data. I found three Louis Ducharmes of about the right age, but none of them seem to be yours. The first two I was able to track through to the 1891 Census, and the third one married in Quebec in 1888.
I have two recommendations:
1. Look for other Ducharmes in the same county in the US Census. It is unlikely that he would have come alone. Finding at least one sibling would help.
2. Widen the search. Your Louis Ducharme may actually have been a Louis Charron. A branch of the Charron family went by Ducharme.