I believe you are mostly correct.Charles Carroll of Carrollton (born 1737) had only a single son Charles (born 1775), a single grandson Charles (born 1801), and a single great grandson Charles (born 1828).So if your earliest male Carroll ancestor was born before about 1848, from wherever, you cannot be descended from Charles Carroll of Carrollton.
The great grandson DID have sons Thomas, John, Oswald and Albert.Their mother was Caroline Thompson and they would all have to be born after about 1848.
Charles Carroll the Immigrant (born 1660) in Ireland, had two sons who survived childhood who were Charles (born 1702, father of Charles Carroll of Carrollton) and Daniel (born 1707).Daniel had offspring including Charles (born 1729) and Eleanor (born 1731), who did marry Daniel Carroll (born 1730), son of Daniel Carroll Settler.They did have children including Thomas, said to be the father of Major General William Carroll, who became the 40th Govenor of Tennessee.
Charles the Immigrant and Daniel the Settler are not known to be related.However they both married into the same Darnall family.Charles married Mary Darnall in 1693, and Daniel married Eleanor Darnall in 1727.Mary Darnall was an aunt of Eleanor Darnall.
Charles Carroll of Carrollton, grandson of the Immigrant, signed the Declaration of Independence.
Daniel Carroll, son of the Settler and husband of Eleanor, signed the Constitution of the United States.