You may be interested to know that John C. Ivory was listed among the injured in the Gasconade Bridge Disaster on November 1, 1855, when a Pacific RR locomotive and fourteen cars with some 600 people (mostly wealthy businessmen and politicians)aboard plunged 36 feet into the Gasconade River when the bridge collapsed. At least thirty died as the cars fell on top of each other and more than 100 were injured. Although your ancestor survived, this may help to explain your family's train legend. (One of my gggrandmother's cousins, Emilien C. Yosti was killed.)
There is also a street here in St. Louis named after John C. Ivory (i.e., Ivory Street).
I have a copy of the Daily Democrat article from November 5, 1855 in which he is listed if you would like to have a copy.