Entry in the 1917 book The Beginnings of Public Education in Virginia,1776–1860 by A. J. Morrison. “This academy was in existence in 1813. In 1834, Martin [of Martin’s Gazetteer of Virginia] says of it: ‘A flourishing and useful school a few years since, but we believe there has been no teacher there for some time past. We now, however, see an advertisement which states that the school will he opened on the 15th of January, 1835, with teachers fully competent to teach all the branches of education usually taught in our schools.’ The arms lately used by the Port Royal Guards in the county of Caroline, were authorized to be transferred to the trustees or principal of the Rappahannock Academy in 1848. About this time the principal was William Mahone, Virginia Military Institute, 1847, later a general in the Confederate Army. In 1873, the trustees were authorized to sell the academy, and 264¾ acres of land appurtenant, the proceeds to be turned over to the county school board of Caroline county, to be by them applied to the use and benefit of the public free schools of the county.”
You might contact the Caroline Co. Historical/Genealogical Society to see if they have more details.