We didn't have as much time in the library as I'd hoped, so I never got hold of that book, but I did get some information another way. I posted a message on the Nance forum and a Dave Nance replied to me with this. It seems the Nances don't believe that Joab M. married a Nance, either, but there are connections elsewhere down the line:
I have run across a series of books, "The Duke Family"(1979) and "The Duke Family, Vol II" (1995), by Evelyn Duke Brandenberger, which has a great deal of information about the connections between the Dukes and other families in Virginia. The books seem to be very well researched, with lots of annotations to wills, property records, and similar records that support its conclusions. The book states that Joab Mountcastle (d. 1773) married Anne Duke, dau. of James Duke (m. Mary Byrd, d. 1751).
The idea that there was a Nance-Mountcastle (or "Moundcastle") connection is not incorrect, however. According to the major Nance family genealogies, "Nance Memorial" and "Nance Register", Zachariah Nance, son of Eaton Nance and grandson of James and Ann Nance, married Mary Moundcastle.
These books say that Mary was the daughter of Benjamin Moundcastle, but I think, based on information in the Brandenberger book, that this part of their information is incorrect, and that Mary Moundcastle was in fact the daughter of Joab Moundcastle.
I should note, that some of the information which the Brandenberger books have about the Nance connections, is very wrong. Specifically, it confuses the Zachariah who married Mary Moundcastle, with a Zachariah Nance who was actually his uncle. To illustrate:
James and Ann Nance. Children:
1) Eaton Nance. Child:
a) ZACHARIAH Nance, m. Mary Mountcastle
2) ZACHARIAH Nance, m. Susannah Duke Sherman
(other children are omitted from this chart, to simplify it)
I am pretty sure that there was another connection as well. Susannah Duke Sherman was almost certainly the daughter of Susannah Duke (who m. Michael Sherman). Susannah Duke was the daughter of Thomas and Elizabeth (Marston) Duke, and Thomas Duke was the son of Henry Duke. Henry Duke also had a son James Duke -- who was the father of the Anne Duke who married Joab Mountcastle.
The 1979 Brandenberger book says, in part: "Joab Mountcastle, one of the executors of the James Duke Estate, was a son-in-law of James Duke, having married his daughter Anne. Joab Mountcastle died early in 1773. In his Will written the 5th of September, 1770, proved in Charles City County on May 5, 1773, he mentioned his wife Anne; sons John, David, William, Benjamin, Edmund, Joseph, Richard and Henry Mountcastle; his daughters, Elizabeth Fear (Ferrer), Mary Lanced, and Sary Charles. Witnessed by John Christina, Samuel Christian, and Eaton Nance. Securities were John Christian, William Marston, and Henry Duke (Charles City Co. VA Will Bk 1766-1774, p. 447)." The 1995 (Vol. II) book notes that the "Mary Lanced" in the Will was probably really Mary "Nance". This matches with the book's assertion that Mary Mountcastle, dau. of Joab and Anne (Duke) Mountcastle, married Zachariah Nance.