I don't know why his birth record says that his father was Charles Fiske but his mother was Gratia Wilson and she also went by Mary Wilson. I show that his father was Jonathan Fisk and Jonathan was the son of Benjamin Fiske and Margery Wood.
I show James' known siblings as: David Anson Fisk Pauline Ann Fisk Benjamin Wilson Fisk Lucius Carey Fisk Elizabeth Melita Fisk Sarah Jane Fisk Jonathan Edwin Fisk Charles Austin Fisk William Prentiss Fisk (James Wood Fisk) George Fisk Henry Fisk
All were supposedly born in Milford, Massachusetts.
If you have a subscription to Ancestry.com, they have the 1850 Census image with James in this family but they transcribed the name as Hiske instead of Fiske.