I know it's been a while since you posted your message but I have a Henry Kaye I've been researching. Mine is not a blood relation but instead the best friend of my gg grandfather who came to America with him. They are from Marsh Huddersfield born a 1837-1840. I realize this is too early but I would suspect a good lead would be through a brother of my Henry. I have his parents as Benjamin and Hannah, and brothers as Fredrick and Joseph, and a sister Elizabeth. He likely had more as this is from the 1861 census and I don't have the censuses for his family in 1841 but you can order it fairly cheap from Huddersfield. I'll get you the address if you want. I do have 1851 but it lists a different Benjamin and a Joshua with wife Ruth as the only heads of households.
I looked on the 1891 census and there is a Henry G Kaye age 2. I can email you the image if you send me your regular email address. His parents appear as Joseph and Emily. No other siblings are listed. He is the only one in Huddersfield of that age and they are in Marsh. I don't think Joseph is the brother of my Henry as there seems a generation in between. I don't see the same family with Henry on the 1901 census in Huddersfield. Hope this helps you. I can search for Joseph and Emily in all of Yorkshire if you want. Let me know.