Hi. My information might pertain to ancestors of your grandfather. I'm descended from Pyes from Georgia, and I know that some were slave owners. My ancestor, at the time of his death in 1842, had 38 slaves in his will. But they are not listed by first name unfortunately. But my understanding at the time is that many or all would have gone by the last name of Pye. That was in Putnam Co, Georgia. I think some Pyes related to my line moved to Alabama, but I don't have information on those.
I feel for you and wish you luck in your search. If I come across anything, I'll post it here.