Don't think I have anything helpful about your grandfather Frederick Hall Alfred.
Charles Alfred in 1870 had 6 children - Edward (age 10), Maggie (9), Rosa (7, A. Charles (6), Jeanie (4), and Mary (9 months).All except Mary were born in Canada, Mary in NY.Obviously Edward or A. Charles could have been the father of your grandfather (born c. 1892).
Richard Alfred had 3 sons - Albert Richard (b. 1866), Frederick (b. 1869), and Richard (b. 1876).Frederick's two children were Mildred and Donald.Richard is too young to be your grandfather's father.The eldest, Albert Richard would be a possiblity.
The 1886-1888 Brooklyn, NY city directories list Albert R. Alfred, bookbinder, living at 832 Bushwick Avenue - same address as his father, Richard Alfred.Albert R. Alfred disappears from Brooklyn directories after 1888.
I am not aware of any connections of my Alfred family members to Port Byron, NY.But my Alfred family information is essentially limited to Richard Alfred's children, Grace Ella and Frederick.
I believe my grandmother, Grace Ella Alfred was known as Grace Ella, not just Ella.