I ASSUME you mean the Albert Kelley who died in Dayton, OH in 1897 and served in Company D, 87th PA Infantry. According to U.S. National Homes for Disabled Volunteer Soldiers, 1866-1938 he was born in Philadelphia. Finding him in the census with any certainty before 1870 is a tall order. Kelley/Kelly is too common and without additional data it is hard to prove who his parents were by the census.
I would start with when he died. I suggest trying to get his obituary in Dayton. It may not list his parents but it might list surviving siblings. He received a pension for his Civil War service and you should be able to get a copy of his pension file from the National Archives in Washington, DC for $75. Although quite pricey and it's had to say what might be in his file it might have exactly the info you are looking for. It should list where he was born and will list everywhere he lived from when he enlisted to his death. It should also have affidavits from people who knew him.
For Philadelphia research the Historical Society of Pennsylvania has a large collection of Philadelphia records.