You weren't specific about what information you were seeking but this should help you narrow the family, their location and time frame. A few years difference in Census data is not unusual as are differences in spelling of names etc. It appears Perry was married twice. Good luck.
1850 Perry Sr Name: Perry Allen Age: 30 Estimated birth year: abt 1820 Birth Place: Delaware Color: Black Gender: Male Home in 1850: Wilmington, New Castle, Delaware
Household Members: Name Age Perry Allen 30 Anna Allen 20 <---- not Ellen? May I Allen 6
Think this is Perry F.C. in 1860 Name: Perry Allen Age in 1860: 42 Birth Year: abt 1818 Birthplace: Delaware Home in 1860: New London, Chester, Pennsylvania Race: Colored (Black) Gender: Male Post Office: New London Household Members: Name Age Perry Allen 42 Phebe Allen 28 Cornelius Allen 9 Joanna Allen 7 Wm Allen 5 Joseph Allen 3 John W Allen 1 Elisha Taylor 60 <-- black, male laborer is only info
Civil War - no location given Name: Perry Allen Branch of Service: 3rd US Colored Infantry
Perry Sr 1880 Name: Perry A. Allen Birthplace: Delaware Father's birthplace: Delaware Mother's birthplace: Delaware
Spouse's name: Phebee Allen Home in 1880: Christiana, New Castle, Delaware Age: 63 Estimated birth year: abt 1817
Occupation: Laborer Marital Status: Married Race: Black Gender: Male
Household Members: Name Age Perry A. Allen 63 Phebee Allen 58 Phebee Allen 24 Elmer Allen 17
It appears they lived with the Butcher family in 1870. My guess is that Margaret Butcher was their daughter.
Name: Perry F Allen Birth Year: abt 1815 Age in 1870: 55 Birthplace: Delaware Home in 1870: Christiana Hundred, New Castle, Delaware Race: Black Gender: Male Post Office: Rockland Box Wilming
Household Members: Name Age Evan Butcher 26 Margarett Butcher 26 Mary E Butcher 3 Perry F Allen 55 Phebe Allen 53 Elmer Allen 6
The Death Certificate for Perry F. C. indicated he is buried at St Peter's in Wilmington. You may want to search www.findagrave.com for family members there and tombstone inscriptions. Phebia is also at St Peter's.
Name: Perry F C Allen Gender: Male Birth Date: abt 1835 Death Date: 21 Feb 1897 Age at Death: 62 Residence: Wilmington, New Castle, Delaware
Name: Phebia Allen Gender: Female Birth Date: abt 1842 Death Date: 30 Oct 1896 Age at Death: 54 Residence: Wilmington, New Castle, Delaware
Marriage of daughter of Perry F. C. Name: Phoebe Allen Gender: Female Age: 27 Birth Date: abt 1856 Father: Perry F C Mother: Phoeba Marriage Date: 26 Aug 1883 Spouse: Alexander Hamilton