The short answer is that most of the Kent parish registers are available in at least microform (fiche or film) at the Centre for Kentish Studies (CKS) (Tel 01622-694363).However, the registers (or parts of them) for a few parishes still remain in the possession of the local incumbent - Goudhurst was a case in point a few years ago but may have changed now.According to my ageing list, CKS hold copies of Lamberhurst registers for the period 1564-1868, and nearby Cranbrook for 1559-1867.They may hold more now.The other problem is that parts of some registers are missing, e.g. Cranbrook.
Some parish registers and Bishops Transcripts are held at Canterbury and it may be possible to view them by special arrangement - see their website via Genuki's Kent pages.The Society of Genealogists in London holds various transcripts and you can see what they maintain by going to http://www.sog.org.uk/prc/ken.htmlhttp://www.sog.org.uk/prc/ken.html . For example, for Ashford they hold C 1570-1853 M 1570-1856 B 1570-1842, and for Lamberhurst they hold only marriage transcripts for 1564-1837.
Your John Potten may well have died by 1881 but he may have been away from home and was either not recorded or recorded under the name Potter - a not uncommon error but one that makes tracing extremely difficult.
Do not assume that just because Neri Potten was a farmer, his sons wouldn't be labourers.The majority of the Pottens were Ag Labs even if their fathers were farmers.A very high proportion of the population in agricultural type counties were either Ag Labs or Labourers.I do not think there would have been so much stigma attached to their "job description" in those days!
Another route you might try is obtaining Peter's death certificate (provided it was after 1837) - it should tell you where and when he was born.This at present could be an arduous search from the General Register Indexes. "FreeBMD" will eventually solve much of the problem by allowing on-line searches for the period 1837 to 1900.One Hundred million entries need to be transcribed.At present the project is only 25% complete and do not include any John or Peter Pottens at present although other Pottens are recorded.