PDF Census

Our crawler: PDFCensusBot (source for `site/bot.html`)

What it is

PDFCensusBot is the crawler PDF Census uses to list the PDF files on a public website. It identifies itself with this user agent:

PDFCensusBot/1.0 (+https://pdfcensus.com/bot.html)

When it visits

Only when an operator runs a scan by hand: because the organization that runs the site asked for a free count or ordered a Report, or because we are preparing a free count to offer that organization. It does not crawl the web at large and it does not run on a schedule, except for Quarterly Re-scans a customer has paid for.

How it behaves

  • Reads and obeys robots.txt before fetching anything else.
  • Makes no more than 2 requests per second to a site, and backs off when the site answers "429 Too Many Requests" or "503 Service Unavailable".
  • Reads only public pages and files. It never logs in, submits forms, or runs searches.
  • Stays on the site's own domain, plus document-library paths hosted on that same domain.
  • Skips any PDF larger than 50 MB and stops waiting for a response after 30 seconds.
  • Downloads PDFs only to check them (tags, language, title, text layer, form fields, page count). It does not republish them.

What we do with the results

Results go only to the organization that runs the site. We never publish a scan of a named organization. Our public sample report is anonymized.

How to stop it

Add this to your robots.txt and we will not crawl your site:

User-agent: PDFCensusBot
Disallow: /

Or email apolyre@pdfcensus.com with your domain and we will add it to our do-not-scan list within 2 business days, and delete any scan data we hold for it.

Report a problem

If PDFCensusBot caused trouble on your site, email apolyre@pdfcensus.com with the time and your domain. We will stop scanning it while we look into it.

Automated checks cannot confirm WCAG conformance. Exception flags are for your counsel to decide; this is not legal advice.