Crawler Observatory

The Crawler Observatory is a monthly report on how AI crawlers treat real websites: which bots arrive, which files they read, and what they receive in return. It turns server-log behaviour into evidence you can act on.

• Which AI crawlers request pages, such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot
• Which files they read first: robots.txt, llms.txt and sitemaps
• Response codes, so blocked or broken pages show up
• The share of requests from search crawlers compared with training crawlers

• Report 01: 16 December
• Then monthly, around the middle of each month
• Each report ships with its method and a data download

Every report states the sample of sites, the log period, how we verify user agents against published operator documentation, and what we exclude. Results describe the sample, not the whole web, and we say so in each edition.

We do not publish figures before the first report is complete, so this page contains no sample data.

• User agents can be spoofed, so we verify them where operators publish verification details.
• Crawl behaviour changes often, so a month of data shows a trend, not a rule.
• Server logs show requests, not whether a page was later cited in an answer.

You may quote short passages and charts with a link back to the report. Each report includes a suggested citation.