AI Crawlers & Technical

ClaudeBot: What It Is, What It Crawls and How to Control It

If you read your server logs, you may have seen a user agent called ClaudeBot. It belongs to Anthropic, the company that builds the Claude assistant. Site owners often ask the same things. What does it collect? Is it safe? Should I block it?

This guide answers those questions in plain terms. It also shows the exact robots.txt rules to use, and how to confirm that a visit is genuine.

Key takeaways

• ClaudeBot is Anthropic’s crawler for collecting public web content that may be used for model training.
• Anthropic documents three agents: ClaudeBot, Claude-User and Claude-SearchBot.
• All three respect robots.txt rules, according to Anthropic.
• Blocking ClaudeBot opts you out of training collection, not out of search or user-triggered fetches.
• Check your logs before you decide, and verify suspicious traffic.

What is ClaudeBot?

ClaudeBot is a web crawler operated by Anthropic. It visits public pages and gathers content that the company may use to develop and improve Claude. It behaves like any other crawler. It requests pages, reads the HTML and moves along links.

Anthropic says the crawler follows the instructions in a site’s robots.txt file, and it respects the Crawl-delay directive. It also says the bot avoids pages that sit behind a password or login. Public pages are the only pages in scope.

The three Anthropic agents

Anthropic separates its web activity by purpose. This matters because each agent has a different effect on your site.

Older articles also mention the names anthropic-ai and claude-web. Those names appear in earlier guidance, so keep them in mind when you read old posts. Today, Anthropic’s documentation centres on the three agents above.

Should you block ClaudeBot?

There is no single right answer. It depends on what you value.

Many publishers split the decision. They block the training crawler and allow the search crawler. That choice keeps their pages eligible for citation while declining training use. However, remember that a block in robots.txt is a request, not a lock. Well-behaved crawlers follow it, and others may not.

How to block or allow ClaudeBot

Edit the robots.txt file in your site root. Each agent gets its own group.

To block only the training crawler:

User-agent: ClaudeBot
Disallow: /

To allow the search and user agents explicitly:

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

To slow the crawler instead of blocking it:

User-agent: ClaudeBot
Crawl-delay: 10

After you change the file, load it in a browser to confirm it saved. Then watch your logs for a few days. Crawlers fetch robots.txt often, so changes usually take effect quickly.

What ClaudeBot collects and what it skips

ClaudeBot reads the same public HTML a browser would load. It does not log in, fill forms or solve captchas. Anthropic has said that it follows robots.txt and avoids content behind a login. Therefore, private account pages and paid areas should stay out of reach as long as they require a sign-in.

Still, be careful with pages that look private but are not. A staging site without a password, an old PDF folder or a public search results page can be crawled like any other URL. If you do not want those pages collected, protect them with a password or a noindex rule plus a robots.txt block.

How to spot ClaudeBot in your logs

Search your access log for the string “ClaudeBot”. A typical visit shows a normal GET request for a page, a 200 status and the user agent text. Look at how often it comes, which paths it requests and how many bytes it pulls.

Our free AI crawler log analyzer does this for you. Paste a log sample, and it groups requests by AI bot, status code and path. That view shows whether a crawler hits your important pages or wastes time on filters and duplicates.

How to verify that a visit is real

Anyone can type “ClaudeBot” into a fake user agent. Therefore, a log line alone proves nothing. Anthropic has said that it does not publish a fixed list of IP ranges, because its infrastructure uses shared cloud addresses. So IP allowlists are not a reliable test.

Use these checks instead:

1. Confirm the request behaves like a crawler. It should fetch robots.txt, then follow links politely.
2. Watch for abnormal speed. A real crawler rarely sends hundreds of requests per second.
3. Rate-limit unknown traffic at the CDN or firewall.
4. Contact Anthropic through its published channel if you suspect abuse.

Managing crawl load

Any crawler can add load. Most sites will not notice ClaudeBot, but small servers may. Start by measuring. Count requests per day and compare them with your normal traffic. If the load is high, add a Crawl-delay value first, because that keeps your pages available while it slows the visits.

If that does not help, rate-limit the user agent at your CDN. Additionally, check your cache rules. A well-cached page costs almost nothing to serve, so a crawler hardly matters.

Does blocking ClaudeBot hurt SEO?

No. ClaudeBot is not a search engine crawler for Google or Bing. Blocking it does not change your rankings there. It only affects what Anthropic’s training pipeline can collect.

On the other hand, blocking Claude-SearchBot could reduce how often your pages show up in Claude’s web answers. Because AI search is growing, think about that trade-off before you block every agent.

How ClaudeBot fits into AEO

Answer engine work depends on access. If a crawler cannot read your page, a model cannot cite it. For that reason, review all AI crawlers together instead of one by one. Our guide to ChatGPT search covers the OpenAI side of the same decision, and what is AEO explains the wider strategy.

A simple policy works for most sites. Allow the search and user agents, make a deliberate choice on training crawlers, and review the policy every quarter.

Frequently asked questions