AI crawlers / ClaudeBot

ClaudeBot: what it is and how to block or charge it

ClaudeBot is run by Anthropic. Anthropic's crawler for collecting public web content that may be used to train Claude models.

Collects content for AI training
Run by
Anthropic
Its name in your logs
ClaudeBot
Published addresses
14 ranges. Check an address
Same company
Claude-User, Claude-SearchBot, anthropic-ai

Block ClaudeBot in robots.txt

User-agent: ClaudeBot
Disallow: /

robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match ClaudeBot in the User-Agent header.

Or charge ClaudeBot per page

Instead of blocking it, answer ClaudeBot with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.

Is ClaudeBot reading your site?

See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.