AI crawlers / Claude-User
Claude-User: what it is and how to block or charge it
Claude-User is run by Anthropic. Fetches a page when a Claude user asks Claude to read or look something up on the web.
Fetches pages live when a user asks
- Run by
- Anthropic
- Its name in your logs
Claude-User- Published addresses
- 14 ranges. Check an address
- Same company
- ClaudeBot, Claude-SearchBot, anthropic-ai
Block Claude-User in robots.txt
User-agent: Claude-User
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match Claude-User in the User-Agent header.
Or charge Claude-User per page
Instead of blocking it, answer Claude-User with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is Claude-User reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.