AI crawlers: who they are and how to block or charge them
The 29 AI crawlers and AI user agents that visit websites most, who runs each one, what it collects, and the exact robots.txt lines to block it. Or skip the choice between "free" and "blocked": charge them per page with pay-per-crawl.
How many sites block each one? See the state of AI crawler blocking across 53 well-known sites. How many were asked to pay, and how many paid? See the live AI crawler ledger.
- AI training
- AI search
- Fetches for a person
- Mixed use
AI training crawlers (11)
Copy your pages to train AI models. Blocking them costs you nothing in traffic: training doesn't send visitors. This is the group most sites block or charge.
AI search crawlers (6)
Index your pages so AI search can cite and link you. Blocking them can keep you out of AI answers, which do send some visitors. Most sites allow these.
On-demand fetchers (6)
Fetch a page live when someone asks an assistant about it. These act for a person in the moment. Blocking them means assistants can't read your page when asked.
Mixed-use crawlers (6)
Collect pages for several uses, including AI. Judge these one by one on their pages.
Which crawlers can be checked by address
Anyone can put "GPTBot" in a user agent. These are the crawlers whose operators publish the addresses they crawl from, so a claim can be checked.
| Operator | Crawler names checked by address | Its published list |
|---|---|---|
| OpenAI | GPTBot | openai.com/gptbot.json |
| OpenAI | OAI-SearchBot | openai.com/searchbot.json |
| OpenAI | ChatGPT-User | openai.com/chatgpt-user.json |
| Perplexity | PerplexityBot | www.perplexity.ai/perplexitybot.json |
| Perplexity | Perplexity-User | www.perplexity.ai/perplexity-user.json |
| DuckDuckGo | DuckAssistBot | duckduckgo.com/duckassistbot.json |
| Common Crawl | CCBot | index.commoncrawl.org/ccbot.json |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai | claude.com/crawling/bots.json |
| Mistral | MistralAI-User | mistral.ai/mistralai-user-ips.json |
| Amazon | Amazonbot | developer.amazon.com/amazonbot/ip-addresses/ |
13 AI crawler names, from 10 lists their operators publish. A request using one of these names from outside its operator's list is counted as "(fake)", and pay-per-crawl treats a request from these addresses as an AI crawler whatever name it sends. Any other AI crawler name is recognised by name only.
| Operator | Search crawlers confirmed by address | Its published list |
|---|---|---|
Googlebot, GoogleOther, googleother-image, googleother-video | developers.google.com/static/search/apis/ipranges/googlebot.json | |
| Microsoft | bingbot | www.bing.com/toolbox/bingbot.json |
| Apple | Applebot | search.developer.apple.com/applebot.json |
These lists are used only to confirm search crawlers (for example by the bot check). Pay-per-crawl never charges by them, because the same addresses fetch pages for search. A name that shares them, such as GoogleOther, is confirmed here but can't be told apart from search by its address.
Block all of them
Or pick and choose with the robots.txt generator.
Code30 lines
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: YouBot
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: ElevenlabsCrawler
User-agent: ElevenlabsBot
User-agent: PetalBot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: AI2Bot
Disallow: /Blocking is all or nothing: a blocked crawler also can't cite or link to you. Pay-per-crawl answers AI crawlers with HTTP 402 and a price instead, so the ones that pay get the page and you get paid. Search engines (Googlebot, Bingbot) are not on this list.