AI crawlers: who they are and how to block or charge them

The 29 AI crawlers and AI user agents that visit websites most, who runs each one, what it collects, and the exact robots.txt lines to block it. Or skip the choice between "free" and "blocked": charge them per page with pay-per-crawl.

How many sites block each one? See the state of AI crawler blocking across 53 well-known sites. How many were asked to pay, and how many paid? See the live AI crawler ledger.

AI training crawlers (11)

Copy your pages to train AI models. Blocking them costs you nothing in traffic: training doesn't send visitors. This is the group most sites block or charge.

GPTBotOpenAIClaudeBotAnthropicanthropic-aiAnthropicGoogle-ExtendedGoogleCCBotCommon CrawlBytespiderByteDanceApplebot-ExtendedApplemeta-externalagentMetaFacebookBotMetaElevenlabsCrawlerElevenLabsAI2BotAllen Institute for AI

AI search crawlers (6)

Index your pages so AI search can cite and link you. Blocking them can keep you out of AI answers, which do send some visitors. Most sites allow these.

OAI-SearchBotOpenAIClaude-SearchBotAnthropicPerplexityBotPerplexityYouBotYou.comPetalBotHuaweiTimpibotTimpi

On-demand fetchers (6)

Fetch a page live when someone asks an assistant about it. These act for a person in the moment. Blocking them means assistants can't read your page when asked.

ChatGPT-UserOpenAIClaude-UserAnthropicPerplexity-UserPerplexityMistralAI-UserMistral AIDuckAssistBotDuckDuckGoElevenlabsBotElevenLabs

Mixed-use crawlers (6)

Collect pages for several uses, including AI. Judge these one by one on their pages.

GoogleOtherGoogleAmazonbotAmazoncohere-aiCohereDiffbotDiffbotImagesiftBotImageSiftomgiliWebz.io

Which crawlers can be checked by address

Anyone can put "GPTBot" in a user agent. These are the crawlers whose operators publish the addresses they crawl from, so a claim can be checked.

OperatorCrawler names checked by addressIts published list
OpenAIGPTBotopenai.com/gptbot.json
OpenAIOAI-SearchBotopenai.com/searchbot.json
OpenAIChatGPT-Useropenai.com/chatgpt-user.json
PerplexityPerplexityBotwww.perplexity.ai/perplexitybot.json
PerplexityPerplexity-Userwww.perplexity.ai/perplexity-user.json
DuckDuckGoDuckAssistBotduckduckgo.com/duckassistbot.json
Common CrawlCCBotindex.commoncrawl.org/ccbot.json
AnthropicClaudeBot, Claude-User, Claude-SearchBot, anthropic-aiclaude.com/crawling/bots.json
MistralMistralAI-Usermistral.ai/mistralai-user-ips.json
AmazonAmazonbotdeveloper.amazon.com/amazonbot/ip-addresses/

13 AI crawler names, from 10 lists their operators publish. A request using one of these names from outside its operator's list is counted as "(fake)", and pay-per-crawl treats a request from these addresses as an AI crawler whatever name it sends. Any other AI crawler name is recognised by name only.

OperatorSearch crawlers confirmed by addressIts published list
GoogleGooglebot, GoogleOther, googleother-image, googleother-videodevelopers.google.com/static/search/apis/ipranges/googlebot.json
Microsoftbingbotwww.bing.com/toolbox/bingbot.json
AppleApplebotsearch.developer.apple.com/applebot.json

These lists are used only to confirm search crawlers (for example by the bot check). Pay-per-crawl never charges by them, because the same addresses fetch pages for search. A name that shares them, such as GoogleOther, is confirmed here but can't be told apart from search by its address.

Block all of them

Or pick and choose with the robots.txt generator.

Code30 lines
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: YouBot
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: ElevenlabsCrawler
User-agent: ElevenlabsBot
User-agent: PetalBot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: AI2Bot
Disallow: /

Blocking is all or nothing: a blocked crawler also can't cite or link to you. Pay-per-crawl answers AI crawlers with HTTP 402 and a price instead, so the ones that pay get the page and you get paid. Search engines (Googlebot, Bingbot) are not on this list.