AI crawlers / GPTBot
GPTBot: what it is and how to block or charge it
GPTBot is run by OpenAI. OpenAI's crawler for collecting public web content that may be used to train its models.
Collects content for AI training
- Run by
- OpenAI
- Its name in your logs
GPTBot- Published addresses
- 15 ranges. Check an address
- Same company
- OAI-SearchBot, ChatGPT-User
Block GPTBot in robots.txt
User-agent: GPTBot
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match GPTBot in the User-Agent header.
Or charge GPTBot per page
Instead of blocking it, answer GPTBot with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is GPTBot reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.