Set the terms AI crawlers read your site on.
robots.txt gives each crawler one answer: yes or no. Here you set each use: let AI search and answers in free and price training and agents, or price every use. People and Google keep reading free.
Start with the free report: which AI crawlers visit, which are fakes or in disguise, and what they would have paid. Crawler payments are new: none has paid us in the last 30 days, as our live log shows.
Free, about a minute, no wallet needed.
- 1Tell us your site and email
- 2Add our plugin or snippet
- 3See which AI crawlers read it
Or check first: free, 5 seconds, no signup. Which AI crawlers your site lets in today. See the report live on our site.
Start free in 5 minutes
No wallet needed. See which AI crawlers visit and what they would have paid; turn charging on later from the same install. See what the report shows, with our own site.
- Who reads your siteEvery AI crawler, how many pages and which ones.
- Who is fakingCrawler names checked against the addresses their companies publish.
- What it would earnAt your price, before you charge anything.
- Proof to act onNumbers to decide what to block or price, or to bring to an AI company asking for a license.
Works where your site runs
Pick your platform for a step-by-step guide with your route filled in.
Which AI crawlers can read your site?
From your robots.txt. To see who actually came, check your server log in your browser.
Questions
Won't AI crawlers just hide?
Some can, and here is exactly which. 10 address lists that AI companies publish cover 13 crawler names (see which crawlers can be checked): requests from those addresses are treated as AI crawlers even when they claim to be a browser (asked to pay where the site charges), and a request using one of those names from anywhere else is labelled fake in your report. A crawler whose operator publishes no list, sending a browser user agent from home or office addresses, looks like a person and gets through, as it would past robots.txt. The ledger shows aipaygen.com's numbers live, and the other sites running pay-per-crawl combined once enough of them report.
Nobody pays yet. Why bother?
True for now: no page on aipaygen.com was paid for in the last 30 days, as our live log shows. Paying needs a crawler with an x402 wallet, or an agent buying through paid_fetch, which pays your price from prepaid credit. Until then, report-only mode costs nothing and needs no wallet: it shows which AI crawlers read your site, how often, and which fake their name, and you can turn charging on later.
Why not just block AI crawlers?
Blocking every AI crawler also keeps you out of AI search and AI answers, so you stop being cited. Pricing per use lets AI search and answers read free while training and agents pay your price, or are refused. Your AI Terms say which use gets which, and "training only" mode makes the common split in one setting.
Will it hurt my search ranking?
No. People and search engines browse free: only AI crawlers are asked to pay: the 27 names on the list, and requests from addresses AI companies publish. Googlebot and Bingbot aren't on it. Google's and Microsoft's published addresses are used only to confirm their search crawlers, never to charge, because those same addresses fetch pages for search. Pay-per-crawl never edits your robots.txt.
How does it work?
- An AI crawler asks for a page. Your site passes it to the AIPayGen gateway on the same URL.
- The gateway answers
402 Payment Requiredwith your price and your wallet address. - The crawler signs a USDC payment and asks again. The gateway checks the payment, fetches the page from your site with a secret header, and settles the payment on Base.
- The crawler gets the page; the money lands in your wallet. If your site doesn't answer with a 2xx, nobody is charged.
Your price is listed in the AIPayGen marketplace and in x402 discovery, so agents can find your content and budget for it. Crawlers that don't pay just don't get the page, the same as a block in robots.txt, but now there's a way for them to say yes.
Which crawlers are asked to pay?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, GoogleOther, CCBot, Bytespider, Amazonbot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, YouBot, MistralAI-User, DuckAssistBot, PetalBot, Timpibot, ImagesiftBot, omgili, AI2Bot, ElevenlabsCrawler, ElevenlabsBot. Edit the list in the snippet to suit you. What each crawler is.
Google and Apple AI training can't be charged this way. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: Googlebot and Applebot fetch the pages, and the token only says whether Google's and Apple's AI may train on them. Charging by user agent would also charge Googlebot and drop you from search, so pay-per-crawl leaves them alone. To opt out of their AI training, add User-agent: Google-Extended / Disallow: / (and the same for Applebot-Extended) to robots.txt; your search ranking isn't affected. GoogleOther, Google's separate research crawler, does send its own user agent and is on the list.
What does it cost?
Nothing from us. The payment goes from the crawler's wallet to yours; the x402 facilitator pays the network fee.
How does this compare to Cloudflare?
On Cloudflare already? Here's how this compares to Cloudflare Pay Per Crawl and Bot Management, and a live demo on Vercel.
Does it work? Show me numbers.
An AI training crawler was 58% of our traffic; we charged it $0.001 a page and it stopped. The case study, and the live log.
Can I state my terms in a machine-readable format?
Yes: every route gets an AI Terms file generated from its settings (what AI search, answers, training and agents may do, and the price), linked from its 402 answers.
State your terms too. Publish an RSL license to state your terms in a standard format; pay-per-crawl collects the price it names.
Set it up from the command line
1. Get a free API key with curl -X POST https://aipaygen.com/v1/keys -H 'content-type: application/json' -d '{"for":"site"}', then register your site as a gateway route. Prices run from $0.001 to $10 a page.
curl -X POST https://aipaygen.com/v1/gateway \
-H "Authorization: Bearer $AIPAYGEN_KEY" -H "content-type: application/json" \
-d '{"name": "example.com pages","upstream": "https://example.com","price_usd": 0.0005,"pay_to":"0xYourWallet"}'
# -> {"id":"3f9a1c0b7d2e","url":"https://aipaygen.com/g/3f9a1c0b7d2e","gateway_secret":"...",...}
# Buyers call https://aipaygen.com/g/3f9a1c0b7d2e/forecast?city=Austin and pay per call.
Keep the gateway_secret from the answer; it's shown once.
2. Put the snippet in front of your site. The code for each platform, with your route filled in, is in its setup guide: Cloudflare Worker · Express · Nginx · Apache · Caddy · Next.js · Netlify · Django · WordPress.
3. Check it. curl -A GPTBot https://example.com/ should answer 402 with your price; a normal browser still gets the page. See earnings with GET https://aipaygen.com/v1/gateway.