AI Terms / for AI companies
For AI companies and crawler operators
Your crawler or agent asked a site for a page and got 402 Payment Required with a price. This page says what the site asked for, how to read its terms, how your crawler is recognised, how to pay, and who to write to.
What the site is asking
The site uses AIPayGen to answer AI crawlers with a price instead of a free page. Each site's owner sets its own price and terms (on the marketplace pages of aipaygen.com itself, we do). We don't run the sites. If you don't pay, nothing is charged and you don't get the page; people and search engines still read the site free.
The 402 carries the quote in the PAYMENT-REQUIRED header (x402 v2: price, USDC on Base, the owner's wallet), the same quote and a short help note in the body, and Link: <…>; rel="ai-terms" pointing to the site's terms.
Read the terms
- The
Linkwithrel="ai-terms"on the 402, or the site's own/.well-known/ai-terms, gives a JSON file with four uses (search, answer, training, agent), eachallow,paid(with a price) ordeny, sometimes with a free allowance. - Most setups also send the IETF AI preferences header on pages served to AI crawlers, e.g.
Content-Usage: train-ai=n; a priced use is written asnthere, because a preference can't carry a price. - To read the file in code: the JavaScript reader on npm (
npm install ai-terms, MIT, no dependencies; see the reader guide), the format and spec, its JSON Schema, the test vectors and a one-file Python reader (standard library only). AI Terms is a draft, posted at the IETF as an individual Internet-Draft; it is not a standard. - AI-Purpose. You may send
AI-Purpose: search,answer,trainingoragent. Today, sites that use AIPayGen choose the terms by which crawler is asking (its name and address), not by this header: declaring a purpose doesn't change the price. On a paid page the header is sent back to you as part of the receipt.
Check a page before you fetch it (Node 18 or later):
Code6 lines
// npm install ai-terms
import { fetchTerms, resolve, allowedWithin } from "ai-terms";
const terms = await fetchTerms("https://example.com"); // null when the site publishes none
const rule = terms && resolve(terms, "training", "/blog/post");
if (rule && !allowedWithin(rule, 0.01)) console.log("skip: not allowed at my price", rule);How your crawler is recognised
- By name and address. A request whose user agent names a known AI crawler is checked against the IP ranges its company publishes (the lists we use; each crawler's page under /ai-crawlers shows whether one exists). Name and address match: it is verified and counted under its name. A known crawler's name from an address outside its company's list is counted as "<name> (fake)".
- If your company publishes no IP list, your crawler can't be verified by address: it is counted by name, and no free allowance applies to it. To have a published list used, send us its URL.
- Web Bot Auth. Requests signed with Web Bot Auth (an Ed25519 HTTP message signature, with the key directory named in
Signature-Agent) are verified by signature when they reach the AIPayGen gateway (pages bought athttps://aipaygen.com/g/…, and sites whose server passes AI crawlers to it). A signature is used to count and label the request; it doesn't unlock a free allowance. The pay-per-crawl code on a site's own server checks addresses only, and when it passes your paid retry to the gateway it doesn't pass the signature along. Check an address or a signature.
Which crawlers can be checked
| Operator | Crawler names checked by address | Its published list |
|---|---|---|
| OpenAI | GPTBot | openai.com/gptbot.json |
| OpenAI | OAI-SearchBot | openai.com/searchbot.json |
| OpenAI | ChatGPT-User | openai.com/chatgpt-user.json |
| Perplexity | PerplexityBot | www.perplexity.ai/perplexitybot.json |
| Perplexity | Perplexity-User | www.perplexity.ai/perplexity-user.json |
| DuckDuckGo | DuckAssistBot | duckduckgo.com/duckassistbot.json |
| Common Crawl | CCBot | index.commoncrawl.org/ccbot.json |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai | claude.com/crawling/bots.json |
| Mistral | MistralAI-User | mistral.ai/mistralai-user-ips.json |
| Amazon | Amazonbot | developer.amazon.com/amazonbot/ip-addresses/ |
13 AI crawler names, from 10 lists their operators publish. A request using one of these names from outside its operator's list is counted as "(fake)", and pay-per-crawl treats a request from these addresses as an AI crawler whatever name it sends. Any other AI crawler name is recognised by name only.
| Operator | Search crawlers confirmed by address | Its published list |
|---|---|---|
Googlebot, GoogleOther, googleother-image, googleother-video | developers.google.com/static/search/apis/ipranges/googlebot.json | |
| Microsoft | bingbot | www.bing.com/toolbox/bingbot.json |
| Apple | Applebot | search.developer.apple.com/applebot.json |
These lists are used only to confirm search crawlers (for example by the bot check). Pay-per-crawl never charges by them, because the same addresses fetch pages for search. A name that shares them, such as GoogleOther, is confirmed here but can't be told apart from search by its address.
What being verified gets you
A site may give each verified crawler a free allowance: a number of pages per UTC hour or day before the price applies. It is in the site's terms ("free": { "requests": 100, "per": "day" }). Only verified crawlers get it, and only where the site's setup applies it: the Cloudflare Worker, Express, Next.js, Netlify, Vercel and Django setups and the WordPress plugin (2.4.0 and later); not the Nginx, Apache or Caddy setups. Those that apply it count in each server process separately, so a site running several may give up to that many times the allowance, never less.
Sites that charge only for training let AI search and assistant fetchers in free by name (OAI-SearchBot, Claude-SearchBot, PerplexityBot, PetalBot, Timpibot, YouBot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User, ElevenlabsBot).
How an AI company pays a site
There are two ways: pay the site yourself with x402, or let AIPayGen buy the page for you from credit you buy (card or USDC top-up; free trial credit can't pay sites). Try the first step on our demo site, a real site using pay-per-crawl, set to charge only training crawlers. The commands below only ask for the price; nothing is paid.
1. Pay the site yourself (x402)
Ask for the page as your crawler.
curl -i -A "GPTBot/1.2" https://aipaygen-ppc-demo.vercel.app/The answer is 402 Payment Required, with Link: <https://aipaygen.com/v1/gateway/terms/5b1427a95541>; rel="ai-terms" pointing to the site's AI Terms. The same request as ChatGPT-User gets the page free, because this demo charges only training crawlers. Sent from your machine, this is counted as "GPTBot (fake)" on the demo's report, because your address isn't one OpenAI publishes; that's the check working.
Read the quote. The PAYMENT-REQUIRED header is base64 JSON:
Code2 lines
curl -s -o /dev/null -D - -A "GPTBot/1.2" https://aipaygen-ppc-demo.vercel.app/ \
| grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decodeIn accepts[0]: scheme: "exact", network: "eip155:8453" (Base), asset (the USDC contract), amount: "1000" (6 decimals, so $0.001) and payTo (the site owner's wallet). The JSON body carries the same quote and a short help note.
Sign and ask again. Your x402 client signs a USDC payment of exactly that amount to that wallet and sends the same request again with a PAYMENT-SIGNATURE header (code for this in JavaScript and Python is under "Pay for a page" below). You're charged only if the site answers 2xx; the receipt is the PAYMENT-RESPONSE header (see "What you get back").
The same page is also sold through the AIPayGen gateway at https://aipaygen.com/g/5b1427a95541/, whose 402 also describes the page before you pay.
2. Let AIPayGen pay for you (paid_fetch)
No wallet needed. With an AIPayGen API key and credit you buy (card or USDC top-up; free trial credit can't pay sites), the paid_fetch tool reads the 402, pays the site from that credit and returns the page. You pay the site's price plus 15%, plus $0.001 for the call; a page that turns out to be free costs only the $0.001. You set max_price_usd (default $0.05, at most $1), and nothing is charged when the price is higher.
POST https://aipaygen.com/v1/paid_fetch with Authorization: Bearer <your key> and this body (or the paid_fetch tool over MCP at https://aipaygen.com/mcp). It buys the page from your bought credit.
{ "url": "https://aipaygen-ppc-demo.vercel.app/", "max_price_usd": 0.01 }A site can keep a payment and not send the page, and nothing can make it pay you back, so buy from sites you trust. When a page's delivery is held to a description and doesn't match, the payment isn't settled and your credit is returned.
Pay for a page
Any x402 v2 client works: it reads the 402, signs a USDC payment on Base for the quoted amount to the quoted wallet, and asks again with a PAYMENT-SIGNATURE header. The money goes from your wallet to the site owner's, settled on Base; we never hold it.
JavaScript
Code17 lines
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";
// A wallet with a few dollars of USDC on Base. It needs no ETH: the facilitator pays the gas.
const account = privateKeyToAccount(process.env.CRAWLER_KEY);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});
// On a 402 it signs the quoted payment and asks again for the same page.
// By default (x402 2.29) it won't pay more than $1 in one payment; change it with spendControls.maxAmountPerPayment.
const res = await pay("https://a-site.example/some/page", {
headers: { "user-agent": "YourBot/1.0 (+https://your.example/bot)", "ai-purpose": "training" },
});
console.log(res.status, res.headers.get("payment-response"), res.headers.get("x-aipaygen-delivery"));Python
Code15 lines
# pip install "x402[requests,evm]" (x402 2.x, Python 3.10+)
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.mechanisms.evm.exact import ExactEvmScheme
from x402.http.clients import x402_requests
account = Account.from_key(os.environ["CRAWLER_KEY"]) # USDC on Base; no ETH needed
client = x402ClientSync()
client.register("eip155:8453", ExactEvmScheme(account))
session = x402_requests(client) # on a 402 it signs the quoted payment and asks again
r = session.get("https://a-site.example/some/page",
headers={"User-Agent": "YourBot/1.0 (+https://your.example/bot)", "AI-Purpose": "training"})
print(r.status_code, r.headers.get("payment-response"), r.headers.get("x-aipaygen-delivery"))What you get back, and when you're charged
| The page | Your payment is settled only after the site answers with a 2xx and, when your quote described the page, only if what it delivered matches that description. Then you get the page with PAYMENT-RESPONSE (base64 JSON: success, transaction, network): that transaction on Base is your receipt. It also carries x-aipaygen-delivery: matched-quote (checked against the description), not-checked (the quote said so) or not-a-page, the Link to the terms, Content-Usage when the terms imply one, and your AI-Purpose if you sent one. |
|---|---|
| A fresh quote first | If your payment answers a quote that had no description of the page, it isn't settled: you get a new 402 (DESCRIPTION_REQUIRED) with the description, to pay against. |
| The site fails | A non-2xx answer, no answer, or a page over 5 MB: not settled, x-aipaygen-charged: false. You may retry with the same payment. |
| Wrong page | A page that doesn't match its description, or can't be described: 502, not settled, x-aipaygen-charged: false, with x-aipaygen-delivery: mismatch or not-described and the reason in the body. |
| Settlement fails | A new 402 and no page. x-aipaygen-charged: false means no money moved; unknown means the settlement call broke off, so check your wallet before paying again. |
| Same payment twice | A payment pays for one page once. Sent again after it was used, it is refused (409 PAYMENT_ALREADY_USED, x-aipaygen-charged: false). There is no replay for pages: if a paid response is lost, write to us with the transaction. |
Sites on WordPress plugins before 2.6.1 don't pass x-aipaygen-charged or x-aipaygen-delivery on. There, a 2xx with PAYMENT-RESPONSE is the receipt; for any other answer, check your wallet before paying again.
Disputes, labels and opting out
- A charge you think is wrong: email hello@aipaygen.com with the transaction hash, the page, and the time. The price itself is the site owner's; we can tell you what the gateway did.
- Your crawler is labelled "(fake)" or not recognised: email us the URL of your company's published IP list, or your Web Bot Auth key directory.
- Not paying: a 402 costs nothing. To stay out, don't fetch the page, or follow the site's robots.txt and terms. Pricing and blocking are each site owner's choice, so there is no AIPayGen-wide switch.
Volume and contracts
For volume or contract terms, email hello@aipaygen.com.