AI Terms / for AI companies

For AI companies and crawler operators

Your crawler or agent asked a site for a page and got 402 Payment Required with a price. This page says what the site asked for, how to read its terms, how your crawler is recognised, how to pay, and who to write to.

What the site is asking

The site uses AIPayGen to answer AI crawlers with a price instead of a free page. Each site's owner sets its own price and terms (on the marketplace pages of aipaygen.com itself, we do). We don't run the sites. If you don't pay, nothing is charged and you don't get the page; people and search engines still read the site free.

The 402 carries the quote in the PAYMENT-REQUIRED header (x402 v2: price, USDC on Base, the owner's wallet), the same quote and a short help note in the body, and Link: <…>; rel="ai-terms" pointing to the site's terms.

Read the terms

Check a page before you fetch it (Node 18 or later):

Code6 lines
// npm install ai-terms
import { fetchTerms, resolve, allowedWithin } from "ai-terms";

const terms = await fetchTerms("https://example.com");   // null when the site publishes none
const rule = terms && resolve(terms, "training", "/blog/post");
if (rule && !allowedWithin(rule, 0.01)) console.log("skip: not allowed at my price", rule);

How your crawler is recognised

Which crawlers can be checked

OperatorCrawler names checked by addressIts published list
OpenAIGPTBotopenai.com/gptbot.json
OpenAIOAI-SearchBotopenai.com/searchbot.json
OpenAIChatGPT-Useropenai.com/chatgpt-user.json
PerplexityPerplexityBotwww.perplexity.ai/perplexitybot.json
PerplexityPerplexity-Userwww.perplexity.ai/perplexity-user.json
DuckDuckGoDuckAssistBotduckduckgo.com/duckassistbot.json
Common CrawlCCBotindex.commoncrawl.org/ccbot.json
AnthropicClaudeBot, Claude-User, Claude-SearchBot, anthropic-aiclaude.com/crawling/bots.json
MistralMistralAI-Usermistral.ai/mistralai-user-ips.json
AmazonAmazonbotdeveloper.amazon.com/amazonbot/ip-addresses/

13 AI crawler names, from 10 lists their operators publish. A request using one of these names from outside its operator's list is counted as "(fake)", and pay-per-crawl treats a request from these addresses as an AI crawler whatever name it sends. Any other AI crawler name is recognised by name only.

OperatorSearch crawlers confirmed by addressIts published list
GoogleGooglebot, GoogleOther, googleother-image, googleother-videodevelopers.google.com/static/search/apis/ipranges/googlebot.json
Microsoftbingbotwww.bing.com/toolbox/bingbot.json
AppleApplebotsearch.developer.apple.com/applebot.json

These lists are used only to confirm search crawlers (for example by the bot check). Pay-per-crawl never charges by them, because the same addresses fetch pages for search. A name that shares them, such as GoogleOther, is confirmed here but can't be told apart from search by its address.

What being verified gets you

A site may give each verified crawler a free allowance: a number of pages per UTC hour or day before the price applies. It is in the site's terms ("free": { "requests": 100, "per": "day" }). Only verified crawlers get it, and only where the site's setup applies it: the Cloudflare Worker, Express, Next.js, Netlify, Vercel and Django setups and the WordPress plugin (2.4.0 and later); not the Nginx, Apache or Caddy setups. Those that apply it count in each server process separately, so a site running several may give up to that many times the allowance, never less.

Sites that charge only for training let AI search and assistant fetchers in free by name (OAI-SearchBot, Claude-SearchBot, PerplexityBot, PetalBot, Timpibot, YouBot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User, ElevenlabsBot).

How an AI company pays a site

There are two ways: pay the site yourself with x402, or let AIPayGen buy the page for you from credit you buy (card or USDC top-up; free trial credit can't pay sites). Try the first step on our demo site, a real site using pay-per-crawl, set to charge only training crawlers. The commands below only ask for the price; nothing is paid.

1. Pay the site yourself (x402)

Ask for the page as your crawler.

curl -i -A "GPTBot/1.2" https://aipaygen-ppc-demo.vercel.app/

The answer is 402 Payment Required, with Link: <https://aipaygen.com/v1/gateway/terms/5b1427a95541>; rel="ai-terms" pointing to the site's AI Terms. The same request as ChatGPT-User gets the page free, because this demo charges only training crawlers. Sent from your machine, this is counted as "GPTBot (fake)" on the demo's report, because your address isn't one OpenAI publishes; that's the check working.

Read the quote. The PAYMENT-REQUIRED header is base64 JSON:

Code2 lines
curl -s -o /dev/null -D - -A "GPTBot/1.2" https://aipaygen-ppc-demo.vercel.app/ \
  | grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decode

In accepts[0]: scheme: "exact", network: "eip155:8453" (Base), asset (the USDC contract), amount: "1000" (6 decimals, so $0.001) and payTo (the site owner's wallet). The JSON body carries the same quote and a short help note.

Sign and ask again. Your x402 client signs a USDC payment of exactly that amount to that wallet and sends the same request again with a PAYMENT-SIGNATURE header (code for this in JavaScript and Python is under "Pay for a page" below). You're charged only if the site answers 2xx; the receipt is the PAYMENT-RESPONSE header (see "What you get back").

The same page is also sold through the AIPayGen gateway at https://aipaygen.com/g/5b1427a95541/, whose 402 also describes the page before you pay.

2. Let AIPayGen pay for you (paid_fetch)

No wallet needed. With an AIPayGen API key and credit you buy (card or USDC top-up; free trial credit can't pay sites), the paid_fetch tool reads the 402, pays the site from that credit and returns the page. You pay the site's price plus 15%, plus $0.001 for the call; a page that turns out to be free costs only the $0.001. You set max_price_usd (default $0.05, at most $1), and nothing is charged when the price is higher.

POST https://aipaygen.com/v1/paid_fetch with Authorization: Bearer <your key> and this body (or the paid_fetch tool over MCP at https://aipaygen.com/mcp). It buys the page from your bought credit.

{ "url": "https://aipaygen-ppc-demo.vercel.app/", "max_price_usd": 0.01 }

A site can keep a payment and not send the page, and nothing can make it pay you back, so buy from sites you trust. When a page's delivery is held to a description and doesn't match, the payment isn't settled and your credit is returned.

Pay for a page

Any x402 v2 client works: it reads the 402, signs a USDC payment on Base for the quoted amount to the quoted wallet, and asks again with a PAYMENT-SIGNATURE header. The money goes from your wallet to the site owner's, settled on Base; we never hold it.

JavaScript

Code17 lines
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

// A wallet with a few dollars of USDC on Base. It needs no ETH: the facilitator pays the gas.
const account = privateKeyToAccount(process.env.CRAWLER_KEY);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});

// On a 402 it signs the quoted payment and asks again for the same page.
// By default (x402 2.29) it won't pay more than $1 in one payment; change it with spendControls.maxAmountPerPayment.
const res = await pay("https://a-site.example/some/page", {
  headers: { "user-agent": "YourBot/1.0 (+https://your.example/bot)", "ai-purpose": "training" },
});
console.log(res.status, res.headers.get("payment-response"), res.headers.get("x-aipaygen-delivery"));

Python

Code15 lines
# pip install "x402[requests,evm]"   (x402 2.x, Python 3.10+)
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.mechanisms.evm.exact import ExactEvmScheme
from x402.http.clients import x402_requests

account = Account.from_key(os.environ["CRAWLER_KEY"])  # USDC on Base; no ETH needed
client = x402ClientSync()
client.register("eip155:8453", ExactEvmScheme(account))
session = x402_requests(client)  # on a 402 it signs the quoted payment and asks again

r = session.get("https://a-site.example/some/page",
                headers={"User-Agent": "YourBot/1.0 (+https://your.example/bot)", "AI-Purpose": "training"})
print(r.status_code, r.headers.get("payment-response"), r.headers.get("x-aipaygen-delivery"))

What you get back, and when you're charged

The pageYour payment is settled only after the site answers with a 2xx and, when your quote described the page, only if what it delivered matches that description. Then you get the page with PAYMENT-RESPONSE (base64 JSON: success, transaction, network): that transaction on Base is your receipt. It also carries x-aipaygen-delivery: matched-quote (checked against the description), not-checked (the quote said so) or not-a-page, the Link to the terms, Content-Usage when the terms imply one, and your AI-Purpose if you sent one.
A fresh quote firstIf your payment answers a quote that had no description of the page, it isn't settled: you get a new 402 (DESCRIPTION_REQUIRED) with the description, to pay against.
The site failsA non-2xx answer, no answer, or a page over 5 MB: not settled, x-aipaygen-charged: false. You may retry with the same payment.
Wrong pageA page that doesn't match its description, or can't be described: 502, not settled, x-aipaygen-charged: false, with x-aipaygen-delivery: mismatch or not-described and the reason in the body.
Settlement failsA new 402 and no page. x-aipaygen-charged: false means no money moved; unknown means the settlement call broke off, so check your wallet before paying again.
Same payment twiceA payment pays for one page once. Sent again after it was used, it is refused (409 PAYMENT_ALREADY_USED, x-aipaygen-charged: false). There is no replay for pages: if a paid response is lost, write to us with the transaction.

Sites on WordPress plugins before 2.6.1 don't pass x-aipaygen-charged or x-aipaygen-delivery on. There, a 2xx with PAYMENT-RESPONSE is the receipt; for any other answer, check your wallet before paying again.

Disputes, labels and opting out

Volume and contracts

For volume or contract terms, email hello@aipaygen.com.