Draft 0.1 · open for comment
AI Terms: tell AI what it may do with your pages, and what it costs.
AI Terms is a file a website publishes at /.well-known/ai-terms that says, for each use of its content by AI (search, answer, training, agent), whether that use is allowed, allowed for a price, or not allowed. What that means, in plain words.
It adds what robots.txt can't say: the purpose, the price and how to pay, who the crawler really is, and a receipt for every paid read.
What robots.txt can't say
robots.txt answers one question: may this crawler name fetch this path? It has no idea of a use (AI search versus training), no price, no way to check a crawler is who it says, and no record of what was taken. AI Terms keeps robots.txt and adds those four.
Four uses
Each use is allow, paid (with a price per page or request) or deny, with optional attribution. A paid or denied use can start with a free allowance, like "free": { "requests": 100, "per": "day" }: a crawler that reads at a steady pace stays free, and only a burst meets the terms. Only crawlers whose identity checks out get the allowance. Parts of a site can override with paths.
How a read works
- AskThe site publishes its terms per use at /.well-known/ai-terms: what is free, what costs what, what needs attribution.
- ProveThe crawler's identity is checked against its operator's published addresses, and on the AIPayGen gateway also by its signed requests (Web Bot Auth).
- AgreeThe crawler declares its purpose in an AI-Purpose header, and so accepts that use's terms. Sites that use AIPayGen today choose the terms by which crawler is asking (its name and address), not by this header; on a paid page it is sent back as part of the receipt.
- PayPaid uses answer 402 with an x402 quote; the crawler pays the site's wallet directly and gets the page.
- ReceiptThe paid response carries the settled transaction and the declared purpose: a record of licensed use for both sides.
An example file
A site that lets AI search and answers in free with attribution, and charges $0.002 a page for training and agents:
Code59 lines
{
"ai_terms": "0.1",
"site": "https://example.com",
"site_route": "abc123def456",
"updated": "2026-11-16",
"uses": {
"search": {
"access": "allow",
"attribution": "required"
},
"answer": {
"access": "allow",
"attribution": "required"
},
"training": {
"access": "paid",
"price": {
"amount": "0.002",
"currency": "USD",
"per": "page"
}
},
"agent": {
"access": "paid",
"price": {
"amount": "0.002",
"currency": "USD",
"per": "page"
}
}
},
"default": "paid",
"identity": {
"verify": [
"published-ip-ranges"
],
"unverified": "treated-as-claimed"
},
"purpose_header": "AI-Purpose",
"payment": {
"protocol": "x402",
"version": 2,
"endpoint": "https://aipaygen.com/g/abc123def456",
"network": "eip155:8453",
"asset": "USDC"
},
"receipts": "x402 PAYMENT-RESPONSE header: the settled transaction on Base, verifiable on-chain",
"preview": {
"shows": [
"title",
"description",
"excerpt",
"words"
],
"delivery": "per-quote",
"described_quote": "https://aipaygen.com/g/abc123def456{+path}"
},
"spec": "https://aipaygen.com/ai-terms"
}Built on what exists
robots.txt still decides what may be fetched. llms.txt guides what to read. RSL licenses can be linked from the terms. The four uses are also written in the IETF AI preferences vocabulary (train-ai, ai-use, search), as far as it can say them, and sent as a Content-Usage header and as Content-Usage lines in robots.txt. Identity is checked by published IP ranges, and on the AIPayGen gateway also by signed requests (Web Bot Auth). Payment is x402. AI Terms is the file that ties them into one loop a machine can follow.
Make your file
No signup. Pick what each use may do, copy the file, and serve it at /.well-known/ai-terms on your site. Charging needs somewhere to send the payment, so paid uses come from a free AIPayGen route, which generates and serves the file for you.
How to put it on your site Steps for Nginx, Apache, Caddy, Next.js, Vercel, Netlify, GitHub Pages and WordPress.
It goes at /.well-known/ai-terms.
The same terms, for crawlers that don't read AI Terms yet
Add these lines to your robots.txt. Training-only crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended and others) are told to stay out when training isn't allowed; search and on-demand fetchers aren't. The Content-Usage lines are the IETF aipref preference signal; Anthropic says that since 30 October 2026 it has begun recognizing a "no AI training" preference based on this IETF work. The User-agent: __usage_ai-training group states the same training preference as a cross-vendor policy token, proposed to the IETF aipref group in draft-kelley-aipref-training-preference-signal: Allow welcomes training, Disallow doesn't. If your robots.txt already names a crawler in its own group, copy the Content-Usage line into that group too.
And, if you can set response headers, send these on your pages (not on a 402, which carries a price, not the page; a 402 gets the Link only):
Check a file
Enter a site to check the file it serves, or paste a file. Checked against draft 0.1: the JSON Schema plus the rules a schema can't express. Building a reader? Run it against the test vectors.
For agent builders
Before your agent reads or pays for a page, read the site's terms. This is the whole check, with the Python reader (one file, standard library only):
Code10 lines
import ai_terms
terms = ai_terms.fetch("https://example.com") # None if the site publishes none
if terms:
rule = ai_terms.resolve(terms, "agent", "/blog/post")
if not ai_terms.allowed_within(rule, max_price_usd=0.01):
raise SystemExit("not allowed for agents at my price")
# on every request, say what you are doing:
headers = {"AI-Purpose": "agent"}Three rules cover a well-behaved client: don't use a page for a use the site denies, pay before a use that is paid, and cap what you'll pay, because a file can name any price. Writing a reader in another language? Check it against the test vectors and open a pull request; it will be listed.
Use it today
Every site on AIPayGen pay-per-crawl gets its terms generated from its settings at /v1/gateway/terms/<your-route>, linked from its 402 answers, and charged at the edge. The format is open: implement it anywhere, and send comments or pull requests on GitHub.