AI Terms / read it in code

Read AI Terms in your code

AI Terms is a file a site publishes at /.well-known/ai-terms that says, for each use of its pages by AI (search, answer, training, agent), whether it is allowed, allowed for a price, or not allowed. Two small open-source readers let a crawler or agent check it before it reads or pays.

JavaScript and TypeScript: ai-terms on npm

npm install ai-terms

Or pin this release: npm install ai-terms@^0.1.0. Version 0.1.0, MIT, no dependencies, Node 18 or later. It is an ES module (use import) with TypeScript types included. npm · source

Code10 lines
import { fetchTerms, resolve, allowedWithin, contentUsage } from "ai-terms";

const terms = await fetchTerms("https://example.com");   // null when the site publishes none
if (terms) {
  const rule = resolve(terms, "agent", "/blog/post");       // { access: "allow" | "paid" | "deny", ... }
  if (!allowedWithin(rule, 0.01)) throw new Error("not allowed at my price");
  console.log(contentUsage(terms, "/blog/post"));           // e.g. "train-ai=n, ai-use=y"
}
// Say what you're doing on every request:
const headers = { "AI-Purpose": "agent" };
FunctionWhat it does
fetchTerms(site, { timeoutMs?, userAgent?, purpose?, fetch? })Reads https://<host>/.well-known/ai-terms. https only, redirects not followed, at most 100,000 bytes read. Returns the terms, or null when there are none or they can't be read.
resolve(terms, use, path?)The rule for one use (search, answer, training or agent) on one path: access, and price, attribution or free when set. The path is compared without its query.
allowedWithin(rule, maxPriceUsd)True when the use is allowed, or paid at or under your cap. Always cap what you pay: a file can name any price.
freeAllowance(rule)The free allowance on a paid or denied use ({ requests, per }), or null. Sites give it to verified crawlers only.
contentUsage(terms, path?)The IETF AI preferences value the terms imply for a path, e.g. "train-ai=n, ai-use=y" ("" when nothing can be stated). prefsFor gives the same as an object.
validate(doc, { origin? })Checks a document against AI Terms 0.1: { valid, errors, warnings }.
parse(text)The terms in a fetched text, or null when it isn't an AI Terms file.
termsLink(linkHeader)The URL in a Link header's rel="ai-terms", or null.

It also exports prefsFor, globMatch, normalizePath, isEndpointUrl, the USES list and the format's types. fetchTerms in a browser needs the site to allow cross-origin reads; everything else runs anywhere.

Python

One file, standard library only: readers/python/ai_terms.py (MIT). Copy it into your project.

Code7 lines
import ai_terms   # copy ai_terms.py into your project: one file, standard library only

terms = ai_terms.fetch("https://example.com")
if terms:
    rule = ai_terms.resolve(terms, "agent", "/blog/post")
    if not ai_terms.allowed_within(rule, max_price_usd=0.01):
        raise SystemExit("not allowed at my price")

Both readers pass the same tests

The published test vectors (validation, path resolution and AI preferences) run against both: 58 of 58 pass. The JSON Schema describes the file. Both readers share an author with the reference implementation, so they are not independent implementations; if you write one, open a pull request on GitHub.

Then pay, if a use is priced

A reader tells you what a site allows and what it costs; it doesn't pay. To pay for a page, see for AI companies.