AI crawlers / meta-externalagent
meta-externalagent: what it is and how to block or charge it
meta-externalagent is run by Meta. Meta's crawler for uses such as training AI models and improving its products.
Collects content for AI training
- Run by
- Meta
- Its name in your logs
meta-externalagent- Published addresses
- None, so it can only be recognised by name. Which crawlers publish them
- Same company
- FacebookBot
Block meta-externalagent in robots.txt
User-agent: meta-externalagent
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match meta-externalagent in the User-Agent header.
Or charge meta-externalagent per page
Instead of blocking it, answer meta-externalagent with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is meta-externalagent reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.