AI crawlers / omgili
omgili: what it is and how to block or charge it
omgili is run by Webz.io. Webz.io's crawler, which collects web content for data feeds sold to analytics and AI customers.
Several uses, including AI
- Run by
- Webz.io
- Its name in your logs
omgili- Published addresses
- None, so it can only be recognised by name. Which crawlers publish them
Block omgili in robots.txt
User-agent: omgili
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match omgili in the User-Agent header.
Or charge omgili per page
Instead of blocking it, answer omgili with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is omgili reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.