CrawlWitness signal reader

User-Agent: CrawlWitness/1.0 (+https://api.crawlwitness.com/bot)

CrawlWitness reads the AI-consent signals a site publishes, so that AI companies can respect them and publishers can prove what they declared. It is not a content crawler: it never fetches articles, images or other pages.

What it fetches

• /robots.txt, /llms.txt, /.well-known/tdmrep.json and RSL licenses linked from them
• the home page, only for its HTTP headers and meta tags (noai, TDMRep, AIPREF, Content-Signal)

At most once a day per domain for the notarized history, plus on-demand checks when a customer asks about that domain. Bodies are capped at 1 MiB, redirects are limited and only ports 80/443 are used.

Verify it

Request signing (Web Bot Auth) is not enabled on this instance.

Stop it

Name us in robots.txt: with User-agent: CrawlWitness and Disallow: / we read nothing but robots.txt itself (we have to read it to know your choice); narrower Disallow rules are honored path by path. Rules for User-agent: * are what we report to AI companies as your general policy, so they do not stop the daily read of these few policy files.