AI-use policy generator

Choose what AI may do with your website. You get the exact lines for robots.txt (per AI bot, Cloudflare Content-Signal and IETF AIPREF Content-Usage), TDMRep, meta tags, HTTP headers and llms.txt. In the EU, a machine-readable reservation like this is what makes an opt-out from text and data mining effective (Art. 4(3) Directive 2019/790). Free, nothing is stored.

Your files

robots.txt

# AI-use policy generated with CrawlWitness (https://api.crawlwitness.com/generator)
# Add these lines to your robots.txt. Search engines keep crawling as usual.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Content-Usage: train-ai=n, search=y
Allow: /

# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: CloudVertexBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-training-data-crawler
User-agent: AI2Bot
User-agent: Diffbot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: PetalBot
User-agent: KimiBot
User-agent: ICC-Crawler
User-agent: Cotoyogi
User-agent: Reflectionbot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: LinerBot
User-agent: FishBot
User-agent: Novellum
User-agent: bigsur.ai
User-agent: NavuBot
User-agent: QualifiedBot
User-agent: atlassian-bot
User-agent: make.com
User-agent: magpie-crawler
User-agent: Awario
User-agent: w4mwnpbXf3MFAbxOkJRw
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: BorderxBot
User-agent: SelectikaScraper
User-agent: CitibotSiteCrawler
User-agent: AdpResearchBot
User-agent: WARDBot
User-agent: ygs-scraper-bot
Disallow: /

.well-known/tdmrep.json

[
  {
    "location": "/",
    "tdm-reservation": 1
  }
]

meta.html

<meta name="tdm-reservation" content="1">
<meta name="robots" content="noai, noimageai">

headers.txt

TDM-Reservation: 1
Content-Usage: train-ai=n, search=y
X-Robots-Tag: noai, noimageai

llms.txt

# example.com

> AI-use policy of example.com. Machine-readable signals: robots.txt (per AI bot, Content-Signal, Content-Usage), /.well-known/tdmrep.json and TDM-Reservation headers.

## AI use

- AI model training: not permitted
- Retrieval / AI answers (RAG): permitted
- AI search indexing: permitted

Verified record of this policy: https://api.crawlwitness.com/verify/example.com