CrawlWitnessDaily AI-consent report · 2026-11-01
How the world's top websites treat AI bots today, and what changed. (1,542 · Tranco)
15%reserve AI training
16%block GPTBot
15%block ClaudeBot
14%block Google-Extended
17%block CCBot
Recent changes
| github.com | AI training | No signal → No signal |
| github.com | RAG and AI answers | No signal → No signal |
| github.com | AI search | No signal → No signal |
| wikipedia.org | AI training | No signal → No signal |
| wikipedia.org | RAG and AI answers | No signal → No signal |
| wikipedia.org | AI search | No signal → No signal |
Major AI bots in robots.txt
| Bot | Operator | Blocked | Partly blocked |
|---|---|---|---|
GPTBot | OpenAI | 16% | 47% |
ClaudeBot | Anthropic | 15% | 47% |
Google-Extended | 14% | 49% | |
CCBot | Common Crawl | 17% | 47% |
Applebot-Extended | Apple | 13% | 51% |
meta-externalagent | Meta | 14% | 50% |
Bytespider | ByteDance | 17% | 47% |
PerplexityBot | Perplexity | 11% | 50% |
OAI-SearchBot | OpenAI | 7% | 54% |
ChatGPT-User | OpenAI | 9% | 52% |
Signals found
| AIPREF (IETF) | 0% |
| Content-Signal | 4% |
| TDMRep | 1% |
| RSL 1.0 | 0% |
| llms.txt | 15% |
Daily AI-consent report
| reserve AI training | Recent changes | |
|---|---|---|
| 2026-11-01 | 15% | 6 |