CrawlWitnesswellfound.com: may AI use this website?
What wellfound.com tells AI bots today, read from its robots.txt and the newer AI-consent standards, checked daily and signed by CrawlWitness.
Analysed: 2026-11-13 22:26 UTC · Global rank (Tranco) #18,213
Major AI bots in robots.txt
| Bot | Operator | Access |
|---|---|---|
GPTBot | OpenAI | Partly blocked |
ClaudeBot | Anthropic | Partly blocked |
Google-Extended | Partly blocked | |
Applebot-Extended | Apple | Partly blocked |
meta-externalagent | Meta | Partly blocked |
Bytespider | ByteDance | Partly blocked |
CCBot | Common Crawl | Partly blocked |
OAI-SearchBot | OpenAI | Partly blocked |
PerplexityBot | Perplexity | Partly blocked |
ChatGPT-User | OpenAI | Partly blocked |
Signals found
| AIPREF (IETF) | — |
| Content-Signal | — |
| TDMRep | — |
| RSL 1.0 | — |
| llms.txt | — |
Notarised history
10 versions recorded since 2026-11-08; last change 2026-11-13. Each version is signed and anchored on Base. /v1/historial/wellfound.com
Recent changes
| 2026-11-13 | ChatGPT-User | Allowed → Partly blocked |
| 2026-11-13 | OAI-SearchBot | Allowed → Partly blocked |
| 2026-11-13 | PerplexityBot | Allowed → Partly blocked |
| 2026-11-13 | Bytespider | Allowed → Partly blocked |
| 2026-11-13 | meta-externalagent | Allowed → Partly blocked |
| 2026-11-13 | Applebot-Extended | Allowed → Partly blocked |
| 2026-11-13 | CCBot | Allowed → Partly blocked |
| 2026-11-13 | Google-Extended | Allowed → Partly blocked |
| 2026-11-13 | ClaudeBot | Allowed → Partly blocked |
| 2026-11-13 | GPTBot | Allowed → Partly blocked |
What it means by jurisdiction
EU: commercial text and data mining is allowed unless the rightholder reserves it in a machine-readable way (Art. 4(3) Directive 2019/790). General-purpose AI providers must respect those reservations (AI Act Art. 53), enforceable by the Commission since August 2026.
UK: the data-mining exception covers only non-commercial research. In 2026 the government dropped the proposed opt-out exception, so commercial AI training generally needs a licence.
US: there is no statutory opt-out; whether training is fair use is being decided in court. Signals like robots.txt are evidence of what the owner permitted.
Japan: Article 30-4 broadly allows use for AI training, except where it unreasonably prejudices the rightholder; technical measures such as robots.txt can matter in that assessment.
Information, not legal advice.
Own wellfound.com? Verify it, sign your AI-use declaration and get a dated evidence certificate.
CrawlWitness Publisher · 7 days freeBuilding AI or crawling the web? Check thousands of domains by API, MCP or x402 and pay per call.
API · MCP · x402 · 7 days free