CrawlWitness

20minutos.es: may AI use this website?

What 20minutos.es tells AI bots today, read from its robots.txt and the newer AI-consent standards, checked daily and signed by CrawlWitness.

Analysed: 2026-11-17 22:29 UTC · Global rank (Tranco) #4,242

AI trainingReserved
TDMRep (reserva art. 4 DSM): no
RAG and AI answersReserved
TDMRep (reserva art. 4 DSM): no
AI searchNo signal

Major AI bots in robots.txt

BotOperatorAccess
GPTBotOpenAIAllowed
ClaudeBotAnthropicAllowed
Google-ExtendedGoogleAllowed
Applebot-ExtendedAppleAllowed
meta-externalagentMetaAllowed
BytespiderByteDanceAllowed
CCBotCommon CrawlAllowed
OAI-SearchBotOpenAIAllowed
PerplexityBotPerplexityAllowed
ChatGPT-UserOpenAIAllowed

Signals found

AIPREF (IETF)—
Content-Signal—
TDMRep✓
RSL 1.0—
llms.txt—

Notarised history

5 versions recorded since 2026-10-31; last change 2026-11-15. Each version is signed and anchored on Base. /v1/historial/20minutos.es

Recent changes

2026-11-17ChatGPT-UserPartly blocked → Allowed
2026-11-17OAI-SearchBotPartly blocked → Allowed
2026-11-17PerplexityBotPartly blocked → Allowed
2026-11-17BytespiderBlocked → Allowed
2026-11-17meta-externalagentBlocked → Allowed
2026-11-17Applebot-ExtendedBlocked → Allowed
2026-11-17CCBotBlocked → Allowed
2026-11-17Google-ExtendedPartly blocked → Allowed
2026-11-17ClaudeBotBlocked → Allowed
2026-11-17GPTBotBlocked → Allowed
2026-11-15ChatGPT-UserAllowed → Partly blocked
2026-11-15OAI-SearchBotAllowed → Partly blocked
2026-11-15PerplexityBotAllowed → Partly blocked
2026-11-15BytespiderAllowed → Blocked
2026-11-15meta-externalagentAllowed → Blocked
2026-11-15Applebot-ExtendedAllowed → Blocked
2026-11-15CCBotAllowed → Blocked
2026-11-15Google-ExtendedAllowed → Partly blocked
2026-11-15ClaudeBotAllowed → Blocked
2026-11-15GPTBotAllowed → Blocked

What it means by jurisdiction

EU: commercial text and data mining is allowed unless the rightholder reserves it in a machine-readable way (Art. 4(3) Directive 2019/790). General-purpose AI providers must respect those reservations (AI Act Art. 53), enforceable by the Commission since August 2026.

UK: the data-mining exception covers only non-commercial research. In 2026 the government dropped the proposed opt-out exception, so commercial AI training generally needs a licence.

US: there is no statutory opt-out; whether training is fair use is being decided in court. Signals like robots.txt are evidence of what the owner permitted.

Japan: Article 30-4 broadly allows use for AI training, except where it unreasonably prejudices the rightholder; technical measures such as robots.txt can matter in that assessment.

Information, not legal advice.

Own 20minutos.es? Verify it, sign your AI-use declaration and get a dated evidence certificate.

CrawlWitness Publisher · 7 days free

Building AI or crawling the web? Check thousands of domains by API, MCP or x402 and pay per call.

API · MCP · x402 · 7 days free

Check another domain