Corroborated event · 4 outlets
Google, OpenAI, Microsoft and Anthropic said they will adopt a new common robots.txt signal that lets website operators block scraping for AI training.
AI brief
Google, OpenAI, Anthropic and Microsoft pledged to support a new common signal for robots.txt that blocks scraping for AI training.
Why it matters: A shared opt-out signal would give site owners one standardized way to block AI training crawlers across major providers.
Written by AI from Engadget AI's published text.
Background
From our own funding, model and news trackers, not from the source.
OpenAI
- Latest funding
- $30B · 31 Oct 2026
- Newest model on the API
- GPT-6.1 Terra Pro · 26 Oct 2026 · $1 in / $5 out per million tokens
Anthropic
- Latest funding
- Growth · $65B · 28 May 2026
- Newest model on the API
- Claude Haiku 5.5 · 7 Oct 2026 · $0.1 in / $0.5 out per million tokens
More coverage of this story (3)
Google, OpenAI, Microsoft and Anthropic back new scraping opt-out standard
Google, Anthropic and others to support new AI crawling opt-out standard
SiliconANGLE AI · 31 Oct 2026, 02:00
The biggest names in AI are backing a new way to opt out of scraping
The Verge AI · 30 Oct 2026, 20:26
Making it easier for websites to express AI training preferences
OpenAI News · 30 Oct 2026, 09:30
Primary source
“Google, OpenAI, Anthropic and Microsoft pledge to support a new common signal for robots.txt that blocks scraping for AI training.”
Reporting from a publisher outside the organisation being covered.
Read original story ↗