Corroborated event · 4 outlets
Google, OpenAI, Microsoft and Anthropic said they will adopt a new common robots.txt signal that lets website operators block scraping for AI training.
AI brief
OpenAI said it will honor a shared robots.txt signal that lets website operators communicate their AI training preferences to participating providers.
Why it matters: It gives site operators a common mechanism to signal whether their content may be used for AI training, adopted by participating providers.
Written by AI from OpenAI News's published text.
Background
From our own funding, model and news trackers, not from the source.
OpenAI
- Latest funding
- $30B · 31 Oct 2026
- Newest model on the API
- GPT-6.1 Terra Pro · 26 Oct 2026 · $1 in / $5 out per million tokens
More coverage of this story (3)
Google, OpenAI, Microsoft and Anthropic back new scraping opt-out standard
Google, Anthropic and others to support new AI crawling opt-out standard
SiliconANGLE AI · 31 Oct 2026, 02:00
Google joins OpenAI and Anthropic in promising new scraping controls
Engadget AI · 30 Oct 2026, 22:45
The biggest names in AI are backing a new way to opt out of scraping
The Verge AI · 30 Oct 2026, 20:26
Primary source
“OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.”
Published by the organisation it describes; treat claims as first-party statements.
Read original story ↗