Corroborated event · 4 outlets
Google, OpenAI, Microsoft and Anthropic said they will adopt a new common robots.txt signal that lets website operators block scraping for AI training.
AI brief
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that lets publishers opt out of having their content scraped for AI use.
Why it matters: The standard would give publishers a shared way to signal their preferences about AI training data across tools, according to the companies.
Written by AI from The Verge AI's published text.
Background
From our own funding, model and news trackers, not from the source.
OpenAI
- Latest funding
- $30B · 31 Oct 2026
- Newest model on the API
- GPT-6.1 Terra Pro · 26 Oct 2026 · $1 in / $5 out per million tokens
Anthropic
- Latest funding
- Growth · $65B · 28 May 2026
- Newest model on the API
- Claude Haiku 5.5 · 7 Oct 2026 · $0.1 in / $0.5 out per million tokens
More coverage of this story (3)
Google, OpenAI, Microsoft and Anthropic back new scraping opt-out standard
Google, Anthropic and others to support new AI crawling opt-out standard
SiliconANGLE AI · 31 Oct 2026, 02:00
Google joins OpenAI and Anthropic in promising new scraping controls
Engadget AI · 30 Oct 2026, 22:45
Making it easier for websites to express AI training preferences
OpenAI News · 30 Oct 2026, 09:30
Primary source
“OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable.”
Reporting from a publisher outside the organisation being covered.
Read original story ↗