THE CRUNCH
OpenAI, Google, Microsoft and Anthropic say they will adopt a shared signal that lets publishers opt out of AI scraping across multiple companies at once, rather than setting preferences separately with each provider. The proposal, reported earlier by Search Engine Roundtable, comes from Kevin Kelley, a technical lead for standards at Anthropic, and is being put forward through the Internet Engineering Task Force (IETF), the body that develops internet standards.
The signal, called a policy token, would live in a site's robots.txt file, which has long told crawlers what they may and may not access, whether for search indexing, archiving or AI training. The participating companies say their AI crawlers will restrict access to a site if they encounter the token.
Timing differs by company: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months. Microsoft also plans granular controls letting publishers opt out of AI training or AI grounding, the process of pointing a model at verified sources to reduce hallucinations.
Meta's communications head Andy Stone signalled support, calling a common signal an important step, but did not say whether Meta would honour it. The announcement arrives as publishers contend with a continuing influx of AI crawlers, and alongside other responses such as the Really Simple Licensing standard for charging crawlers and Cloudflare blocking AI crawlers by default.