For some time, website content being repeatedly crawled by various AI crawlers for model training has been a widespread concern for content publishers. Because OpenAI, Google, Microsoft and Anthropic each use different crawler identifiers (such as GPTBot, Google‑Extended, ClaudeBot, etc.), site operators who want to restrict their content from being used for AI training often had to add rules in robots.txt for each crawler individually, an onerous process prone to omissions.
Around October 30, 2026, OpenAI, Google, Microsoft and Anthropic announced they will jointly support a new unified standard that allows content publishers to signal once to multiple AI companies that they "do not consent to having content used for generative AI training." The announcement drew attention in Chinese tech circles around November 1 after multiple media reports.
How the new standard works
The mechanism was proposed to the Internet Engineering Task Force (IETF) as a draft by Kevin Kelley, Anthropic's technical lead for standards. The core idea is to add a shared "policy token" to an existing robots.txt file on a website.
An example syntax is:
text
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
Here, User-Agent: __usage_ai-training is the unified identifier; Disallow: / indicates a site‑wide prohibition on using content for training, while Allow can specify exceptions for particular directories (for example, press kits). AI companies that recognize this signal will restrict their crawlers from using the indicated content for generative model training.
Unlike the previous approach of configuring rules for each vendor separately, the new standard aims to let publishers "set once, take effect across parties." It is primarily targeted at generative AI training use cases and generally does not affect indexing by traditional search engines (such as Googlebot or Bing crawlers), so sites can still appear in regular search results. Microsoft additionally offers more granular controls to distinguish between "training" and "AI grounding" (i.e., using a source to verify references and reduce hallucinations), and emphasizes that these settings will not affect visibility in Bing search.
Rolling timelines by company
Adoption schedules differ among companies:
- Anthropic has already begun supporting the standard;
- OpenAI plans to implement it within the coming months;
- Microsoft has stated it will support the standard starting February 1, 2027;
- Google plans to support it in early 2027 and will align it with its existing Google‑Extended controls.
Meta has also expressed support for establishing such a common signal but has not announced a timeline. It is important to note that the draft is currently an IETF individual submission (Internet‑Draft) and is not yet an official Internet standard; its practical force depends on voluntary compliance by companies.
Significance and limitations
The initiative is seen as an important response to long‑standing complaints from content owners. Previously, site operators had to continually track each crawler name and maintain individual rules, which imposed a high management burden. A unified signal lowers the operational barrier and helps move the preference of whether content may be used for AI training from bilateral agreements toward a more standardized, machine‑readable public protocol.
However, there are clear limits. First, the signal is not retroactive: it cannot require the deletion or cessation of use of content already crawled and used for training; it only governs new crawls after the policy takes effect. Second, it mainly targets generative training scenarios and does not comprehensively block all crawler access—search indexing and some real‑time retrieval uses typically remain allowed. Finally, the protocol is voluntary, so other AI companies or smaller crawlers that do not participate are not bound by it.
For the content industry, this is a pragmatic process improvement rather than a fundamental resolution of copyright and data usage disputes. Whether the unified opt‑out will expand to more companies, become a broad industry practice, or be tied to paid licensing and content‑use commercial models remains to be seen.
Overall, the joint support from OpenAI, Google, Microsoft and Anthropic for a unified AI training opt‑out standard marks a move toward coordination among leading AI companies on content crawling rules. For site operators, it provides a simpler, single‑declaration option; for the industry, it represents a step from fragmented management toward a standardized signal for data‑use preferences. The real‑world effect will depend on rollout progress and further participation.
▶️ Enjoy this article? Creating content takes real effort!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares are our greatest support! 🙏
For more AI news visit: https://www.airosetta.com