OpenAI, Google and Microsoft support new standard for websites — The Verge

OpenAI, Google, Microsoft and Anthropic have announced support for a new shared standard that will allow publishers to use a single setting to prevent AI crawlers from collecting website materials for model training. This was reported by The Verge.

A single signal in robots.txt

The standard предусматривает the use of a shared policy token signal in the robots.txt file. This file has long been used to tell web crawlers whether they may index a website for search, archiving or artificial intelligence model training.

Currently, website owners have to define rules separately for the tools of each AI company. The new mechanism is intended to make it possible to apply the same settings to crawlers from several companies at once.

More current news is available on the UA.News Telegram channel Telegram.

When companies will add support

The signal was proposed by Kevin Kelly, Anthropic’s technical lead for standards, within the Internet Engineering Task Force. Anthropic has already introduced support for it. Google and Microsoft plan to add it in 2027, while OpenAI said it intends to implement it over the coming months.

The companies said their AI crawlers will not access websites if they detect the shared signal. Microsoft also plans to provide more detailed settings: publishers will be able to separately prohibit the use of content for AI training or for grounding model responses with verified sources. At the same time, according to Microsoft, such websites will continue to appear in Bing search results.

Amid the active use of AI crawlers, publishers and technology companies are already using other tools to protect content. In particular, the Really Simple Licensing standard makes it possible to charge crawlers, while Cloudflare blocks AI crawlers by default.