Google joins OpenAI and Anthropic in promising new scraping controls

Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley.

As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley's proposal will change that by getting multiple AI companies to recognize and honor a single common signal. Publishers have to set up a new user agent in the robots.txt file that's specifically for AI training, which will look like this:

User-Agent: __usage_ai-training

Disallow: /

Allow: /press-kit/

That will block crawlers specifically meant to gather data for AI training. The "Allow" section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there's a press kit on it. But this new user-agent section doesn't prevent crawling altogether or prevent content from showing up on Google search. For instance, Google will specifically interpret it as a stop sign specifically for the "Google-Extended" user agent, which crawls websites for materials used to train Gemini models.