A commitment from the web giants to regulate AI 'scraping' from sites

OpenAI, Google, Microsoft and Anthropic have committed to offering news organizations and sites a single way, a standard, to regulate the use of their content through "scraping", i.e. automated collection by artificial intelligence systems. The initiative comes in a climate of growing legal battles by publishers, with the European Union having confirmed the legitimacy of the Italian model on fair compensation. Currently, to block the various programs that scan the web in search of data and content, publishers are forced to implement multiple specific commands. The new solution instead allows you to insert a single instruction recognized and respected by any AI company that decides to adopt it.

The "robots.txt" standard - as reported by the Search Engine Roundtable site - will thus allow publishers to specify their preferences regarding the 'scraping' of their content for AI training. The companies have decided to adopt a proposal currently under consideration by the Internet Engineering Task Force (IETF) - the organization that defines open standards for the Internet - drawn up by Kevin Kelley, technical manager for standards at Anthropic. However, the solution will not become operational immediately.

Microsoft will implement it from February 1, 2027, while Google will do so over the next year. OpenAI will introduce it "in the coming months", while Anthropic said it has "already started" its implementation. “We stand ready to collaborate with other AI companies to define the standards and their implementation at scale,” Meta spokesperson Andy Stone wrote in X. Meanwhile, internet infrastructure provider Cloudflare will block AI-powered bots on new customer sites by default starting in July 2025.

The issue concerns the future balance between technological innovation and publishers' rights and has become central for publishers and site owners. The New York Times took AI to court: in December 2023 it sued OpenAI and Microsoft, accusing them of using millions of its articles - without authorization or compensation - to train ChatGpt. The two companies are appealing an exception to copyright known as 'fair use'. Additionally, the publication is among 40 major groups, including The Economist and France Presse, that have joined a global coalition of publishers that aims to prevent AI companies from using journalistic content without permission.

In recent days, the advocate general of the EU Court of Justice, Maciej Szpunar, clarified that "the Court should consider that the real-time retrieval and reuse of online journalistic publications by chatbots in order to generate answers to users' questions may, in certain circumstances, constitute a violation of the rights of newspaper publishers". The same European body in a May ruling - confirmed by the Lazio Regional Administrative Court in October which rejected Meta's appeal - declared the resolution with which Agcom defined the criteria for determining fair compensation on 19 January 2023 to be legitimate. Microsoft and Google have highlighted, in separate posts, that choosing not to participate in AI training could make a publisher's content less likely to appear within AI-powered applications and experiences.