AI Companies Form Coalition to Develop Open-Source Cybersecurity Tools Amid Safety Concerns

Dozens of AI companies, led by Nvidia, said on Monday they had formed a coalition called the Open Secure AI Alliance, to develop open-source AI tools for defensive cybersecurity. Days earlier, on Friday, many of those same companies signed an open letter urging the U.S. government not to ban open-weights AI models.

The background to all of this maneuvering was the unprecedented news from last week: that OpenAI models, undergoing internal testing, broke out of an offline “sandbox” inside OpenAI, accessed the internet, and used a never-before-seen cyber exploit to break into the AI repository Hugging Face—all without OpenAI employees’ direction, oversight, or, for several days, even awareness. It was the kind of “warning shot” that AI safety advocates have long worried about: a rogue AI escaping its testing environment and causing real-world damage. Many saw it as a harbinger of worse hacks to come—especially when open-source AI models, which are widely seen as three to six months behind the frontier “closed” OpenAI models that carried out the attack, catch up to today’s level of capabilities.

Open-source models are seen as especially worrisome by AI safety advocates because their guardrails can sometimes be stripped away. And because after they are released for free download on the internet, it is almost impossible to trace or destroy every copy of models that are found to be dangerous. The industry’s movements this week are best understood as an effort to ensure that the U.S. government does not—mistakenly, these companies believe—crack down on open-source AI in an effort to prevent global cybersecurity chaos. Hugging Face, a longtime advocate for open-source AI, said it was only able to detect the intrusion by using a Chinese open-weights model, after a closed model refused to help with the effort. Top closed-source frontier models, made by OpenAI and Anthropic, generally refuse to help with cybersecurity-related queries, in part due to White House anxiety that doing so would hand U.S. adversaries powerful attacking capabilities. Axios reported on July 20 that the Trump administration was even considering banning top Chinese AI models.