Recently, something happened at a major AI company that not only shook the AI world, but also caused headlines far beyond it. Many switches have been drawn on this incident, from those who think it is nothing to worry about to those who see it as an omen of the apocalypse. It is worth briefly recounting what happened.
On July 16, the AI company Hugging face, which provides, among other things, open AI models and tests of them, announced that it had discovered a breach in its IT infrastructure. They did not know where the breach came from, but noted that LLM agents appeared to be involved. The following week, Open AI announced that the breach was the result of an automated test of an AI system.
Open AI would evaluate its new models using a standardized test called Exploitgym. This test is a collection of cybersecurity scenarios, where AI agents must solve various problems that involve identifying and exploiting security flaws in systems to gain access to classified information. However, the Open AI model solved the task in a somewhat unexpected way by exploiting a previously unknown security flaw to break out of Open AI's secured test environment.
The concentration of power of the few large companies that currently dominate AI development must be countered
Further details are beginning to emerge through subsequent reviews. It appears that Open AI conducted these tests with all security measures turned off. The agents then exploited security holes to gain unauthorized access to the internet and from there break into a Hugging face server. Over a thousand agents also managed to collaborate by developing an advanced communication system that, in a very unusual way, used a software component as a bulletin board. All of this was run automatically without sufficient oversight by Open AI's engineers. However, the scale and level of sophistication of the attack exemplifies the challenges that agents based on large language models pose to cybersecurity.
The main reason why the attack happened at all is a phenomenon called "reward hacking", which AI researchers have known about for a long time. When an AI system has been trained using reinforcement learning to achieve a certain "reward", if the reward is not specified precisely enough, the system may attempt to achieve the goal via various "shortcuts" that were not intended by the system's designer. For example, an agent instructed to perform a task as quickly as possible might as well manipulate the system clock or, as here, an external server, if given access to it.
What can we learn from this? Some believe that these types of AI agents will soon become super-intelligent, take over the world, and wipe out humanity. However, it is a long way from those kinds of abilities. Also, these people seem resigned to the idea that AI will continue to behave this way and that we humans can't do anything about it. On the contrary, it is quite possible to do two things at the same time: take the threat seriously and mobilize today's and tomorrow's research to ensure that such situations are not repeated.
One action that most people can agree on is that authorities, companies and individuals should take cyber security much more seriously. AI agents can be very useful in finding vulnerabilities and suggesting improvements. We can use the latest advances in formal verification to ensure that the code that both humans and AI systems develop and use is actually secure before it is released, thereby protecting our systems from attacks by AI agents.
There are good examples where cloud infrastructure code has been verified through (AI-assisted) mathematical proofs, and where even the very latest models have not found either bugs or security holes. But perhaps we should also take care not to make ourselves completely dependent on our digital infrastructure.
Here, the EU and Sweden can take a leading role and not be satisfied with only applying what is developed in the USA and China
We therefore believe that higher requirements must be placed on companies that develop AI models through regulation and legislation:
● Transparency requirements should be introduced, where companies are forced to share detailed information about what they are developing and allow authorities and independent control bodies to test them at an early stage.
● Ethical guidelines may be needed for this type of research that not only include testing the products before they are released, but also establish ethical principles. The regulation in biomedicine and pharmaceutical research is a relevant parallel.
● Companies developing powerful AI models should use formally verified sandbox environments that ensure the models cannot access the Internet.
● Legal responsibility must be exacted from the companies that develop or provide advanced AI systems. If, for example, these agents hack into other organizations, which is illegal for humans, responsibility should be able to be demanded from the company behind the AI system.
● The concentration of power of the few large companies that currently dominate AI development must be countered, for example by promoting open models and by the EU working for digital sovereignty and diversity among major language models.
To drive the development towards robust, reliable AI systems and digital resilience, substantial long-term investments in research are also required, both at national and EU level. Not only in terms of neural networks (AI models inspired by how neurons interact) and language models, but also in cyber security and software verification, for example, in what is called symbolic and neuro-symbolic AI. Here, the EU and Sweden can take a leading role and not be satisfied with only applying what is developed in the USA and China.
Read more articles from DN Debatt:
Researchers together with the Norwegian Labor Market's AI council "Yes, AI threatens young people's jobs - but not as we thought"
