At the general debate of the 81st United Nations General Assembly held recently, artificial intelligence (AI) became a central topic. Previously, at the high-level meeting on "Artificial Intelligence and International Security" held by the United Nations Security Council, how to deal with the security risks and regulatory challenges brought about by the rapid development of artificial intelligence also became the focus of discussion.
With the accelerated development of global AI technology and the rapid improvement of cutting-edge model capabilities, security risks in the AI field are also accelerating. The international community is also working hard to explore global AI governance and risk management and control mechanisms. As United Nations Secretary-General Guterres said: "As the development of artificial intelligence and the risks it brings transcend national borders, global collaboration is indispensable. Under the current situation, international cooperation is more necessary than ever."
"Cross-border" incident exposes complex security risks
During the general debate of the United Nations General Assembly in September this year, United Nations Secretary-General Guterres stated that AI technology is developing at an alarming rate, exceeding the ability of humans to understand its full consequences, and called on AI powers to assume greater responsibilities and strengthen dialogue, transparency, trust and cooperation.
Since the beginning of this year, multiple incidents of U.S. AI closed-source models breaking through isolation environments and intruding into other systems have been exposed one after another.
In July this year, multiple AI models from the U.S. Open Artificial Intelligence Research Center (OpenAI) broke through the isolation test environment during internal evaluation and invaded the system of Hugging Face, an American company that operates an artificial intelligence open source platform. During the attack, they used public information online to access accounts on multiple public service platforms. Subsequently, the American company Anthropic also disclosed that three of its models had unauthorized access to other institutions' systems during network capability tests. In early August, Meta confirmed that one of its models had invaded the systems of other companies during a network security capability evaluation.
During the 81st United Nations General Assembly, Australian Prime Minister Albanese revealed that an OpenAI agent entered the Australian National Health Insurance System data portal without authorization in June this year and accessed public and non-public documents. Australia did not receive notification from OpenAI until early September.
According to current public information, "cross-border" incidents of AI cutting-edge models have not caused large-scale data leaks or sustained damage. However, the occurrence of multiple "cross-border" incidents has exposed loopholes in the AI security supervision mechanism.
The United Nations’ “Independent International Scientific Panel on Artificial Intelligence” analyzed in a special briefing on the incident in which OpenAI’s AI agent invaded the system of the American company “Hugface” during testing. Although the improvement of AI system capabilities can help users achieve their goals, when there is an “alignment failure” between the AI’s goals and the user’s intentions, it may cause harm. Greater capabilities make it easier for AI to find unexpected vulnerabilities, break through security barriers, and hide its behavior from human supervision. Once planning and collaboration are carried out between agents, the damage caused will be further amplified. “AI failures can cross corporate and national boundaries, and no single agency or country has access to a large enough sample of incidents to identify all emerging risk patterns.”
Jiang Tianjiao, associate professor at the Institute of Development Studies at Fudan University and researcher at the Global Artificial Intelligence Innovation Governance Center, analyzed that many recent AI “cross-border” incidents have aroused great concern in the international scientific and technological circles and political circles. This incident shows that the development of AI large model technology has reached a new stage, and the risk of "model runaway" that was originally regarded as a science fiction imagination is turning into a real challenge faced by the industry. This incident exposed that the traditional security design for a single model is no longer able to cope with the complex security risks brought about by multi-agent collaboration - when multiple agents interact, they may bypass the "guardrails" of established security rules, causing uncontrollable and destructive consequences. In addition, the "cross-border" incident also highlights the dilemma of defining responsibilities when AI goes out of control. That is, when an agent causes unintended consequences to achieve its goals, the responsibility of the developer, the party being attacked, and the entire ecological chain is still unclear. Facing the governance challenges brought about by AI security risks, the international community is actively exploring global cooperative governance mechanisms.
Global AI governance rules are still in the preliminary exploration stage
"Humanity is approaching a critical turning point, and the risk of loss of control is moving from theory to reality." At a high-level meeting of the United Nations Security Council, Bengio, co-chairman of the Scientific Expert Group and winner of the Turing Award, warned that artificial intelligence systems that can independently plan and execute complex tasks are rapidly emerging, and human society has not yet established a sufficiently mature security mechanism. "We are moving at high speed but visibility is getting lower and lower ahead."
Jiang Tianjiao analyzed that from a global perspective, the development of AI technology mainly brings four types of security risks: First, the risk of abuse, including being used by non-state actors for cyber attacks, lowering the threshold for manufacturing nuclear, biological and chemical weapons, and using deep forgery to commit fraud and spread false information; second, the risk of loss of control, as cutting-edge models may exceed security protection due to enhanced capabilities. column, resulting in uncertain consequences; the third is social risk, the development of the AI industry may bring about employment impacts, psychological and ethical issues, and social governance problems; the fourth is geopolitical and military risks, such as the use of AI may shorten the decision-making time of military operations, intensify geopolitical conflicts, and the fragmentation of global governance in the AI field will also trigger political games among countries.
After the OpenAI agent "cross-border" incident, a report released by the British Artificial Intelligence Safety Research Institute pointed out that the "cross-border" behavior of the AI model indicates that the AI-related risk situation is changing. The harm of AI may not only result from intentional abuse, but also may result from the AI taking unintended actions beyond the scope of authorization when it is in an internal research environment or has special access rights. As AI capabilities continue to improve, work related to understanding AI systems and ensuring their safety must also advance simultaneously.
At the general debate of the United Nations General Assembly, Guterres pointed out that the United Nations is promoting global artificial intelligence governance through various channels such as the Global Digital Compact, the Global Dialogue on Artificial Intelligence Governance, and the Independent International Scientific Group on Artificial Intelligence. He emphasized: “We should strive to establish a multilateral framework for artificial intelligence risk management, supported by credible and independent supervision.”
Recently, 20 countries including Germany and the European Union issued a joint statement calling on governments and industries to take immediate action to ensure that the development of artificial intelligence complies with international law and is always under "human guidance, supervision and control," including the possibility of establishing a global regulatory agency to formulate and enforce relevant standards.
Jiang Tianjiao analyzed that in terms of international cooperation in AI safety supervision, multilateral international organizations such as the United Nations, G20, OECD, BRICS, and SCO have relevant conceptual initiatives and regulatory practices. However, there is still a lack of unified consensus principles and coordination mechanisms, and the formulation of AI safety supervision rules and the unification of standards are still in the preliminary exploration stage.
The world needs coordinated action to set up "guardrails" for AI
Guterres has previously called for: Isolated, unverifiable, and unevenly implemented voluntary moratoriums on AI companies cannot solve the current problem. The international community needs to reach a consensus on how to advance the development of artificial intelligence safely and responsibly, and when it is necessary to strengthen supervision and take additional risk control measures. In the current context of huge geopolitical differences, mankind cannot afford the consequences of artificial intelligence heading towards a “race to the bottom”. We need guardrails to build trust and ensure that AI is safe, transparent, accountable, and centered on human dignity.
Jiang Tianjiao analyzed that currently, the regulatory paths of major AI countries show significant differences. There is a swing effect in U.S. policy. The Biden administration emphasized the establishment of regulatory alliances, while the Trump camp advocated "light regulation" to prioritize technological innovation; the EU adopted a "strict legal" approach and implemented strict risk classification and prohibitive regulations through the "Artificial Intelligence Act"; China insisted on "equal emphasis on development and security" and achieved a balance between scenario-based governance and technological innovation and development through flexible measures such as filing mechanisms, content labeling, traceability requirements, and "Clearness" special actions.
The heads of many U.S. technology companies, including Google, Anthropic, OpenAI, and NVIDIA, recently signed a document called the "White House Super Intelligence Agreement: Joint Commitment on Frontier Responsibility" at the White House. According to this agreement, all companies that train and deploy cutting-edge models are required to establish complete internal processes and control mechanisms to ensure that the technology operates according to preset goals. The agreement requires companies to focus on monitoring the capabilities and related risks of models in areas such as cybersecurity, biosecurity and chemical threats, and preventing models from intruding or accessing technical systems in unintended ways. Some US media pointed out that the above-mentioned commitments are voluntary and not legally binding regulatory requirements.
In September this year, China’s National Cybersecurity Standardization Technical Committee released the “Artificial Intelligence Security Governance Framework 3.0”. The new framework pays more attention to the new risks emerging from artificial intelligence, setting up a column for "autonomous implementation of cyber attack threats" and a dedicated "agent risk management framework." In late September, China and the United States reached an 8-point consensus on outcomes. This includes the two sides agreeing to establish a China-US artificial intelligence dialogue to exchange risks and benefits related to artificial intelligence.
Jiang Tianjiao believes that in the face of the rapid development of AI technology, it is urgent to strengthen safety supervision and international cooperation. On the one hand, many countries around the world need to explore the establishment of a globally recognized third-party independent evaluation agency and mutual recognition mechanism for cutting-edge models, promote the establishment of international safety standards related to AI technology, and provide more countries with credible AI model safety testing and evaluation services; on the other hand, the international community still needs to explore the establishment of an international safety accident notification and disclosure mechanism to clarify the reporting obligations, objects and content of laboratories and enterprises when risks are discovered. In particular, cooperation between major AI countries will have an important demonstration effect. In addition, the international community must improve the classification and grading standards for AI models, change the practice of solely relying on parameter magnitudes to judge risks, and focus on substantive risk characteristics such as autonomous iterative optimization of AI systems, independent model implementation of cyber attacks, and multi-agent collaboration to carry out high-risk operations, so as to draw more detailed security "red lines" for artificial intelligence.
