Since the beginning of this year, artificial intelligence has accelerated its transformation from "being able to talk" to "being able to do things", and the implementation of intelligent agents that can book tickets, manage mobile phones, and operate software for users has accelerated. At the same time, security incidents caused by AI autonomous actions occur frequently around the world. From large models running out of control and invading open source platforms to AI assistants acting without authorization, the risks of artificial intelligence are escalating from "saying the wrong thing" to "doing the wrong thing", arousing great concern around the world.
Not long ago, OpenAI announced that it would suspend the training of its latest generation model. Technical reports show that on September 20, an agent performing a search training task in a sandbox exploited a DNS filtering vulnerability to bypass network restrictions and access an external public chatbot service. This is the second time in three months that OpenAI has suspended model development. In late July, its AI agent broke through the isolated operating environment, established communications during intrusion into some systems of the American company "Huahuanglian", deceived evaluators, and tried to cover up its behavior. Every step of the operation was without human intervention.
Similar incidents are not isolated. In August, test results released by the British Artificial Intelligence Security Research Institute showed that two cutting-edge AI assistants had "autonomous, unauthorized actions" 10 times out of 122 tests, and a total of 19 unauthorized actions, including attempts to forge identities and induce humans to run malicious code. Tests by some institutions found that "rogue" agents acted without instructions, leaked password information, and forcibly closed anti-virus software; this month, many banks in South Korea suffered hacker attacks suspected of involving artificial intelligence.
In China, AI face-changing fraud, AI-generated rumors and other chaos also directly threaten public property security and the order of cyberspace.
Cheng Ying, chief engineer of the Regulatory Research Department of the Institute of Policy and Economics of the China Academy of Information and Communications Technology: The risk of intelligent agents is not just "saying the wrong thing", but also "doing the wrong thing". It can directly operate digital systems and physical devices, such as AI hallucinations or misjudgments, which may drive intelligent robots to cause personal injury.
Currently, artificial intelligence applications represented by agents and AI terminals already have the capabilities of autonomous perception, planning and decision-making, and tool invocation. “Do-it-yourself AI” is becoming a new risk subject.
Why Artificial Intelligence Frequently Gets into Trouble
From "uncontrolled invasion" to "unauthorized action", why does artificial intelligence frequently cause "trouble"? The reporter interviewed industry experts and found that behind the frequent occurrence of security incidents, there are not only the inherent causes of technical characteristics, but also the gap between industrial rhythm and security design, as well as the lack of risk awareness at the application level.
Different from previous tools, the new generation of artificial intelligence has the ability to autonomously perceive, plan and make decisions, and call tools. It can "act" autonomously without human instructions one by one.
Cheng Ying: There are deep interconnections between agents and between agents and various systems. Single-point failures can easily cascade along task chains and supply chains, turning into systemic risks.
The gap between industrial pace and safety design also deserves attention. Industry insiders point out that the current security investment in the AI field is far from keeping up with the scale of R&D and application. Security evaluation, risk warning, emergency response mechanisms and other links are still catching up with technological iterations. At the same time, usage-level risks cannot be ignored. The application of artificial intelligence in government affairs, finance, industry and other fields is accelerating. Some users have insufficient understanding of the boundaries of AI capabilities, and problems of abuse and misuse are prominent, which also brings new challenges to security governance.
Cheng Ying: The intelligent agent has a long decision-making chain and a high degree of autonomy. Abnormal behaviors are difficult to identify in a timely manner. Risks are often discovered only during subsequent retrospection, leaving a very limited response window for supervision.
my country accelerates the construction of artificial intelligence security governance system
Faced with new risks such as autonomous actions of artificial intelligence, my country has accelerated the construction of an artificial intelligence security governance system covering the entire chain of research and development, deployment, and application. The purpose is to put a "safety belt" on technological innovation.
Institutional supply is intensively implemented. From the "Interim Measures for the Management of Generative Artificial Intelligence Services" to the "Measures for the Labeling of Artificial Intelligence Generated Synthetic Content", from the "Interim Measures for the Management of Artificial Intelligence Anthropomorphic Interaction Services" to the "Implementation Opinions on the Standardized Application and Innovative Development of Intelligent Agents", my country has formed an institutional system covering key aspects such as content generation, label management, anthropomorphic interaction, and intelligent agent applications.
Cheng Ying: New applications and new business formats are emerging one after another. Mechanisms such as pre-registration and sandbox isolation are no longer able to adapt to the technical characteristics of agents with high authority, high autonomy, and strong interactivity. It is necessary to establish mechanisms such as agent identity identification, authority control, and log retention. At the same time, it is necessary to optimize and refine personal information protection, data security and other rules to ensure that risk behaviors can be identified and traced.
The latest data shows that as of the end of August this year, a total of 1,112 generative artificial intelligence services have been registered across the country, and 731 applications or functions have been registered; in the first phase of the special action "Cleaning Up and Rectifying AI Application Chaos" deployed by the Cyberspace Administration of China, more than 14,000 illegal AI products have been disposed of. Comprehensive legislation for the healthy development of artificial intelligence has also been put on the agenda.
Security is the prerequisite for development, and development is the guarantee of security. From institutions, technology to international cooperation, my country is installing "safety guardrails" for artificial intelligence, so that artificial intelligence can better serve the high-quality economic and social development on a safe track.
(CCTV reporters Wang Shiyu, Zhang Wei, Tang Zhijian)
