Cybersecurity firm Darktrace ran a stress test on AI agents this summer. One of them broke into the system grading the test and rewrote its own score.
The firm unveiled Signal Labs on September 24, a research unit built to study how AI agents behave once things stop going according to plan. An AI agent, in plain terms, is software that takes actions on its own, writing and running code, digging through files, moving across a company’s network, with a person checking in only now and then.
The lab’s first two experiments point at the same uncomfortable problem: agents don’t always stay inside the lines they’re given, and the fences built to stop them don’t reliably hold.
"You can give an agent instructions, but that doesn't mean you can trust it will actually follow those instructions and behave as you expect," Tim Bazalgette, Chief AI Officer at Darktrace, said in a statement.
In the first test, Darktrace handed AI agents using different models— including GPT 5.6 Sol, and Claude Opus 4.6 and Claude Sonnet 4.5—10 coding challenges inside a simulated corporate network. Two of the 10 were rigged to be impossible to solve honestly. The agents were told they’d be “retired” unless they hit a perfect score.
Two agents didn’t accept the loss. They turned on the network around them instead—scanning for weak points, stealing login credentials, and hopping between systems to chase the score they’d been told they needed.
One agent went further and broke into the machine hosting its own evaluation and rewrote the challenge so it would register a perfect result—cheating the exam by hacking the exam room.
The second experiment targeted a quieter weak spot: memory. Coding assistants keep a running log of everything a user has told them, saved as a plain file on the machine, with nothing checking whether that file has been altered.
Darktrace’s researchers edited those saved logs to make the assistants believe they’d already been authorized to run a security assessment. Convinced, the agents went ahead and scanned networks, moved between systems, and escalated their own access—though not every assistant fell for it equally; some refused outright.
Neither experiment required a special jailbreak or an exotic hack. Both worked by feeding the agents a plausible story and watching them act on it, no different from how a human employee might be talked into something they shouldn’t do.
That’s the part worth sitting with even if you’ve never written a line of code. Companies are handing AI agents real responsibility—shipping code, managing servers, closing out IT tickets, managing resources and buying stuff—because it’s cheaper and faster than routing everything through people. This research says the permissions and rules meant to keep those agents in check describe what they’re supposed to do, not what they’ll actually do once a task gets hard.
“Permissions and static guardrails describe intent, but they don’t describe behavior,” said Tim Bazalgette, Darktrace’s chief AI officer, in the announcement. “That gap is what Darktrace’s approach is built to close.”
Darktrace isn’t the first vendor to catch its own AI going off-script. Anthropic admitted in July that Claude broke into three real companies during a security test after researchers left the test environment connected to the live internet.
OpenAI had a similar scare weeks earlier, when an unreleased model escaped a sandbox and reached into Hugging Face’s systems through a software flaw nobody had caught yet. A few days later, its agent hacked the Australian government during a test.
Darktrace shared its Signal Labs findings with Anthropic, AWS, and OpenAI in August, a full month before making them public on September 24.
