AI Is Getting Really Good at Messing With Cybercriminals

Welcome to Kernel Panic!, a weekly newsletter by Lily Hay Newman and Matt Burgess from inside the new world of privacy and digital security. To receive this newsletter in your inbox each week, sign up here.

Even as AI supercharges digital scamming and cybercrime, it’s also being recruited in the fight to protect potential victims. Governments around the world have consistently struggled to address the global crisis of online crime, grappling with the limitations of attempting to crack down on criminals beyond their borders. This has inspired alternative efforts, like using automation to attempt to go after scammers en masse—perhaps by spamming the spammers to squander their resources. Now, AI platforms are turning these types of experiments into real tools in the fight against cybercrime.

For the last two years, the Australian company Apate, named after the Greek goddess of deception, has been building a system that diverts phone scammers onto calls with AI bots that are trained to keep conversations going as long as possible without, of course, ever actually falling for the scam. The goal is to give scammers enough hope that they stay on the phone, while also collecting intelligence about the scams they’re running.

“What we really like to think is that we’re building the perfect victims for scammers,” Dali Kaafar, the founder and CEO of Apate, tells us. “A minute that a scammer is talking to a bot or an agent is a minute where you’re probably saving hundreds, if not thousands of possible people being reached out to by that exact same scammer,” he says, referencing the ability for scammers to use automated dialing tools.

Kaafar says Apate’s platform, which is used by banks and supported by telecom companies, has around 350,000 bots. They not only pick up calls, but also infiltrate scam chat groups online and respond to text messages with similar goals of frustrating scammers while collecting intelligence from them. He says the company has collected more than 250,000 pieces of information about fraudsters in real-time, from scam URLs to money mule accounts and bank details.

The bots have different personalities, language skills, and profiles, Kaafar says, so scammers hopefully don’t detect that they’re conversing with an AI. Like real people, “sometimes [the bots] do have WhatsApp, sometimes they don’t. Sometimes they pick up the phone, sometimes they just actually hang up on the scammer saying, ‘I'll come back to you later,’” he says.

But are they convincing? Kernel Panic tested the tool using a demo version where users play the role of the scammer and talk to one of Apate’s AI “victims.” The personas are designed to express a healthy amount of skepticism but leave enough openings for scammers to continue to try to win them over, a dynamic that we could immediately feel and found intensely frustrating.

Scammers are likely familiar with this type of skeptical interaction, and Kaafar says that Apate calls regularly go on for more than two hours. In our testing, the Apate system was engaged and responsive, and the timing of the conversation felt natural, even with both of us tag-teaming the AI as a friend, “Lucy,” and her financial adviser “Mickey.” We did not succeed at getting the Apate AI system to invest in our cryptocurrency opportunity even after six minutes of diligent effort. (The returns are truly incredible, though, we swear.)

For now, even with Apate’s massive swarm of bots on the case, it’s only a matter of time before you get your next scam call or text. Cybercriminals initiate billions of messages and calls each year, with the most sophisticated operations running physical, industrial scale scamming sites. And while professional scambaiters and initiatives to infiltrate scam operations have done critical deterrent work for years, law enforcement and research efforts haven’t been able to stop the expansion of digital scamming overall.

But AI could create a new era for anti-scam efforts, not just scammers themselves. Better intelligence sharing between cops, social media companies, banks, and other industries is desperately needed and could potentially be aided by AI-driven monitoring and data analysis. Increasingly, too, innovative efforts have been looking to disrupt cybercriminals by exploiting their “psychological vulnerabilities.” In some cases, these psyops have included directly trolling those behind ransomware attacks. As the focus on disrupting scammers and cybercriminals expands, the use of generative AI as a way to keep them preoccupied, wasting resources, and not launching other attacks appears to be increasingly effective.

Take honeypots, for instance. For years, companies and security researchers have deployed false virtual machines to attract hackers, waste their time, gather information, and understand their secret hacking techniques. Mark Vero, a doctoral researcher at the department of computer science at ETH Zurich, says open source honeypot providers have increasingly been incorporating LLMs into their systems to make them appear more realistic.

In recent research, Vero and colleagues found an LLM-powered honey pot would keep AI agents attacking their system “significantly longer” than a honey pot with more predictable behavior. “The agentic attackers are much more convinced by the LLM simulated honeypots and they also mark them as actual honeypots at a much lower rate,” he says. “If these systems are built well enough, then I think it’s quite advantageous for defenders.”