AI-Generated Impersonation on Social Media Leads to Account Bans and Creator Distress

Alicia is used to being impersonated online.

She creates ASMR content on YouTube, Twitch and X-rated subscription sites, and has seen people use her pictures to catfish fans before.

But when her Instagram account was recently banned in May for violating Meta's community standards on integrity, she made a "shocking" discovery.

Someone had used her identity to create AI-generated images and a fake Facebook profile to sell subscription-based content.

"Every photo, I can tell it looks like me, but not quite. It looks like maybe they're using multiple prompts, so like maybe someone else's body with my face," Alicia told triple j hack.

The fake profile had amassed more than 500,000 followers on Facebook and was even verified by Meta.

"The motivation is almost always money."

Alicia appealed the ban of her real account and tried to report the impostor account, but had no luck.

She even asked her followers on X to report the catfish account, but it was still allowed to stay online.

Several creators shared concerns on X that the sophistication of AI could see catfishing of content creators become more common.

"I was told by [Meta's] AI support to make a new account, and get Meta-verified on that account, so then I can access their enhanced support," she said.

"I did that, and it took hours of getting through their AI chatbot to even talk to a real person.

"Once I got to speak to a real person, they told me they would look into it for me … then I basically just got automated emails for the next month."

Alicia used her account to make a living, build an audience and collaborate with brands and other creators.

"Anyone who runs a business on Facebook or Instagram, they know how important it is for just getting eyes on your business and driving sales," she said.

Alicia said she is lucky she could absorb the financial impact, but other users might not be in the same position.

Meta restored Alicia's account and banned the impostor account after being contacted by triple j hack.

"It's against our policies to impersonate another person, and we have removed the Profile that was shared with us," a spokesperson said.

Meta says it uses a mix of automated and human review technology to enforce its policies, and it encourages users to report content they suspect violates standards.

According to Meta, more than 20 million accounts impersonating large creators were removed from its platforms in 2025.

It says reports of impostor accounts related to large creators are down 33 per cent.

A new AI support assistant rolled out in Australia in March, which Meta says is designed to help resolve people's problems from start to finish.

Does Australia need an online complaints ombudsman?

Research from the Telecommunications Industry Ombudsman (TIO) suggests Australians regularly have issues with complaint resolution online.

The research, which is not platform-specific, found 80 per cent of Australians had an issue with a digital platform in the past 12 months.

More than 50 per cent of those issues were related to account access.

Ombudsman Cynthia Gebert told triple j hack people were frustrated by poor customer service.

The TIO doesn't currently have the power to deal with complaints related to digital platforms.

But the ombudsman is pushing to change that.

"We can deal with phone and internet complaints. So if you were blocked, for example, from your Big Pond account, we can help because it's a Telstra product," Ms Gebert said.

She said too often people just give up without a resolution.

"Two in five people in our search never made a complaint and just gave up, and when they did, two in three are left dissatisfied," Ms Gebert told hack.

"I don't know that that's good enough that we do just give up when so much of our lives is online now."