San Francisco AG Demands Apple, Google Remove 'Nudification' Apps

This week, San Francisco’s attorney general, David Chiu, sent cease-and-desist letters, demanding that Apple and Google remove 13 so-called nudification apps from their app stores, Wired reported.

Nudification apps can make it trivially easy to transform ordinary photos of real people into explicit images. The harmful AI tools allow bad actors to remove clothing, change a person’s features, place them in sexualized positions, and swap victims’ faces onto other people’s naked bodies.

Chiu’s letter warned that app stores were violating “California’s laws that prohibit supporting services that create deepfake pornography,” Wired reported.

Talking to Wired, Chiu said his office was “absolutely horrified” by how ubiquitous the nudifying technology has become, victimizing mostly women and children at an alarming scale as more tools became available.

“These images are used to bully, humiliate, and threaten women and girls,” Chiu told Wired. “This industry has a horrific impact on one’s reputation, mental health, loss of autonomy. There have been victims who’ve been suicidal.”

Wired reviewed the letters and confirmed that Chiu asked Google to remove five apps and Apple to remove eight. No apps were named in the report to “avoid pushing people toward them,” Wired said. However, one app had more than a million downloads and advertised features to sexualize images of women or make “free and uncensored” videos, Wired reported.

Chiu told Wired that allowing any such apps to remain in app stores is unacceptable.

“Generating non-consensual intimate images is illegal, harmful, and completely unacceptable,” Chiu said. His office estimated that Apple and Google have likely “made millions of dollars in fees” by ignoring the harmful apps rather than taking stronger actions or developing better detection to avoid profiting off a public nuisance.

In a statement to Ars, Google spokesperson Dan Jackson said the five apps that Chiu flagged were suspended from the Google Play store for violating policies against harmful content.

“Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content,” Jackson said. “When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like ‘nudify’ on our store.”

Late Friday, Apple sent a statement to Ars, saying that “the App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are against our App Review Guidelines and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts from our program. We are in contact with four others that need to address policy violations or risk being removed as well.”

The Grok elephant in the room

Although Apple has removed some apps flagged by researchers, the iPhone maker otherwise was previously notably silent amid calls to police nudification apps more aggressively in its App Store.

App stores’ detection abilities matter, as Wired’s report noted that researchers have recently warned that some harmful apps have gotten better at avoiding app store removals by only promoting face-swapping features while hiding nudifying capabilities. In a May preprint paper, researchers identified 420 apps touted as generic face-swapping tools and tested 155 to see if they could be used to sexualize images. In 70 percent of apps tested, nudification was possible.

There’s also the Grok elephant in the room to consider. This week, xAI filed a lawsuit confirming that it found instances of Grok-generated child sexual abuse materials (CSAM) and other nonconsensual intimate imagery (NCII) targeting adults. To prevent the Grok misuse, xAI sued the user allegedly responsible for prompting Grok to generate the illegal content, but the question remains whether xAI is liable for the outputs.

Ever since the Grok scandal erupted, app stores have faced questions on whether xAI’s chatbot is violating app store policies against harmful content. Back in April, Apple told Senators that it had privately threatened to remove Grok, NBC News reported, but the xAI app remains in the app store today.

So long as it’s possible to use Grok to generate illegal content, app stores will likely continue to face pressure to take action to prevent harmful outputs, just as Chiu said that app stores should be doing with all apps allowing nudification if they want to comply with California deepfake laws.

Although Google’s spokesperson said that Google continuously tests generative AI apps to make sure they have safeguards preventing NCII and CSAM creation, Jackson did not respond to Ars’ request to comment on whether the way that Grok performs today is an exception to Google’s policy.

In his letters to app stores, Chiu did not request Grok removals, but it is clear he is hoping that his demands will help broadly tighten up enforcement in both app stores, starting with removing the apps that his office flagged.

“My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future,” he says. “It’s our hope that these companies will do the right thing—but if they don’t, we will have to consider all of our legal options.”