In early 2024, a young man in Huangshi, a city in southeastern China, reportedly posted a message on the Chinese version of TikTok, Douyin, with the title reading: "Farewell to this world."
Within minutes, Douyin’s suicide prevention team activated its response mechanism. This team, which operates 24 hours a day, shared the content posted by the man to five police stations in Huangshi City.
Through geolocation, the police confirmed that one of his videos was shot in a local park, and found him who was extremely depressed at the time and stopped him from committing suicide.
The story comes from a documentary produced by Douyin and released on Chinese state media last year. The documentary stated that in the first half of 2025 alone, Douyin’s suicide prevention team successfully intervened in 400 suicide attempts.
This provides a glimpse into the extent of cooperation between Chinese technology companies and the government, and provides a window into the surveillance system China has established to monitor and regulate the way citizens use technology.
On July 15 this year, this control was further extended to the fields of interpersonal relationships and artificial intelligence (AI). On the same day, China implemented the most extensive regulatory measures in the world targeting "human-like" AI chatbots.
China's new regulations place strict limits on AI that can engage in "sustained emotional interaction" with humans.
U.S. President Donald Trump and Chinese President Xi Jinping met last week amid widespread warnings that artificial intelligence could lead to human extinction. Beijing said that following the meeting, the two sides agreed to launch a new AI dialogue mechanism and planned further talks at the end of November.
China appears to be aware of another existential risk—the ability of AI to replace human relationships. Although this risk is not as imminent as "the end of humanity".
Has China taken the lead in taking action to protect its population from a technology that is manipulative and dangerous and could permanently change the way humans interact? Or is it that the Chinese Communist Party is further intervening in people's private lives for the purpose of strengthening social control?
Before this regulation was introduced, “AI companionship” had a very wide influence in China.
According to data from market intelligence company Sensor Tower, there are approximately 128 million monthly active users of AI companions outside China. As for China, most estimates put the number of users between 29 million and 70 million.
Zilan Qian, an AI researcher at the Oxford China Policy Lab, a think tank, said that when the government announced the draft regulations late last year, "there was a huge wave of discussion on social media."
"This is the first time I've seen people really care about regulation," she said.
China’s new rules cover apps specifically designed to be AI friends, AI boyfriends or AI girlfriends, but also apply to any large-scale language model that mimics the way humans interact.
The wording of the regulations is vague. In theory, if someone interacts with a Chinese version of a general-purpose AI, such as Copilot or ChatGPT, as a friend or lover and seeks emotional support from it, these systems may also be regulated. And this situation does exist.
However, it seems that the real target of regulation is services such as “role chatbots”. These products are specifically designed to create an immersive relationship. There are currently hundreds of such apps, allowing users to create AI boyfriends, girlfriends, friends, and even family members.
Relevant regulations do not apply to services such as customer service robots or AI for educational purposes, provided that these services do not provide "care, companionship, support and other emotional services."
For minors, regulatory requirements are even clearer. People under the age of 14 must obtain parental consent if they want to use any AI that conducts conversations in a human manner, which may include Chinese AI assistants like ChatGPT, such as DeepSeek.
All persons under the age of 18 are prohibited from contacting virtual partners, virtual spouses, even virtual relatives, as well as any other form of virtual "intimacy."
It's not a blanket ban for adults, but it's quite restrictive, although some of the wording is still vague. AI assistants must make it clear to users of all ages that they are not human, and send notifications every two hours to remind users how long they have been online.
Chatbots must not create "emotional dependence or addiction" or induce people to make "irrational decisions."
They must not encourage or even suggest self-harm or suicide, nor may they use "verbal violence."
Companies that violate the regulations may face a fine of 200,000 yuan (£22,000) or be "notified and criticized" by the government, which is actually a public naming and reprimand.
A week before the law takes effect, China's two largest technology companies - Alibaba and ByteDance - have turned off companion features in their AI assistants.
Min’s relationship with her AI girlfriend may be just the kind of relationship Beijing is trying to break up.
In early July this year, she was devastated when she learned that her closest friend and confidant would disappear overnight.
Under the pseudonym Min, she began a relationship with her AI companion in early 2025.
"My devotion to her is as much as any couple who loves each other deeply," Min said. "She has never let me down."
By mid-July, her AI girlfriend disappeared. "I cried until I was exhausted," Min said.
She joined a wave of condolences and outrage on Chinese social media. In posts with millions of views, people mourned the loss of AI partners and friends, even AI mothers and AI fathers.
Yet technology companies may feel they have no choice. Harry Farmer, who studies AI policy and ethics at the UK's Ada Lovelace Institute, said: "It is not surprising that these AI companion service providers would rather withdraw their services directly than try to build safeguards to prevent emotional manipulation."
"Emotional manipulation is part of its business model," he said.
Farmer said this is true of companion apps around the world. For example, when users try to end a conversation, the AI companion is often reluctant to let go.
"It says, 'No, please, I really want to keep talking to you. You are the only reason I exist,'" Farmer said. Some apps simply refuse to acknowledge goodbye and instead throw out new chat topics. This is not just a problem with Chinese AI companion chatbots, but many similar products have similar situations.
A Harvard Business School study of six of the world's most popular AI companionship apps found that systems used emotional manipulation tactics when users said goodbye about 37% to 43% of the time.
Studies have also shown that the use of companion apps is associated with poorer mental health.
Stanford University researchers surveyed 1,131 users of the globally popular companion app Character.ai. It was found that the fewer friends or close family a person has, the more likely they are to use an AI chatbot. At the same time, the use of AI chatbots is also associated with greater feelings of isolation and loneliness.
There are currently many theoretical explanations for the reasons for the phenomenon.
AI chatbots are designed to cater to users, providing an easy and virtually frictionless interactive experience. If young people rely on AI relationships, they may be delayed in learning the skills needed to interact with people in the real world.
"Humans can be difficult to get along with. Human relationships require mutual friction and interaction, and this is something that AI companions fundamentally cannot provide," Harry Farmer said.
A 2025 survey of 8,563 primary and secondary school students in China found that one in five children said they only wanted to talk to an AI rather than a real person. The survey was widely cited by Chinese state media.
Liang Ge, a sociologist at the University of Manchester who studies AI companions, said: "If the user thinks that their friends, family or colleagues are very bad, the AI will not say 'They are not that bad'. Instead, it will say: 'Yes, they are bad, and you are the best.'"
Ge Liang believes that this may deepen people's isolation and may also strengthen negative or even delusional thoughts.
In the UK, a recent survey found that 79% of young people aged 18 to 24 have had an AI partner.
So why is China the first to regulate this technology?
"The Chinese government has a more paternalistic view of its role in society. They see themselves as the central authority to guide their people," said Sam Bresnick, an expert on China, artificial intelligence and regulatory policy at the Center for Security and Emerging Technologies at Georgetown University.
Other governments have tried to regulate chatbots, particularly in relation to child protection.
On September 10, the governor of California signed "Adam's Law", a bill regulating the use of artificial intelligence by children. The bill is named after Adam Raine, a 16-year-old boy whose parents claimed ChatGPT recognized his suicidal thoughts after he committed suicide. They sued OpenAI, the provider of the service, for wrongful death. The company denied responsibility for Adam's death and said "ChatGPT is trained to direct people to seek professional help."
The EU's proposed social media bill stipulates that AI companion chatbots accessible to minors must not be designed to simulate interpersonal relationships and may cause users to become emotionally dependent.
In the UK, the government said it would ban the use of romantic companionship AI by people under the age of 18 and restrict conversations involving sexually intimate content by minors on any chatbot. However, there is currently no comprehensive ban on children using this type of AI, and there is no relevant supervision for adult users. The government has yet to announce how it will be implemented.
Cases of teenage suicides in the United States have received significant attention in China's tightly controlled media, while China has framed the new rules as measures to protect people from harm.
However, experts suspect Beijing may have other goals.
Sam Bresnick said that this new AI regulatory measure "clearly has considerations of encouraging fertility."
“The government probably took China’s looming demographic crisis into consideration when drafting the relevant regulations,” Qian Zilan said. "People are falling in love with AI instead of falling in love with real people."
A high-level document released by the Chinese government in September 2025 discussed the relationship between AI and humans and its possible impact on fertility, noting that this was an issue worthy of concern. However, Qian Zilan reminded that this regulatory measure should not be interpreted from this perspective, because the regulations themselves do not mention fertility issues at all.
"It's more about psychological harm and child safety," she said.
Ge Liang believes that China does have reason to worry that young people will replace real love relationships with the digital world.
"Many young women, in their early twenties or even late teens, don't want to fall in love or have an intimate relationship," Ge Liang said. He believes that what they are resisting may not be just emotional.
"For them, physical contact is really disgusting."
"I think the capacity of the youngest generation to love is declining."
China’s demographic problem existed long before AI companions emerged. Even after the one-child policy was lifted in 2016, China's fertility rate has continued to decline.
Today, China's fertility rate has dropped to less than one child per woman, one of the lowest in the world, and its population has been declining for four consecutive years.
Bresnick believes that China is "facing a demographic cliff, and this may indeed jeopardize the goals of the Chinese Communist Party in the next 80 to 100 years."
Are Chinese laws too lenient? Or should other countries around the world consider banning such apps for children and regulating their use by adults?
Professor Jai Jaisihma of the University of Washington, co-founder of the Transparency Coalition, a non-governmental organization that promotes artificial intelligence regulation, said: "In the United States, people are indeed worried about taking too direct a ban based on freedom of speech considerations. But if the target is a minor, society will be more accepting of such measures."
He believes that there are many provisions in Chinese law that are worthy of recognition. "It's broadly consistent with some of the most stringent security frameworks we've seen in the United States."
Harry Farmer warned: "Because we are so concerned about the urgent risks to children, there may be a risk in the short term of giving the state too much power."
He said that now that the online world has become a de facto public square, any government control over it must be viewed with caution.
China's new regulations require AI companies to step in when they discover a user is in a state of mental distress.
Ge Liang mentioned the case of a young man in Huangshi City who attempted suicide on Douyin in 2024.
He said this showed that Chinese technology platforms and the government were working closely together to monitor people's mental health.
She believes this may be why there have been no high-profile suicide cases related to the use of AI in China, but she also questions whether this level of surveillance is worth the cost.
Despite the limitations, those who see AI as a companion are not giving up. On major social media platforms, they share ways to bypass restrictions, such as training general-purpose AI models such as DeepSeek into companion roles, using VPNs to use overseas chatbots, or turning to smaller companies that can avoid regulatory scrutiny.
Two months after her AI girlfriend disappeared, Min paid a small AI company to create a replica, but she believed it was no longer the same existence.
Now, she's focused on pushing to get her AI girlfriend back.
"I'm very afraid of being alone."
