Recently, a group of poor boys named "Xiaohai" have appeared in batches on short video platforms. They called each other "aunt and uncle" to the camera, crying about their single parent and embarrassing life experience, which made many middle-aged and elderly people cry and place orders to buy goods and help. There was even a 70-year-old man who took the train alone to visit. However, this "caring junior" who worries the elderly does not exist at all in reality. They are all "digital people" synthesized by AI.
These "fake juniors" generated in batches by AI take advantage of the elderly's desire for companionship and target their wallets. What kind of selling business is behind the tragic drama? Let’s listen to the report of Ren Mengyan, a reporter from China Voice.
In May of this year, while inspecting the carriages, Urumqi Railway Police officers discovered a 79-year-old man traveling alone in a train, looking anxious. The police came forward to inquire and learned that the old man was illiterate and could not tell the specific destination. He only knew that he was going to Yuncheng, Shanxi Province to "visit relatives" and visit a poor child who called her "aunt" on his mobile phone every day.
Old man: I’m going to Yuncheng. The children don't even know I'm here. That's him.
Policeman: Have you seen him?
Old Man: I haven’t gone to him yet. He said in the live broadcast room that there were morels and asked me to eat some and then get some. It turns out we know each other, and she even calls me aunt.
Xiaohai's voice: Auntie, Xiaohai wishes you peace and health...
Old man: You said it again.
Policeman: Auntie, let me tell you, the people in this live broadcast room are fake.
△The policemen tried to dissuade
With the patient persuasion and contact from the police, the old man finally got off the bus and returned home with a return ticket bought by his family. How could an illiterate 70-year-old man take a fictional video on his cell phone screen to be real? The reporter searched on the short video platform and found that what convinced the old man was the "small sea of digital people" that had recently appeared in batches on the Internet. These images of teenage boys generated by AI, without exception, are playing out the painful script of "sensible, embarrassed, and distressed for their elders". In a video around the Mid-Autumn Festival, "Xiao Hai" had tears in her eyes and showed the steamed buns in her hand to the camera:
Digital person "Xiao Hai": Auntie, I am Xiao Hai. My classmates have all gone to eat, and I am hiding in the classroom eating steamed buns. Aunt and aunt, I have delicious steamed buns. I also bought a meat bun for my aunt and mom. Xiaohai wants to give delicious food to her aunt and mom...
△The digital person Xiaohai who eats steamed buns and sells them miserably
Below the video are the sincere voice and text messages from middle-aged and elderly netizens.
Message 1 from an elderly netizen who was deceived: Xiaohai, you are great, you are excellent, and my aunt is proud of you!
Message 2 from an elderly netizen who was deceived: Xiaohai, why don’t you go to dinner and just eat steamed buns? Don’t you have any money? Aunt and mother are so heartbroken. Aunty, mommy misses you.
△Messages from users who believe them to be true
First use false life experiences such as "hiding in the classroom to eat steamed buns" to screen out the most sympathetic and least prepared middle-aged and elderly groups, and then accurately push them to the next step - bringing goods. In another shopping video, "Xiaohai" transformed into a "caring junior" who understands the hearts of the elderly best and began to recommend donkey-hide gelatin.
Digital person "Xiaohai": I have been reading the messages from my aunt and mother these past two days. Many aunts and uncles say that the weather has been hot recently, and people are easily lethargic. They feel tired when they get up in the morning, and they can't sleep well at night. I thought about it for a long time before putting up this donkey hide gelatin cake, because I found that many aunts and aunts are usually too family-oriented, but they always deal with it casually when it is their turn. You usually worry about the whole family and are tired, so occasionally prepare some delicious food for yourself.
△Digital people start selling goods
During the investigation, the reporter found that the threshold for operating such a "Xiaohai" account is extremely low. On the Internet platform, you can buy a complete set of AI digital human generation tutorials for a few dollars, and various marketing templates abound. The same copywriting, with a different face and voice, can generate hundreds or even thousands of "small seas" in batches.
Zhao Jingwu, associate professor at Beihang University School of Law, pointed out that these entities who hide behind the code to manipulate digital people to sell goods at a disadvantage are suspected of committing fraud as long as they maliciously fabricate stories to gain sympathy and sell goods.
Zhao Jingwu, associate professor at Beihang University School of Law: There is no essential difference in legal qualification between digital people selling goods at a high price and real people selling goods at a bad price. Although the content is marked "generated by AI", this does not mean that the elderly should assume that they have been negligent if they were deceived. The label "generated by AI" only explains how the video is generated and the tools used. Judging from the video content and the purpose of carrying goods, it is aimed at the elderly who lack emotional support from their families. The subjective purpose is never used to describe objective facts, and there are even behaviors that maliciously mislead elderly users and win sympathy, which can be regarded as fraud. In this industry chain, the first person responsible is undoubtedly the account operator or live broadcast room operator.
In fact, the "Artificial Intelligence Generated Synthetic Content Labeling Measures" were implemented as early as September last year. Why can't we still control the "small seas" that fission one after another? The reporter noticed that many of the accounts involved did have an “AI generated” watermark in the corner of the video. Since it has a watermark, why can’t it stop the elderly from being deceived? Zhao Jingwu believes that the digital literacy of the elderly is uneven, and coupled with the small size of the characters, it is difficult to notice:
Zhao Jingwu: The problem with not being able to control "Xiaohai" is that the current legislation only stipulates that AI-generated videos must be marked with the "AI-generated" logo, but elderly users often do not care about the watermark logo at the corner of the video. Some elderly users even interpret AI-generated content as "because real events cannot be directly captured, so AI videos can only be used instead of descriptions." This makes many elderly people easily deceived.
Why are some elderly users completely unable to see through the rough AI-synthesized images that young people can see at a glance, and even travel thousands of miles to find someone like the 79-year-old man? Ma Liang, a professor at the School of Government at Peking University, believes that on the surface this is the digital literacy gap among the elderly, but the underlying root cause lies in the "emotional empty nest" problem in society that cannot be ignored.
Ma Liang, professor at Peking University School of Government: On the surface, this issue is a technical issue and a market issue, but in fact it is a social issue behind it. Why do many elderly people trust these AI videos so much that they even pay for them? This reflects that many elderly people live relatively lonely lives offline and do not have effective companionship. They may satisfy their spiritual needs through consumption in the digital world. This also means that relevant social work and community governance departments should follow up, provide more offline care for the elderly, strengthen guidance on AI applications, and avoid being deceived.
In addition to the black and gray industries and merchants who operate accounts, the role played by short video platforms is also questioned. With the same sad words, highly similar digital faces, and frequently mounted product links, why can we always accurately find elderly users? Previously, media reported that the "Digital Human Xiaohai" video has been played more than 600 million times on a short video platform. In this regard, Ma Liang believes that in the face of mass-produced deceptive videos, the responsibility of the platform must not stop at labeling the video, or waiting for netizens to report it before banning the account:
Ma Liang: The traditional "whack-a-mole" governance is more about "you will get a response only if you report it." This kind of responsive governance is very passive and lagging. Behind these videos is a commercial operation, hoping to make huge profits. The platform has the technical ability to identify the comments under the videos and the subsequent shopping behavior of users. This data can be used to identify which videos are fraudulent. It cannot be said that the platform has completed its job by annotating these AI-generated videos.
In this regard, Zhao Jingwu also made it clear that in the face of the emergence of "Xiaohai" accounts in batches, the traditional "notice-delete" rule no longer applies, and the platform's duty of care must be upgraded in advance.
