On a regular Monday morning, you got a video call from a friend. You friend looked so distressed, flustered, and you got really concerned and asked him what happened. Then he’d tell you his 5-year old kid was hit by a car and sent into the ICU. Your friend said he hasn’t got enough money to pay for the treatment. Now he’s crying, and asking if you could lend him money to save his poor son. What would you do?

“Of course I will give him the money to save the boy. He’s my friend and he needs my help.”

If that’s what you’re thinking, think again.

(資料圖片)

Because it could be a scam. The person you saw “flesh and blood” may not be who you think he is. Even though “your friend” was speaking to you on face-time.

Police in the northern Chinese city of Baotou have uncovered a deepfake fraud in which a man was scammed out of 4.3 million yuan, the most so far stolen in China in this way.

According to disclosures by police in Fuzhou in eastern Fujian province, on April 20, a fraudster stole an individual’s WeChat account and used it to make a video call to a businessman named Guo, an existing contact on the individual’s WeChat app.

The con artist asked for Guo"s personal bank account number and then claimed an equivalent sum had been wired to that account, sending him a screenshot of a fraudulent payment record.

Without checking that he had received the money, Guo sent two payments from his company account totaling the amount requested, the police said.

"At the time, I verified the face and voice of the person video-calling me, so I let down my guard," the article quoted Guo as saying.But an impersonator had used face-swap and voice-mimicking artificial intelligence technologies.

The man only realized his mistake after messaging the friend whose identity had been stolen, who had no knowledge of the transaction.Guo alerted local police.

At the request of the Fuzhou authorities, their colleagues in Baotou later intercepted some of the funds at a local bank, but nearly 1 million yuan was unrecoverable. The police’s investigations are ongoing.

The case unleashed discussion on microblogging site Weibo about the threat to online privacy and security, with the hashtag "#AI scams are exploding across the country" gaining more than 180 million views on Tuesday, but it was seemingly removed from the internet amid fears that the case may inspire copycat crimes.

"If photos, voices and videos all can be utilized by scammers," one user wrote, "can information security rules keep up with these people"s techniques?”

A number of similar frauds have occurred around China as AI technology becomes more and more widely applied. The public security departments of Shanghai and Zhejiang province previously disclosed such cases.

In addition to direct scams, some frauds involve live e-commerce platforms where AI technology is used to replace the faces of live streamers, with those of stars and celebrities, to take advantage of their market appeal and fool people into buying goods, raising related issues around fraud and intellectual property rights.

An illegal industrial chain has formed with people using face-swapping technology for online scams, according to a report by China News Service"s financial channel.

A website providing deepfake software services sells a complete set of models that can be used on various live-streaming platforms for only 35,000 yuan, according to its customer service.

Deepfake technology, which has progressed steadily for nearly a decade, has the ability to create talking digital puppets. The software can also create characters out of whole cloth, going beyond traditional editing software and expensive special effects tools used by Hollywood, blurring the line between fact and fiction to an extraordinary degree.

Deepfake has been used in recent years to make a synthetic substitute of Elon Musk that shilled a cryptocurrency scam, to digitally “undress” more than 100,000 women on Telegram and to steal millions of dollars from companies by mimicking their executives’ voices on the phone.

Another example of how eerily accurate their technology is their replication of actor Leonardo Di Caprio speaking at the United Nations.

According to the World Economic Forum (WEF), deepfake videos are increasing at an annual rate of 900%, and recent technological advances have even made it easier to produce them.

Identifying disinformation will only become more difficult, as deepfake technology will become sophisticated enough to build a Hollywood film on a laptop without the need for anything else.

In most of the world, the authorities can’t do much about it. Even as the software grows more sophisticated and accessible, few laws exist to manage its spread.

Many people are starting to get concerned of deepfake and AI-generated content being used for evil. First, this technology can be used for misinformation; for instance, allowing people to believe a politician made a shocking statement that they never did.Or secondly, to scam people, especially the elderly.

In China, AI companies have been developing deepfake tools for more than five years. In a 2017 publicity stunt at a conference, the Chinese speech-recognition specialist iFlytek made deepfake video of the U.S. president at the time, Donald J. Trump, speaking in Mandarin.

But even the AI pioneer now has fallen victim of the same technology it’s been developing. Shares of iFlytek fell 4.26% on Wednesday after a viral screenshot surfaced of what appeared to be a chatbot-generated article that made unsubstantiated claims against the company, fueling public concern about the potential misuse of generative AI.

The potential pitfalls of groundbreaking AI technology have received heightened attention since US-based company OpenAI in November launched ChatGPT.

China has announced ambitious plans to become a global AI leader by 2030, and a slew of tech firms including Baidu, Alibaba, JD.com, NetEase and ByteDance have rushed to develop similar products.

ChatGPT is unavailable in China, but the American software is acquiring a base of Chinese users who use virtual private networks to gain access to it for writing essays and cramming for exams.

But it is also being used for more nefarious purposes.

This month police in the northwestern province of Gansu said "coercive measures" had been taken against a man who used ChatGPT to create a fake news article about a deadly bus crash that was spread widely on social media.

China has been tightening scrutiny of such technology and apps amid a rise in AI-driven fraud, mainly involving the manipulation of voice and facial data, and adopted new rules in January to legally protect victims.

And a draft law proposed in mid-April by China"s internet regulator would require all new AI products to undergo a "security assessment" before being released to the public.Service providers will also be required to verify users’ real identities, as well as providing details about the scale and type of data they use, their basic algorithms and other technical information.

The global buzz surrounding the launch of ChatGPT has seen a spate of AI-related product launches in China. However, the Fuzhou fraud case has combined with other high profile deepfake incidents to remind people of the potential downsides to such advances in artificial intelligence.

AI regulation is still a developing subject in China. Initial excitement around the potential of ChatGPT and similar AI products in China has given way to concerns over how AI could be used to supercharge criminal activity.

Tech talk aside. For ordinary people, what can we do to avoid being victim of AI-related scams?

First of all, experts suggest you should be mindful of unexpected and urgent calls asking for money from your loved ones or your work.For example, try to ask them some personal questions to verify their identity.

Second, try using a different source, a different channel. Make up an excuse and say you have to call them back. Then, call back on what you know to be the person"s number.

And if you receive such video calls, try to detect if they’re fake through unnatural facial features and expressions. For one thing, unnatural eye movement, and lack of blinking are clear signs of deep fakes. Replicating natural eye movement through body language is harder for deepfake tools. Also, the lighting and the facial features of the image or the video such as the hair and teeth may seem to be mismatched. In the most obvious giveaways are misaligned facial expressions, and sloppy lip to voice synchronizations, unnatural body shapes, and awkward head and body positions.

在一個普普通通的周一上午，你突然接到了一個微信視頻電話，視頻那頭正是你的朋友。朋友看起來非常著急，手足無措，哭著說他的孩子被車撞了，正在ICU搶救，急需一大筆錢救命，懇求你借錢并希望你能馬上轉給他。

危急關頭，你會怎么做？

如果你在想，“當然救人要緊，不能對孩子見死不救，必須馬上轉賬給他”的話，你可能已經落入了“AI換臉”騙局的圈套了。

近日，“AI詐騙正在全國爆發”話題一度沖上熱搜第一，引發網友熱議。最近，類似AI換臉詐騙案件又發生了，這回甚至“更有效率”——9秒鐘被騙走245萬。

今年4月，安慶經開區發生一起“冒充熟人”詐騙案，經開公安分局反詐中心民警調查發現，詐騙分子使用了一段9秒鐘的智能AI換臉視頻佯裝“熟人”，讓受害人放松警惕從而實施詐騙。

4月27日，何先生的微信“好友”突然向其發起視頻通話，電話接通后，何先生看到“好友”正在一間會議室內，就在他準備進一步詢問時，“好友”直接掛斷了電話，并表示在會議中有重要事情交代，需要何先生添加QQ溝通。

隨后，“好友”在QQ上告訴何先生，目前有一個項目招標需要周轉資金，希望何先生先行幫忙墊付。“因為打了視頻電話又是熟人”，“我就沒多想，就轉賬了”，基于對“熟人”的信任，何先生沒有猶豫，立刻讓家人將245萬元到對方指定的賬號上，直到事后撥打對方電話才得知被騙。

接報案后，專案民警連夜行動，于4月28日下午一舉抓獲李某某等3名犯罪嫌疑人，扣押涉案手機26部，凍結、追回電詐資金一百余萬元。5月22日，民警將先行追回的132萬元被騙款返還給何先生。目前，該案件正在進一步偵辦中。

此前，多地都發生過類似案例。

4月20日，福建福州郭先生的好友突然通過微信視頻聯系他，稱自己的朋友在外地競標需要430萬保證金。基于對好友的信任，加上已經視頻聊天核實了身份，郭先生沒有核實錢款是否到賬，就分兩筆把430萬轉到了好友朋友的銀行卡上。之后，郭先生撥打好友電話才知道被騙，騙子通過智能AI換臉和擬聲技術，佯裝好友實施了詐騙。

“當時是給我打了視頻的，我在視頻中也確認了面孔和聲音，所以才放松了戒備。”郭先生說。幸運的是，接到報警后福建福州、內蒙古包頭，兩地警方和銀行，迅速啟動止付機制，成功止付攔截336.84萬元，但仍有93.16萬元被轉移，目前正在全力追繳中。

除了網絡詐騙，AI“明星”帶貨最近也層出不窮，令人傻傻分不清楚。

近期，網上出現了一些“換臉直播”教程。在一個展示換臉直播效果的視頻中，使用者把某明星的模型載入相關軟件后，攝像頭前的人在直播畫面中就有了與明星相似的五官，但臉型和發型還保持原樣。

“點進直播間一看，’迪麗熱巴’居然在直播賣貨。”近日，有網友點開直播間，發現正在賣貨的竟是當紅女星。然而再定睛一看，這些帶貨“明星”很快露出了馬腳——正在賣力帶貨的“明星”們，其實只是使用了AI實時換臉技術的普通主播。AI實時換臉，正在直播間悄然出現。而楊冪、迪麗熱巴、angelababy等當紅女星，成為了AI換臉的重點對象。

在一些線上社交平臺內，所謂“換臉”實則早已有之，其主要是以“特效”或“道具”的形式存在的。這種初代“換臉”技術的運用，目的都很明確，無非是為了“美顏”“搞怪”“娛樂”。與之相較，最近引發關注的“AI換臉”，則完全不是一回事。一些直播間主播換臉成當紅女星，大模大樣、堂而皇之地“帶貨”，這已然超越了“玩笑”的范疇，明顯是一種以不正當手段進行商業化謀利的行為。

需要厘清的是，作為生成式人工智能的最新應用，帶貨主播AI換臉固然是新事物，但卻不存在“監管空白”“無法可依”的情況。此類做法，完全符合民法典所規定的“侵犯肖像權”的構成要件，乃是典型的“未經他人同意”“以營利為目的”。

更有甚者，不法之徒用“AI換臉技術”合成淫穢視頻，收費供他人觀看，甚至根據顧客的需要使用不同的女明星形象進行“私人定制”，還通過出售“換臉軟件”非法獲利。4月9日，據杭州市人民檢察院消息，近日，一80后男子虞某因涉嫌制作、傳播淫穢物品牟利罪被提起公訴。

不僅普通人，上市公司也深受AI假新聞困擾，甚至“掐”起來了。

5月24日，科大訊飛盤中接近跌停，起因直指兩篇網傳“小作文”。第一篇小作文稱，美國正在考慮是否將科大訊飛、美亞柏科等加入“實體名單”，禁止它們使用美國的組件或軟件。第二篇小作文稱，近期，科大訊飛被曝涉嫌大量采集用戶隱私數據，并將其用于人工智能研究。

針對第一個傳聞，科大訊飛24日盤后回復稱，已于2019年10月被列入實體清單。被列入實體清單后，科大訊飛已迅速切換到以國產供應鏈為主的非美供應鏈體系，業務運營未受到重大影響。

而關于第二個傳聞，則是“AI詐騙”。相關流傳文字及截圖，科大訊飛初步判斷該段不實信息系某AI生成式軟件生成。從流傳的截圖可知，該軟件的標志正是百度的大語言模型、生成式AI產品“文心一言”。

科大訊飛剛剛辟謠，就引來百度文心一言市場負責人張文全在朋友圈怒懟，直指“策劃痕跡太重”，“請友商解決好自己的問題，別動不動就碰瓷別人，大家的眼睛都雪亮的。”

對于百度的最新回復，科大訊飛方面暫未作出回應。

實際上，一段時間以來，利用AIGC編造、傳播虛假信息已不是“新鮮事”，人類社會已然步入“后真相時代”。

美國時間5月22日上午，一張“五角大樓附近爆炸”的圖片在海外社交網絡上瘋傳。據媒體報道，美國國防部發言人證實，這張在谷歌搜索和推特上瘋傳的圖片是一條“虛假信息”。這一虛假信息廣泛流傳后，美國股市出現了明顯震蕩，道瓊斯工業平均指數4分鐘之間下跌了約80點。

AIGC的發展正在給虛假信息治理提出前所未有的挑戰。在“越來越像人類”的同時，人工智能所具有的“幻覺”“涌現性”等特點帶來的伴生風險也引發各界關注。

AIGC為什么能成為謠言“推手”？南開大學法學院副院長、中國新一代人工智能發展戰略研究院特約研究員陳兵教授表示，AIGC技術日漸發展成熟，可以完成包括寫郵件、代碼、新聞報道以及論文在內的多種任務，且表述內容與人類對話風格相似，一般人難以區分。此外，其使用門檻和成本低，生成效率高，能夠在短時間內生成大量虛假信息，從而迅速淹沒真相。

中國政法大學數據法治研究院教授張凌寒指出，俗語說“耳聽為虛，眼見為實”“有圖有真相”，可見照片、視頻在公眾潛意識中的真實性遠勝文字。而深度合成技術則恰恰顛覆了這一公眾認知，其可以偽造音頻、圖片、視頻，捏造人物的行為與語言。深度合成技術在近幾年迅速興起，為政治、軍事、經濟犯罪甚至恐怖活動等提供了新工具，帶來嚴峻的安全挑戰。

在北京航空航天大學法學院副教授、北京科技創新中心研究基地副主任趙精武看來，ChatGPT等生成式AI最顯著的技術優勢就是能夠以貼近人類思維和表達的方式呈現信息，這就導致網民更難以甄別其生成信息的真偽，加之人工智能技術能批量生成信息，海量且高效的網絡謠言生成，顯然會導致官方辟謠、賬號封禁等傳統虛假信息治理措施難以發揮預期效果。

深度合成、人工智能技術火爆出圈，如何平衡創新發展與風險防范成為必答題。

此前，國家網信辦、工信部、公安部聯合發布《互聯網信息服務深度合成管理規定》，于今年1月10日起施行。規定明確指出，提供人臉生成、人臉替換、人臉操控、姿態操控等人物圖像、視頻生成或者顯著改變個人身份特征的編輯服務，可能導致公眾混淆或者誤認的，應當在生成或者編輯的信息內容的合理位置、區域進行顯著標識。除了顯著標識幫助用戶區分虛擬與現實，直播平臺還應履行責任，從源頭杜絕利用AI技術侵犯肖像權等行為。

除此之外，4月11日，我國也出臺了第一份對生成式 AI 進行監管的文件，國家互聯網信息辦公室正式發布《生成式人工智能服務管理辦法（征求意見稿）》，更是對此場景給出了極有針對性的預設規范——一些主播AI換臉女星帶貨，不再是“法無禁止即可為”。

作為普通人，面對AI換臉詐騙技術，有什么方法可以識別呢？

AI換臉詐騙層出不窮，不過“AI假臉”必然會有一些瑕疵，可以通過細節來判斷對方是否真是“AI假臉”。

專家提示稱，“AI假臉”的紋理特征存在破綻。例如，偽造后的視頻人物的眼睛或牙齒輪廓細節容易不一致；兩只眼睛瞳孔的顏色不一樣或瞳孔中心反射的細節不一樣；或是很多偽造視頻由于視頻分辨率低于原始視頻分辨率，偽造出來視頻的牙齒邊緣過于整齊等。

另外，“AI假臉”有可能不符合正常人的生理特征，比如，一個健康成年人一般間隔2-10秒眨一次眼，每次眨眼用時0.1-0.4秒，而在偽造視頻中，人的眨眼頻率可能不符合上述規律。

最后，由于人嘴部的運動是最為頻繁且快速的，因此AI軟件無法真實準確地渲染連續動作。因此，“AI假臉”的嘴部特征有可能辨別出真假。此外，偽造后的視頻會造成一定的視頻抖動，導致視頻出現幀間不一致情況。

中國互聯網協會日前也對“AI換臉”新騙局做出提醒，遠程轉賬務必需要多重驗證，把好“錢袋子”。如果有人自稱“家人”“朋友”等誘導你轉賬匯款，務必第一時間提高警惕。在轉賬匯款、資金往來這樣的典型場景，要通過回撥對方手機號等額外通信方式核實確認，不要僅憑單一溝通渠道未經核實就直接轉賬匯款，無論對方是誰。

Executive Editor: Sonia YU

Editor: LI Yanxia

Host: Stephanie LI

Writer: Stephanie LI

Sound Editor: Stephanie LI

Graphic Designer: ZHENG Wenjing, LIAO Yuanni

Produced by 21st Century Business Herald Dept. of Overseas News.

Presented by SFC

編委: 于曉娜

策劃、編輯：李艷霞

播音：李瑩亮

撰稿：李瑩亮

音頻制作：李瑩亮

設計：鄭文靜、廖苑妮

21世紀經濟報道海外部 制作

南方財經全媒體集團 出品

關鍵詞：