The Future of Deepfake Detection
As AI advances, will it be possible to identify fake media?
Until recently, AI-generated images had clear, telltale signs. In hyper realistic images, for example, image models have generated anatomically impossible hands, giving people multiple or too few fingers. Inconsistencies in other aspects of generating people, such as teeth, clearly indicate that such images are not real. One obvious reason is the lack of relevant training data that can capture the natural behavior and characteristics of joints and other real-world phenomena. Additionally, generative models failed to grasp the nuances of prompts, sometimes interpreting them incorrectly and generating implausible images like a salmon fillet in a river based on the prompt “salmon jumping in a river”. However, with each new iteration of popular generative models, the results are improving. Midjourney, one of the most popular text-to-image generators, has largely tackled the “AI hand” problem, producing images that are highly realistic and difficult to distinguish as 100% AI generated.
Additionally, improvements in multimodal AI, or combining images, video, text, and audio, have also generated realistic outputs that are becoming more difficult to identify as being fake. Even Open AI recently made a deepfake detector available to a select group of researchers, in order to prevent disinformation and misuse of deepfakes from their own models. The urgency of deepfake detection spans enterprise and consumer use cases, impacting areas including cybersecurity, financial services entertainment, politics and media, as well as individual personal safety and security. Otherwise, rising fraud and misinformation can have serious implications for the future of trust in synthetic media.
Deepfake fraud has already had serious consequences:
An employee from British engineering company Arup transferred $25 million to scammers impersonating his colleagues on a video call.
A mother received a call from an unknown number and heard her daughter’s voice pleading for help, saying that she had been kidnapped.
Meta and WhatsApp recently established a deepfake helpline in India, as the 2024 elections have led to a spike in AI-generated misinformation. In 2024, over 50 countries will hold elections.
Numerous other cases targeting family members, executives, and young people have been detailed here, here, and here.
While these frauds have been taking place in some form or another, advanced tools to generate audio, images, and videos can create scarily accurate real-time deepfakes, taking these scams to another level.
Traditional content moderation practices in large technology companies cannot keep up with the fast pace and realistic nature of AI-generated content. These systems have typically relied on underpaid workers from low-income countries to manually and painstakingly flag thousands of images and videos every day. In the politics and media space, fact-checking organizations and crowdsourced evaluation can be biased and slow to keep up.
Some current approaches include cross-modal content verification (e.g. WeVerify). This involves identifying and contextualizing social media and web content through social network analysis and external sources. However, available online content itself can have its own biases, and often requires a human-in-the-loop to provide additional verification. Therefore, as the scale and sophistication of AI-generated content increases, the resulting challenges will also require advanced algorithmic solutions.
˚｡⋆ How can deepfakes be detected? ⋆｡˚
Firstly, deepfakes can be in the form of images, videos, audio, or a combination of video and audio created using deep learning. Some types of synthetic media can simply be distinguished by the human eye. Common signs include excessive airbrush effects, the unnatural juxtaposition of people against a background, or distorted features, such as eyes or teeth, and of course, robotic-sounding voices. But, as generation methods improve, these signs are also starting to disappear.
How can detection methods keep up? The following includes a survey of current approaches.
Detecting synthetic images and video:
Signatures and watermarks: some models have “signatures” that they leave on their images and videos, such as forensic cues and artifacts that are easy for detectors to recognize. However, as this is specific to different models, generalizability across model providers is still a challenge. Some have also advocated for deepfake provenance or the addition content credentials to track the sources of AI-generated media (Meta and YouTube have already released a similar tools) rather than relying on passive clues. However, the availability of open-source synthetic media creation tools can potential circumvent undermine authentication measures.
Physical and biological signals: there are certain signals that AI models still cannot imitate in their content. For example, FakeCatcher, the basis of tools such as Intel’s Real Time Deepfake Detector, analyzes color changes (PPG signals) in face pixels that can provide inferences regarding blood flow in different facial regions. PhaseForensics, developed by researchers at the University of California, Santa Barbara, analyzes lip movements and information about these movements at different frequencies. This is also known as robust liveness checks, which evaluates the images and videos for signals of “life”.
Facial landmark analysis and facial recognition: some methods include detectors trained on analyzing different parts of the human face, which may not always be present in synthetic renderings.
Biometric authentication: another method is cross-checking with existing records of known individuals and other personally identifiable information, such as audio recordings. But given the subtle enhancements that AI tools can make, methods only reliant on such PII-based datasets may not be as effective to tackle real-time threats with little or no ground truth. Additionally, enterprises might be wary of constantly sharing swaths of personal user data due to potential security vulnerabilities.
Synchronization and mismatches: AI video generation models have had a hard time to synchronize between audio and lip movements, for example, and algorithmic approaches have been able to spot such inconsistencies. Additionally, there can be mismatches in subtle elements, such as merging boundaries, and grayscale elements that are difficult to spot with the human eye. Tools such as Microsoft’s Video AI Authenticator, however, can spot these differences at the pixel level.
One of the main challenges in detecting synthetic images and video is generalization for new, unseen models, which can evolve in unforeseen ways. Detectors need to identify content from existing models, which they tend to do well especially when trained on one specific model. But, they also need to accurately predict future synthetic content. Reality Defender, for example, takes an “ensemble approach”, where they build multiple and interconnected detection models addressing a variety of features. Additional considerations include image processing methods, which affect how key artifacts are retained. Finally, cross-concept scenarios are also important, for example, a detector can be trained on AI-generated human faces, but the real-time data can include synthetic animal images. In the end, the detectors need to continuously adapt to real-time fraud cases and stay one step ahead.
Detecting audio and voice cloning deepfakes is an area that has been relatively less explored compared to visual media. Approaches that have been explored include finding the biometric signatures of speakers, and comparing it with the new sample. Another approach led by researchers at the University of Florida involved training a model on real and fake recordings, and using data from the vocal tract to determine whether certain sounds in the audio recording were biologically possible. There are arguments that the future of AI and consumer experiences will be mediated by conversational AI, leading more voice commands, voice messages, and data collection from real-time conversations. In that case, it is an open question whether these systems will continue to improve to the extent that detectors cannot identify the difference since they are constantly trained on real-time voice data.
Detecting AI-generated written content is perhaps the trickiest for detection systems, as people can write and use language in an endless variety of ways. This means that there are fewer signals that can be identified or used in a benchmark, compared to analyzing pixels for videos and images. Plagiarism detectors, for a long time, have sought to distinguish between copied and “original” texts, but AI content has introduced a new challenge. Researchers at the University of Maryland have proposed a framework for synthetic texts, in which the models would integrate a “greenlist” of specific words or tokens to indicate that the output text was generated by a particular AI company.
˚｡⋆ Conclusion ⋆｡˚
Current deepfake detection companies focus on enterprise customers, in part due to the fact that platforms have responsibility for its users, and because significant computational power is required to manage large-scale detection systems. However, according to the CEO of Reality Defender, who likened it to the evolution of antivirus software, there is a potential for deepfake detection systems to eventually run on edge devices.
As the volume and variety of frauds increase, deepfake detection and content moderation need to deliver results faster, and ensure that their computational approaches will not “break”. While platform companies such as Midjourney have introduced measures to prevent misuse, such as scrapping free trials, and banning certain prompts, ultimately the restrictions in place depend on the company. In the US, a handful of individual states have laws in place regarding deepfake content, and the EU AI Act also addresses this topic. Still, such legislation ultimately relies on innovation deepfake detection technologies in order to truly safeguard individuals.
Given the increasing speed with which high-quality deepfakes are being created, new detection methods for text, audio, and video is an important space to watch.
Companies in this space:
Fraud prevention, multimodal AI: Reality Defender, Onfido
Image and video: Sentinel, Deep Media, TruePic, Sensity, Nuanced
Politics and media: Alethea
Entertainment: Outtake
Further Reading
More about C2PA, a new standards body for online content (members include Open AI, True Pic, BBC, among other tech and media companies)
The Race to Regulate Deepfakes –
What do you think will be critical for deepfake detection? Reach out, share your thoughts, and keep in touch!