RM新时代有限公司

Recently, ICASSP 2023, a top international conference in the field of speech and signal processing, was successfully held in Greece. The conference invited nearly 4,000 people from major research institutions, experts, scholars, and well-known enterprises such as Google, Apple, Huawei, and Meta AI to discuss technology and industry development trends and exchange the latest achievements.

The paper "Diffusion Model-Based Audio-Driven Talking Face Generation" by the joint research team of CloudWalk Technology and Shanghai Jiao Tong University was accepted by the conference and presented on-site, receiving high attention from many parties.

ICASSP (International Conference on Acoustics, Speech and Signal Processing) is one of the top international conferences in the field of speech and acoustics. Research results presented at ICASSP are considered to represent the cutting-edge level and future development direction of acoustics and speech.

The accepted paper focuses on the cross-modal task of audio-driven talking face video generation, combining speech and vision technologies. The proposed method can generate talking face videos with natural head movements, accurate lip movements, and high-quality facial expressions based on input speech segments. This achievement outperforms previous studies on multiple datasets.

Moreover, in practical scenarios, as digital humans become increasingly widespread, the demand for generating talking face videos synchronized with input audio is growing. This achievement, based on diffusion model cross-modal talking face generation technology, can be extended to a wide range of application scenarios such as virtual news broadcasting, virtual speeches, and video conferencing.
Paper address:https://ieeexplore.ieee.org/document/10094937/

01

Introduction

Audio-driven Talking Face Video Generation: This task involves generating a talking face video of a target person synchronized with the audio, based on a photo of the target person and any speech audio. Due to the more natural, accurate lip movements, and higher fidelity head poses and facial expressions, this task is widely used in fields such as digital humans, virtual video conferencing, and human-computer interaction. As a cross-modal task of vision and audio, audio-driven talking face video generation has received increasing attention.

To construct the mapping from audio signals to facial deformations, existing methods introduce intermediate facial representations, including 2D key points or 3D morphable face models (3DMM). Although these methods achieve good visual quality in audio-driven facial reenactment tasks, the information loss caused by intermediate facial representations may lead to semantic mismatch between the original audio signal and the learned facial deformations.
Moreover, GAN-based methods suffer from unstable training and are prone to model collapse, often generating images with only fixed resolutions. To address these issues, AD-NeRF introduces neural radiance fields, directly feeding audio signals into the implicit function of dynamic radiance fields, and finally rendering realistic synthetic videos. However, methods based on neural radiance fields require large computational loads, leading to long training times and high computational demands.
Moreover, most of these works ignore personalized facial attributes and cannot accurately synchronize audio with lip movements. Therefore, the researchers in this paper propose a method that efficiently optimizes personalized attribute features of facial parts using denoising diffusion models, thereby synthesizing high-fidelity, high-definition videos.

02

Method

The method is based on a key intuition: lip movements are highly correlated with speech signals, while personalized information such as head pose and blinking is weakly correlated with audio and varies among individuals. Inspired by the rapid progress of diffusion models in high-quality image and video generation, the researchers reconstruct a new framework for audio-driven facial reenactment based on diffusion models, optimizing the generation quality and realism of talking face videos.

Difface consists of four main parts: (1) facial attribute decoupling; (2) lip-audio contrastive synchronization; (3) dynamic continuous attribute information modeling; (4) talking face generation based on denoising diffusion models. In the facial attribute decoupling part, the researchers use 3DMM to extract head pose and expression coefficients from the source identity image, and then, following previous work DFA-NeRF, use a fully connected autoencoder to decouple lip movements and blinking information from expression parameters.

In the lip-audio contrastive synchronization module, the researchers introduce a self-supervised cross-modal contrastive learning strategy to deploy a deterministic model that synchronizes audio and lip movement features.

In the dynamic continuous attribute information modeling module, since personalized facial attributes such as head pose and blinking are random and probabilistic, to model the probability distribution of facial attributes and generate long time sequences, the researchers propose a probabilistic model based on a Transformer-based variational autoencoder (VAE). First, VAE can smooth discrete attribute information and map it to a Gaussian distribution; second, the attention mechanism of Transformer is used to fully learn the long-term inter-frame dependencies of time sequences.

In the talking face generation module based on denoising diffusion models, the generated personalized facial attribute sequences are concatenated with synchronized audio embeddings as input conditions for the diffusion model. Then, using a conditional denoising diffusion probabilistic model (DDPM), these driving conditions and the source face are taken as input to generate the final high-resolution talking face video through diffusion generation. These personalized facial attribute sequences and synchronized audio embeddings enrich the diffusion model to maintain consistency in the generated image sequence.

03

Experimental Results

The researchers validated the superior performance of this method for audio-driven talking face video generation through experiments.

? Quantitative Comparison Experiments

The researchers compared this method with existing audio-driven facial video generation methods through quantitative analysis experiments, using multiple objective evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Landmark Movement Distance (LMD), and Audio-Visual Synchronization Confidence (Sync). Specific information is shown in Table 1.

The proposed diffusion generation framework outperforms other methods on all performance metrics. PSNR and SSIM verify that the facial attribute decoupling scheme better captures personalized information such as head pose and blinking. The LMD score of this method indicates stronger lip-audio consistency. Additionally, benefiting from cross-modal contrastive learning between input audio and lip movements, this method significantly surpasses other methods on the Sync metric.

? Qualitative Comparison Experiments

The researchers compared this method with existing audio-driven facial video generation methods. Through learning personalized attributes and optimizing the diffusion model, our method generates facial videos with personalized head movements, more realistic blinking information, and better lip-audio synchronization performance.

? Benefits of Each Module in the Model

To highlight the importance of each module in the model, the researchers conducted ablation experiments. As shown in Table 2, adding the DDPM module brings the greatest improvement in inference speed and visual quality compared to other modules. Second, benefiting from the decoupled facial attribute information and VAE's attribute smoothing and dynamic continuity modeling, the naturalness of the talking face is improved. Furthermore, the lip-audio contrastive learning module significantly enhances the synchronization quality between lip movements and input audio through self-supervision.

? Model Efficiency

The researchers also presented the model's trainable parameter count, inference speed, and output resolution, and compared them with previous SOTA models. Since the denoising diffusion probabilistic model uses variational methods instead of adversarial training and does not require deploying multiple discriminators, it greatly alleviates the problem of model collapse during training. Moreover, with shorter time steps, inference speed is significantly improved, enhancing efficiency.

04

Conclusion

For the task of audio-driven high-fidelity talking face video generation, the joint research team of CloudWalk and Shanghai Jiao Tong University proposed an audio-driven talking face video generation method based on a diffusion framework. It requires only one or a few identity images and input speech audio to synthesize a high-fidelity facial video, achieving state-of-the-art visual quality. Additionally, cross-modal lip-audio contrastive learning is used to improve lip-audio consistency, achieving SOTA performance on public datasets.