Multimodal Deepfake Detection using Audio-Visual Temporal Inconsistency and Cross-Modal Feature Learning

Abstract

With the rise in the fidelity of synthetic video and voice, there is an urgent need for deepfake detectors that are strong against more than one visual artefact. This paper introduces a multimodal audio-visual framework, which integrates an EfficientNet-B4 based visual stream with a wav2vec 2.0 based audio stream. Streams are temporally aligned, and then fused together by bidirectional Cross-Modal Attention, and an Audio-Visual Temporal Inconsistency Detector is used to compute the difference in cosine similarity between the aligned representations. Using Information Gain, Symmetrical Uncertainty and correlation analysis, 12 features are analysed, and the most powerful features are retained for classification. The framework is tested on a collection of FacesForensics++, DFDC Preview and FakeAVCeleb. On FaceForensics++ (c23), the reported full model has an AUC-ROC of 0.963 and an accuracy of 0.957. The stated unsupervised score is 9.7% better than the best stated unsupervised baseline. The article reports the protocol used for the dataset, the result of feature-selection, the ablation methodology, the cross-dataset behaviour, the performance per manipulation, and the computational cost. The results suggest that explicit temporal correspondence (ETC) makes for a predictive advantage and a meaningful forensic clue, but that compression and deployment limitations on the shift of distribution (DS) are still present.