Causal Cross-Modal Consistency and Contradiction Learning for Explainable Video–Audio–Text Fake News Detection

Abstract

With the advent of the increasingly large amounts of multimedia news, the fake-news detection problem has become a complex one that requires gathering evidence from different types of media: a video could contain a credible but incomplete record of events, an audio record a convincing but contradictory account of what happened, and a text provide only a partial and incorrect view of the events. In this work, a Causal Cross-Modal Consistency and Contradiction Learning (C³L) framework is proposed to aid in making explainable video–audio text fake-news detection. Four key aspects related to modelling, viz. representation learning of the modalities, temporal cross-modal alignment, explicit consistency and contradiction representations, and causal relation modelling, are combined along with modality-reliability estimation and evidence oriented explainability into a single architecture. Unlike typical multimodal fusion, that is mainly focussed on the fusion of heterogeneous representations, it takes into account the question of whether multimodal evidence is temporally aligned, consistent, contradictory and even reliable enough for the decision-making. The proposed framework was compared on unimodal baseline, established multimodal approaches, consistency-based approach, and the video audio–text state-of-the-art approach by using Accuracy, Precision, Recall, F1-score and AUC. With the considered experimental setup the C³L produced an accuracy of 93.0%, a precision of 93.5%, a recall of 92.5%, an F1-score of 93.0% and an AUC of 95.5%. In comparison to the analysed state-of-the-art video–audio–text approach, the framework's results showed an increase of 4.5, 5.0, 5.0, 5.0 and 4.0 percentage points in terms of Accuracy, Precision, Recall, F1–score, and AUC, respectively. The results show that explicit modelling of the cross-modal relationship enhances the multimodal authenticity assessment over aggregating the features. The explainability mechanism also provides modality, time, and cross-modal evidence leading to predictions. However, the results of the performance depend on the data set and quality of the input, the concept of data temporal synchronization and manipulation strategies that evolve. The proposed framework is structured and offers strong, interpretable, and causally-meaningful support for a robust multimodal fake-news detection system. These findings form a solid evidence-based foundation for enhanced trustworthy and transparent automated misinformation analysis in the presence of different multimedia contexts.