From Uni- to Multi-modal: Progressive Multi-source Domain Adaptation with Gradient Disentangling for Audio-visual Deception Detection

Audio-visual deception detection is essential for understanding human intent by accurately identifying the truthfulness or deceptive behavior. However, current methods struggle with domain shift between datasets and cannot fully use deception-related information from multiple sources. Aiming at tackling domain shift and modality gap issues across source and target domains, we propose a multi-source multimodal progressive domain adaptation (MMPDA+) framework with gradient disentangling that transfers the audio-visual knowledge from multiple source deception domains to the target one. By gradually aligning the source and target domains at both feature and decision levels, our method effectively bridges domain shift across diverse multimodal deception datasets. Observing the gradient conflicts between unimodal and multimodal learning in both task prediction and domain adaptation, we incorporate a disentangled gradient learning strategy to further enhance the training stability and boost the adapting performance. Extensive experiments conducted on six audio-visual deception datasets demonstrate the effectiveness of the proposed approach, achieving 58.5% accuracy averaged on each target domain. Our code is available at https://github.com/RH-Lin/MMPDA.