Skip to main content

Hierarchical Evidential Fusion for Multimodal Sentiment Analysis Under Signal Degradation

By
Aolin Xiong; Haifeng Hu

Multimodal sentiment analysis integrates language, acoustic, and visual signals, but occlusion, background interference, and sensor failure degrade these signals unpredictably, making fusion unreliable. Most existing methods aim to recover or extract task-relevant information from degraded signals rather than to assess how reliable those signals are. To address this issue, we propose a hierarchical evidential fusion framework (HEF) to achieve reliable decision-making. Specifically, we introduce a hierarchical evidence measurement module that progressively quantifies signal reliability from fine to coarse at both the token and modality levels. Furthermore, we design a reference-conditioned evidential modulation module to mitigate modality collapse caused by modality imbalance and direct weighting. It compares each modality with its own reference and reduces its evidence contribution when the signal is unreliable. Finally, we develop a multimodal evidence fusion module that combines the modulated evidence with learnable prior evidence to make reliable sentiment predictions. Extensive experiments on the widely used MOSI and MOSEI benchmarks demonstrate the effectiveness and robustness of the proposed framework.

Read on IEEE Xplore