Contributed by Hui Ma and co-authors, based on the IEEEXplore® article, “A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations”, published in the IEEE Transactions on Multimedia, 2024.

Introduction

Emotion recognition in conversations (ERC) aims to identify the emotion of each utterance in a conversation [1]. The task has recently become a prominent research topic due to its wide applications in opinion mining, health care, and building empathic dialogue systems. However, for researchers in affective computing and conversation intelligence, ERC remains exceptionally challenging because human emotion is rarely conveyed through text alone. A speaker might sound angry while using neutral words, smile while hiding frustration, or shift emotions as the conversational context evolves. These subtle cues make multimedia signal processing [2,3] essential for developing multimodal ERC methods capable of accurately capturing the true emotion state of the speaker.

The paper introduces SDT, a transformer-based model with self-distillation for multimodal ERC. The proposed work addresses three key challenges: intra- and inter-modal interactions between conversation utterances, different contributions of modalities, and efficient modal representations. This method is highly effective, as it enables dynamic multimodal fusion and enhances unimodal representations via self-distillation without relying on an external teacher, ultimately yielding robust and accurate emotion recognition.

Proposed Method

As illustrated in Fig. 1, SDT introduces three key components. First, after extracting utterance-level features, a modality encoder leverages intra- and inter-modal transformers to capture deep interactions throughout the conversation. Specifically, the intra-modal transformer enhances a specific modality's sequence representation internally, while the inter-modal transformer enables one modality to gather complementary information from another. Second, a hierarchical gated fusion strategy handles the shifting importance of emotional cues. It filters out irrelevant unimodal information and dynamically weights the textual, acoustic, and visual inputs to isolate the most reliable emotional signals for each specific utterance. Finally, a self-distillation mechanism treats the proposed transformer-based model as the teacher and designs three student networks according to the existing modalities. By transferring knowledge of both hard and soft labels to each modality, this mechanism strengthens unimodal representations and boosts overall emotion recognition accuracy.

Experimental Results                                                                             

To demonstrate the effectiveness of our method, we evaluated SDT on the IEMOCAP and MELD datasets. The results show that SDT outperforms previous state-of-the-art baselines on both datasets. In addition, extensive studies were conducted to investigate the impact of positional and speaker embeddings, intra- and inter-modal transformers, self-distillation loss functions, and the hierarchical gated fusion strategy. Furthermore, Fig. 2 illustrates the visualization of multimodal representations on the IEMOCAP dataset.

The visualization results indicate that our model with self-distillation yields a better separation and representations of different emotions are less mixed together. Therefore, introducing self-distillation training enables the model to learn more effective multimodal representations. Comprehensive details of all experiments can be found in our published paper [4].

Challenges in Multimedia Signal Processing for ERC

While SDT demonstrates significant advancements, our error analysis highlights several critical open challenges in multimedia signal processing for ERC. First, distinguishing semantically similar emotions, such as "happy" versus "excited", or "angry" versus "frustrated", remains difficult. Second, class imbalance is a persistent bottleneck; it is challenging to correctly detect minority emotions like "fear" and "disgust" on the MELD dataset. Finally, models struggle with emotional shift, occurring when two consecutive utterances spoken by the same speaker exhibit different emotions. Ultimately, achieving perfectly robust emotion recognition will demand context-aware architectures that can track latent speaker states and dynamically route attention based on modality reliability.

Code: The code is available at https://github.com/butterfliesss/SDT.

References:

[1] Z. Lian, B. Liu and J. Tao, "CTNet: Conversational Transformer Network for Emotion Recognition," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 985-1000, 2021, doi: https://doi.org/10.1109/TASLP.2021.3049898.

[2] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang and M. D. Plumbley, "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition," IEEE SPS Blog, 2025.

[3] W. Zhang, S. Jin, P. Zhuang, Z. Liang and C. Li, "Underwater Image Enhancement via a Robust yet Efficient Dual Prior Optimized Method," IEEE SPS Blog, 2024.

[4] H. Ma, J. Wang, H. Lin, B. Zhang, Y. Zhang and B. Xu, "A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations," in IEEE Transactions on Multimedia, vol. 26, pp. 776-788, 2024, doi: https://doi.org/10.1109/TMM.2023.3271019.