Skip to main content
16 November 2026

Multimodal Transformer for Material and Semantic Segmentation

MMSFormer

Contributed by Md Kaykobad Reza, Ashley Prater-Bennette, and M Salman Asif based on the IEEE Xplore® article, “MMSFormer: Multimodal Transformer for Material and Semantic Segmentation”, published in the IEEE Open Journal of Signal Processing, April 2024.
[Code: https://github.com/csiplab/MMSFormer]

 

Introduction

Image segmentation assigns a class label to each pixel in an image and is fundamental to scene understanding in applications such as autonomous driving, robotics, and remote sensing. While single-modality methods using RGB images have made significant progress, they struggle in challenging conditions where additional sensory information—such as infrared, depth, or polarization data—could provide critical complementary cues [1,2]. Multimodal image segmentation leverages diverse sources of information to produce more robust and accurate predictions [3]. However, effectively fusing information from different modalities remains challenging due to the unique characteristics of each modality. Most existing multimodal methods are designed for specific modality pairs (e.g., RGB-Depth or RGB-Thermal) and do not generalize well to other combinations [4,5]. Some recent models can handle arbitrary modalities but rely on complex fusion strategies or require additional semantic labels [6,7]. We propose MMSFormer (Multi-Modal Segmentation TransFormer), a new model incorporating a novel fusion block that can effectively combine information from arbitrary (heterogeneous) combinations of modalities. Our approach is both simple and computationally efficient while achieving state-of-the-art performance on three different datasets for multimodal material and semantic segmentation tasks.

Proposed Model Architecture

Nov1626Blog
FIGURE 1: Overall architecture of MMSFormer model and proposed fusion block.

MMSFormer consists of three main modules: (1) modality-specific encoders, (2) a multimodal fusion block, and (3) a shared MLP decoder. Each input modality is processed by its own Mix Transformer (MiT) encoder [8], which extracts hierarchical features at four different scales. The features from all modalities are then fused at each scale using our proposed fusion block before being passed to a lightweight MLP decoder for final segmentation prediction.

The key innovation lies in our multimodal fusion block. Given features from M modalities, the fusion block first concatenates them along the channel dimension and passes them through a linear fusion layer to reduce dimensionality. It then applies parallel convolutions at multiple kernel sizes (3×3, 5×5, and 7×7) to capture multi-scale spatial patterns. A Squeeze-and-Excitation block [9] serves as channel attention in the residual connection, dynamically re-calibrating feature importance across channels. This design allows the block to effectively fuse information from any number and combination of input modalities without requiring modality-specific customization.

Experimental Results

We evaluated MMSFormer on three challenging datasets: MCubeS (multimodal material segmentation with four modalities) [3], FMB (RGB-Infrared semantic segmentation under diverse weather conditions) [10], and PST900 (RGB-Thermal semantic segmentation) [11]. MMSFormer achieves new state-of-the-art results on all three benchmarks, with particularly significant improvements on the FMB dataset. Our model consistently outperforms existing methods across all modality combinations, demonstrating the generalizability of the proposed fusion strategy. Importantly, our fusion block is also computationally efficient requiring significantly fewer parameters and FLOPs than competing fusion methods while delivering superior performance.

FIGURE 2: Visualization of predicted segmentation maps for different modality combinations on MCubeS and FMB datasets.

Incremental Modality Integration

A critical finding of our work is that performance improves progressively as additional modalities are incorporated. Starting from RGB-only predictions on the MCubeS dataset, each additional modality (AoLP, then DoLP, then NIR) yields a consistent boost in segmentation accuracy. This demonstrates the fusion block's ability to effectively leverage and combine complementary information from each new modality without diminishing returns. Ablation studies further reveal that different modalities assist in identifying different types of materials—polarization data enhances recognition of road markings and glass, while near-infrared is particularly effective for classifying asphalt, concrete, and cobblestone.

Key Takeaways

  • More modalities, better results. Unlike prior methods that require redesigning the fusion strategy for each new modality, MMSFormer’s plug-and-play fusion block handles any combination out of the box. Practitioners can incrementally add sensors (thermal, polarization, NIR) and expect consistent gains.
  • Efficiency matters for real-world deployment. Our fusion block uses 2–8× fewer parameters and up to 7× fewer FLOPs than competing approaches. This makes multimodal segmentation practical for resource-constrained settings such as embedded robotics or edge devices.
  • Different sensors reveal different materials. The ablation studies provide actionable guidance: if an application needs to distinguish road markings or glass, polarization data (AoLP) helps most; for asphalt or concrete classification, near-infrared is the strongest complement to RGB.
  • Simple designs can outperform complex ones. The combination of linear fusion, parallel multi-scale convolutions, and channel attention is straightforward yet outperforms methods with far more complex cross-attention or gating mechanisms.

Future Directions

One current limitation is that each modality requires its own dedicated encoder, meaning the number of parameters grows with the number of input modalities. Future work will explore the use of a shared encoder across all modalities, which could significantly reduce model size while potentially enabling the model to learn cross-modal representations at the feature extraction stage itself. Additionally, extending MMSFormer to other multimodal tasks—such as panoptic segmentation, object detection, and video understanding—could further demonstrate the generality of the proposed fusion strategy. Handling missing modalities during inference is another challenge in multimodal learning. In our latest works, we address this through parameter-efficient adaptation [13] and cross-modal proxy token generation [14].

Conclusion

MMSFormer presents a simple, efficient, and flexible approach to multimodal segmentation that generalizes across arbitrary modality combinations. The progressive performance gains with each added modality validate the effectiveness of our fusion strategy, opening avenues for applying multimodal approaches to a broader range of signal processing and computer vision tasks.

References

[1] Minaee, Shervin, et al. "Image segmentation using deep learning: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44.7, no. 7, pp: 3523-3542, 2021. DOI: 10.1109/TPAMI.2021.3059968

[2] Zhang, Yifei, et al. "Deep multimodal fusion for semantic image segmentation: A survey." Image and Vision Computing 105 (2021): 104042.

[3] Liang, Yupeng, et al. "Multimodal material segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. DOI: 10.1109/CVPR52688.2022.01918

[4] Chen, Xiaokang, et al. "Bi-directional cross-modality feature propagation with separation-and-aggregation gate for RGB-D semantic segmentation." European conference on computer vision. Cham: Springer International Publishing, pp: 561-577, 2020. DOI: 10.1007/978-3-030-58621-8_33

[5] Sun, Yuxiang, Weixun Zuo, and Ming Liu. "RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes." IEEE Robotics and Automation Letters, vol. 4, no. 3, pp: 2576-2583, 2019. DOI: 10.1109/LRA.2019.2904733

[6] Zhang, Jiaming, et al. "CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers." IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 12, pp: 14679 - 14694, 2023. DOI: 10.1109/TITS.2023.3300537

[7] Zhang, Jiaming, et al. "Delivering arbitrary-modal semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. DOI: 10.1109/CVPR52729.2023.00116

[8] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in neural information processing systems, vol. 34, pp: 12077-12090, 2021.

[9] Hu, Jie, Li Shen, and Gang Sun. "Squeeze-and-excitation networks." Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. DOI: 10.1109/CVPR.2018.00745

[10] Liu, Jinyuan, et al. "Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation." Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023. DOI: 10.1109/ICCV51070.2023.00745

[11] Shivakumar, Shreyas S., et al. "Pst900: Rgb-thermal calibration, dataset and segmentation network." 2020 IEEE International Conference on Robotics and Automation (ICRA) IEEE, 2020. DOI: 10.1109/ICRA40945.2020.9196831

[12] Zhou, Wujie, et al. "CACFNet: Cross-modal attention cascaded fusion network for RGB-T urban scene parsing." IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp: 1919-1929, 2024. DOI: 10.1109/TIV.2023.3314527

[13] Reza, Md Kaykobad, Ashley Prater-Bennette, and M. Salman Asif. "Robust multimodal learning with missing modalities via parameter-efficient adaptation." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 2, pp: 742-754, 2025. DOI: 10.1109/TPAMI.2024.3476487

[14] Reza, Md Kaykobad, et al. “Robust Multimodal Learning via Cross-Modal Proxy Tokens.” Transactions on Machine Learning Research, 2025, openreview.net/forum?id=Wtc6wvcYJ0.