MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer Video

Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. However, existing deep fusion models generally depend on convolutional operations, which fails to preserve global context information. To compensate for this defect and achieve accurate fusion, we propose a multiscale adaptive Transformer to fuse multimodal medical images termed MATR. We introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the MATR outperforms other state-of-the-art methods in both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability.