Skip to main content

Bridging Semantic Logic Gaps: A Cognition-Inspired Multimodal Boundary Preserving Network for Image Manipulation Localization

By
Songlin Li; Zhiqing Guo; Yuanman Li; Zeyu Li; Yunfeng Diao; Gaobo Yang; Liejun Wang

Existing image manipulation localization (IML) models mainly rely on visual cues, but ignore the semantic logical relationships between content features. In fact, the content semantics conveyed by real images often conform to human cognitive laws. However, image manipulation technology usually destroys the internal relationship between content features, thus leaving semantic clues for IML. In this paper, we propose a cognition inspired multimodal boundary preserving network (CMB-Net). Specifically, CMB-Net utilizes large language models (LLMs) to analyze manipulated regions within images and generate prompt-based textual information to compensate for the lack of semantic relationships in the visual information. Considering that erroneous textual descriptions caused by LLM hallucinations may degrade IML accuracy, we propose an image-text central ambiguity module (ITCAM). It assigns weights to the text features by quantifying the ambiguity between text and image features, thereby reducing the negative impact of unreliable textual cues. We also propose an image-text interaction module (ITIM) that aligns visual and text features using a correlation matrix for fine-grained interaction. Finally, inspired by invertible neural networks, we propose a restoration edge decoder (RED) that mutually generates input and output features to better preserve boundary-related information in manipulated regions. Extensive experiments show that CMB-Net outperforms most existing IML models. Our code is available on https://github.com/vpsg-research/CMB-Net

Read on IEEE Xplore