Change Only What You Need: Dual Dynamic Hypergraph-Driven Fusion for GAN Inversion

Most existing GAN inversion methods aim to strike a good trade-off between fidelity and editability. However, a pre-trained GAN latent space only encodes information from in-domain regions while the input image often involves out-of-domain regions. As a result, extending the original latent space by encoding out-of-domain regions to improve fidelity will negatively affect the editability of the model. To address this, we propose a novel dual dynamic hypergraph-driven fusion (DHFusion) method, which consists of a dual dynamic hypergraph-CAM network (DH-Net) and an editing-driven fusion network (EF-Net). Specifically, DH-Net first employs the differential activations between the initial inverted and edited images to dynamically construct dual hypergraphs from the perspectives of short-range and long-range spatial dependencies. In this way, high-order relationships between attribute-relevant regions are effectively modeled, enabling our model to generate an accurate and comprehensive edit-aware mask for locating the edited regions. Subsequently, EF-Net leverages this mask as weights to perform editing-driven feature-level fusion between the original image and the initial edited image, generating high-fidelity edited images with the reduced ghosting effect. Extensive quantitative and qualitative experiments demonstrate that our method outperforms several state-of-the-art methods. Our work clearly shows the potential of dual dynamic hypergraphs for GAN inversion. Our code is available at https://github.com/Huang-KT/DHFusion