Skip to main content

Decoupling Latent 3D Facial Structure for Largely Masked Face Image Inpainting

By
Guosong Zhu; Zeping Ma; Zhen Qin; Yi Ding; Zhiguang Qin

Largely masked image inpainting has consistently remained a prominent and inherently challenging task in the field of image processing, particularly when it comes to face images. However, previous works have primarily focused on completing masked regions based on remaining pixels, thereby disregarding the valuable information concealed within facial structure and 3D space. To address this issue, we propose a novel inpainting framework specifically designed for face images. This two-stage inpainting framework consists of three components: a robust facial structure estimator, a preliminary inpainting module with latent facial geometric structure restoration, and a fine inpainting module with facial texture perceptual refinement. Specifically, the facial structure estimator with ID consistency constraint decouples 3D facial geometry and texture in canonical view from images, which are critical representations of 3D faces, encompassing all the key information required for 3D face reconstruction. The Transformer-based preliminary inpainting module integrates facial geometry restoration to complete facial priors, ensuring facial structure authenticity. The fine inpainting module leverages facial priors to guide up-sampling, with canonical texture constraints incorporated to improve facial texture coherence and perceptual realism. Experiments demonstrate that the proposed framework achieves favorable inpainting performance on CelebA-HQ dataset and competitive performance on FFHQ dataset, particularly in terms of facial identity preservation.

Read on IEEE Xplore