Weakly-Supervised Prompt-Guided Privacy Enhancement for Soft-Biometric Attributes in Facial Images

Widely deployed face recognition poses privacy risks: images collected for identity verification are repurposed to infer sensitive soft-biometric attributes such as age, gender, and race, which may constitute nonconsensual data exploitation. Yet most prior privacy enhancement methods are strongly supervised, over-reliant on attribute labels and classifier-specific losses, and offer limited flexibility beyond predefined protection targets. To address these limitations, we propose the Information Fusion Secure Network (IFSNet), a weakly supervised, prompt-driven image-level privacy enhancement framework. It obscures sensitive soft-biometric attributes while preserving identity utility, and does not require per-image sensitive-attribute annotations during training or inference. First, we leverage external textual prompts to guide additive offsets in the latent space, thereby producing privacy-enhanced images that reflect the injected prompt semantics rather than the original attribute cues. IFSNet learns these offsets via a contrastive objective. Second, to retain identity, we use a geometric treatment of face embeddings in a local tangent space, which yields a simple regression objective. Finally, we optimize an image-specific privacy offset for each input while keeping the encoder and generator frozen, and use $\lambda $ to control the manipulation strength. Under a white-box threat model, we assess privacy on five face datasets using Normalized Privacy Obfuscation. Additionally, we evaluate verification performance on three identity-labeled datasets in terms of equal error rate and false non-match rate at fixed false match rates. The overall trade-off is quantified by the Privacy-Gain Identity-Loss Coefficient. Across diverse settings, IFSNet reduces sensitive-attribute inference while retaining identity-verification utility and provides a favorable privacy–utility trade-off.