Multimodal Style Control in Voice Conversion Using Textual and Acoustic Prompts

Conventional voice conversion (VC) systems primarily rely on reference speech or pre-trained speaker embeddings to transfer speaking style. However, their ability to control fine-grained acoustic characteristics is limited, as they can only utilize attributes that are implicitly embedded in the reference speech. In this paper, we propose a text- and speech-guided, style-controllable voice conversion framework that enables intuitive and interpretable manipulation of speaking style through the integration of two modalities. The proposed framework employs a speech prompt to preserve speaker identity and a natural language text prompt to provide high-level stylistic guidance. To integrate the multimodal cues, we train a flow matching decoder using a masked infilling based training objective, which learns to reconstruct acoustic features so as to match the desired target identity and style. Within the decoder, a cross-attention mechanism fuses acoustic features and semantic embeddings derived from natural language prompts, enabling the model to effectively utilize the information provided in the joint conditioning. Experimental results demonstrate that the proposed method achieves high-fidelity zero-shot conversion, preserving the target speaker's identity while enabling text-driven voice editing with enhanced expressiveness and intelligibility.