Marco-Voice: New speech synthesis system blends voice cloning with emotion control

Researchers have developed Marco-Voice, an integrated speech synthesis system capable of generating expressive speech that combines voice cloning with fine-grained emotional control. The platform addresses longstanding technical hurdles in creating natural-sounding speech that stays true to a speaker's voice while conveying different emotional states.

How it works#

At the core of Marco-Voice is a speaker-emotion disentanglement mechanism. This allows the system to separate and independently adjust two dimensions of speech generation:

• Speaker identity: Preserves the unique characteristics that make a voice recognizable
• Emotional style: Modulates tone and delivery to express different feelings

The approach uses in-batch contrastive learning to achieve this separation. Additionally, the team developed a rotational emotional embedding integration method that enables smooth transitions between emotional states rather than abrupt shifts.

Building a training dataset#

To train and validate the system, the researchers created CSEMOTIONS, a dataset of high-quality emotional speech in Mandarin. It contains 10 hours of audio from six professional speakers performing speech across seven distinct emotional categories.

Results and impact#

Extensive testing shows Marco-Voice delivers competitive performance across both objective measurements (like spectral clarity) and subjective assessments from human listeners. The system demonstrates substantial improvements in speech clarity and emotional expressiveness compared to prior approaches. This represents meaningful progress in neural speech synthesis, particularly for applications requiring natural, controllable character voices or personalized audio interfaces.

The researchers have made their code and dataset publicly available to support further development in the field.