Emotion Embeddings for Speech Synthesis Fail Human Perception Tests

Researchers have identified a fundamental flaw in how the speech synthesis field measures emotional expressiveness in generated audio. The dominant approach—comparing emotion embeddings between reference and synthesized speech—doesn't align with human judgment, despite performing well on classification tasks.

The Current Approach and Its Promise#

Speech generation systems that require emotional prosody, such as expressive synthesis and voice conversion, rely on automated metrics to assess how well artificial voices capture emotional nuance. The prevailing solution measures cosine similarity between embeddings produced by emotion encoders like emotion2vec, operating under the assumption that these latent representations reliably isolate emotional features from other acoustic properties.

Why the Method Breaks Down#

The researchers challenged this assumption through controlled adversarial experiments and human perception studies. While emotion encoders achieve strong classification accuracy, their learned representations suffer from representational limitations that allow linguistic content and speaker identity to interfere with emotional signals. This interference becomes especially problematic in zero-shot similarity evaluation, where the metric must judge emotional alignment without training on the specific speech samples being compared.

The mismatch appears systematic: the embeddings appear to reward acoustic mimicry—superficial similarity in sound—rather than genuine emotional communication. This distinction matters because two utterances might sound acoustically similar while conveying different emotional intent, or vice versa.

Human Alignment and Research Implications#

Validation tests measuring alignment with human listeners revealed significant divergence. The metric's acoustic vulnerability means it fails to capture what listeners actually perceive as emotionally expressive. This suggests the field may be optimizing for the wrong target when using these embeddings for model training or evaluation.

The findings point to a broader challenge: ensuring that automated speech quality metrics genuinely measure what humans care about, rather than simply capturing correlates that happen to be statistically detectable in learned feature spaces.