New AI System Neutralizes Speaker Accents in Real-Time While Preserving Voice Identity

A team of researchers has unveiled PHONOS, a real-time speech processing system designed to neutralize accent characteristics while maintaining speaker identity. The work addresses a significant gap in speaker anonymization technology: existing systems typically remove vocal timbre but leave accent markers intact—a vulnerability that can expose a speaker's native language or geographic origin.

The Problem with Current Anonymization#

Traditional speaker anonymization tools mask distinctive vocal qualities but fail to conceal linguistic markers that betray a person's background. These residual accent cues narrow the pool of potential speakers a voice could belong to, undermining privacy protections. The challenge lies in removing such cues without fundamentally altering the speaker's voice quality or naturalness.

How PHONOS Works#

The system combines several techniques to achieve accent neutralization in a streaming context. It first generates reference utterances that preserve the speaker's original timbre and speech rhythm but replace non-native speech sounds with native-accented equivalents. To accomplish this, the researchers employ silence-aware dynamic time warping (DTW) alignment paired with zero-shot voice conversion.

These reference utterances then train a lightweight accent translator that operates in real-time with minimal latency—requiring at most 40 milliseconds of look-ahead. The translator maps non-native speech segments to their native equivalents using dual loss functions: cross-entropy and CTC (Connectionist Temporal Classification) losses, enabling the model to learn accurate phonetic conversions.

Performance and Results#

Testing demonstrated that PHONOS achieves an 81% reduction in non-native accent confidence while maintaining subjective listening quality. Human raters reported accentedness ratings aligned with the system's technical improvements. Additionally, the processed speech moved away from the original speaker in embedding space, suggesting weaker linkability if an attacker attempts re-identification using speaker embedding models.

The system runs with end-to-end latency under 241 milliseconds on a single GPU, making it practical for real-world streaming applications.

Implications for Privacy#

This work points toward a more comprehensive privacy model for speech: one that masks not only a speaker's distinctive vocal signature but also linguistic features tied to identity. As speaker recognition and identification systems grow more sophisticated, such multi-layered anonymization approaches may become essential for confidential communication.