NTNU's Hybrid Approach Combines Audio and Language Models to Score Spoken English Proficiency

Researchers at NTNU have developed a method for assessing spoken language ability by blending two complementary neural approaches—one designed to capture acoustic features and another to understand meaning. Their system placed runner-up in the 2025 Speak & Improve Challenge, with a root mean square error (RMSE) of 0.375, slightly behind the winning submission's 0.364 and well ahead of the competition's baseline at 0.444.

The Problem with Single-Modality Approaches#

Current spoken language assessment systems rely on either text-based or audio-based models, each with significant gaps. Traditional BERT-based systems work from automatic speech recognition (ASR) transcripts, capturing semantic content but missing the prosodic and phonetic nuances that reveal true speaking ability—things like rhythm, intonation, and pronunciation clarity. Conversely, acoustic models like wav2vec 2.0 excel at encoding these acoustic properties yet struggle to extract meaningful linguistic interpretation from raw sound.

The Hybrid Solution#

The NTNU team proposed integrating wav2vec 2.0 for acoustic feature extraction with Phi-4, a multimodal large language model (MLLM), using a score fusion strategy. This combination allows the system to leverage both the phonetic precision of audio analysis and the semantic understanding of advanced language models, compensating for where each falls short independently.

Results and Implications#

The second-place finish demonstrates the effectiveness of cross-modal integration for language proficiency assessment. The gap between RMSE scores—0.364 (winner), 0.375 (NTNU), and 0.384 (third place)—underscores how closely matched top approaches have become, suggesting that further improvements may require increasingly sophisticated fusion methods or additional modalities beyond speech and text.