New system improves automated speaking tests by understanding images and grammar errors in detail

Automated systems that evaluate spoken English from non-native speakers often miss important context and gloss over the specific types of grammar mistakes test-takers make. Researchers have now developed an improved scoring model that tackles both these blind spots.

The Problem with Current Systems#

Existing automated speaking assessment (ASA) platforms struggle with two key limitations. They don't fully leverage visual content (like images or example answers) when judging whether a speaker answered the question properly—even though these cues should inform relevance judgments. Additionally, these systems typically rely on surface-level grammar checks that fail to categorize different kinds of errors, missing opportunities to provide detailed feedback on language use.

A Dual-Solution Approach#

The new hybrid model introduces two major upgrades. A multifaceted relevance module brings together the original question, any associated images, exemplar responses, and the speaker's actual answer to create a richer picture of content relevance. Rather than treating these inputs separately, the system integrates them for a more holistic assessment.

The second enhancement focuses on grammar. Researchers derived fine-grained error features by applying advanced grammar error correction (GEC) techniques and detailed classification schemes. This allows the system to identify not just whether a speaker made a mistake, but what kind of mistake it was—a distinction that matters for meaningful evaluation.

Results and Impact#

Testing and ablation studies confirm that both components work together to meaningfully improve how the system evaluates content relevance, grammatical accuracy, and overall speaking performance. By incorporating richer, more nuanced signals—particularly visual context and detailed error analysis—the approach points toward more comprehensive and useful automated speaking assessments for language learners.