New Bilingual Benchmark Tests How Well AI Catches Medical Errors in English and Japanese

A team of researchers has unveiled MedRECT, a new evaluation benchmark that assesses how well large language models can identify and correct mistakes in medical documents written in English and Japanese. The work addresses a gap in AI evaluation: while LLMs have shown potential in healthcare applications, their capacity to handle clinical text errors remains poorly understood, especially outside English.

The Dataset and Task Design#

MedRECT breaks medical error handling into three distinct subtasks: spotting errors, isolating the sentences that contain them, and fixing the mistakes. The Japanese portion (MedRECT-ja) comprises 663 samples drawn from Japan's medical licensing exams, whereas the English version (MedRECT-en) contains 458 samples sourced from a separate collection called MEDEC. By sourcing from different materials, the two subsets naturally reflect different error types and distributions.

Model Performance and Key Findings#

Researchers tested 11 large language models under 17 different setups, mixing proprietary systems, open-source models, medical specialists, and various reasoning approaches. A standout result: Qwen3-32B achieved notably higher accuracy when operating in "thinking mode" compared to standard mode. The improvement was dramatic—sentence extraction accuracy climbed 24.5 percentage points on the Japanese subset and 10.3 points on the English set. Surprisingly, several general-purpose reasoning models outperformed all three dedicated medical-domain systems on error detection and sentence extraction tasks.

Performance gaps between languages emerged in the data: models generally posted lower scores on the Japanese samples, though the researchers cautioned that direct numerical comparison is misleading since the datasets differ in source material and error composition.

Fine-tuning and Real-World Impact#

When researchers applied LoRA fine-tuning—a lightweight adaptation technique—models improved at extracting error sentences and matched reference corrections more closely across all three evaluation metrics in both languages. This suggests that tailoring models to medical correction tasks, even with modest computational investment, yields measurable gains.

The open-source release of MedRECT, along with accompanying code and data, creates a shared foundation for future research into medical error correction and reasoning across multiple languages.