Fine-tuning and data augmentation help LLMs better extract medical jargon from patient records

Electronic health record systems now allow patients to access their clinical notes through OpenNotes, but the density of medical terminology often leaves them unable to grasp key information. A new study examines how large language models can be trained to automatically extract and rank the most critical medical jargon terms—making those notes more comprehensible to their readers.

Researchers evaluated both commercial and open-source LLMs against a curated set of 90 expert-annotated EHR notes. Their goal: determine which combination of techniques most effectively identifies and prioritizes clinically relevant medical terms in a low-resource environment.

Testing prompting and training strategies#

The team systematically compared multiple approaches: general versus structured prompts, zero-shot versus few-shot prompting, fine-tuning on gold-standard data, and augmentation using GPT-4o to generate additional training examples. To understand how dataset size affected outcomes, they scaled up their augmented training sets from 10 to nearly 10,000 examples and validated results using 10-fold cross-validation.

Performance was measured using two metrics: F1 score and Mean Reciprocal Rank (MRR), applying both relaxed string matching and stricter Jaccard Index matching criteria. The researchers also performed detailed error analysis to understand failure modes.

Open-source models excel on small, curated data#

When evaluated on the gold-standard dataset using Jaccard-based matching, open-source models showed the strongest performance. DeepSeek 8B achieved the best F1 score at 0.431, while BioMistral 7B reached the highest MRR at 0.577. However, this advantage did not hold under relaxed matching criteria—commercial models like GPT-4o continued to outperform open-source alternatives even with augmentation or fine-tuning applied.

Key findings on techniques#

Several patterns emerged across experiments:

• Few-shot prompting offered no benefit over zero-shot with unmodified models
• Prompting style itself was consequential, suggesting that how instructions are framed significantly affects extraction quality
• Fine-tuning on a modest gold-standard dataset improved performance across the board