LLMs Show Mixed Results in Analyzing Teacher Training Simulations

Researchers have tested large language models (LLMs) on their ability to automatically assess teacher candidates' responses in digital learning simulations—a task that could unlock more practical feedback for trainer educators at scale.

The Challenge of Open-Ended Responses#

Digital simulations offer teacher candidates a safe space to practice classroom interactions through conversational exchanges. However, these training environments typically feature open-ended questions that allow for nuanced and varied answers, making it difficult to evaluate performance using traditional automated tools. Researchers sought to tackle this problem by harnessing recent advances in language models.

Testing Multiple Models and Techniques#

The team evaluated DeBERTaV3 and Llama 3, experimenting with three different training approaches: zero-shot learning (no prior examples), few-shot learning (a handful of examples), and fine-tuning on domain-specific data. The researchers also assessed performance on two other models—Phi-4-mini and Qwen-3—to broaden their findings.

Results revealed substantial variation in how well each model identified specific teaching behaviors embedded in trainee responses. The variation wasn't random; rather, it depended significantly on which type of characteristic the model was being asked to detect.

Llama 3 Handles New Tasks Better#

A critical finding emerged when models encountered new characteristics they hadn't been trained to identify. DeBERTaV3's performance dropped substantially in these scenarios, whereas Llama 3 remained more stable and actually outperformed its competitor. This distinction matters for real-world deployment: teacher educators often need to introduce fresh evaluation criteria as simulations evolve or educational goals shift.

Implications for Educational Tech#

These results suggest that Llama 3 is the better choice for teacher simulation systems where trainers expect to continuously update what behaviors they're measuring. The research provides guidance for educators and technologists looking to integrate automated LLM-powered feedback into training platforms, helping move beyond manual analysis while maintaining flexibility as curriculum demands change.