Study Reveals How LLMs Generate Plausible Wrong Answers in Educational Quizzes

Researchers investigating how large language models understand and represent student misconceptions have found that LLMs apply markedly different strategies when generating distractor answers—the intentionally incorrect but believable options on multiple-choice tests—depending on the subject domain.

The Challenge of Modeling Misconceptions#

Accurate representation of how students reason incorrectly is fundamental to building effective educational AI systems. A research team examined this problem by studying how LLMs tackle the specific task of distractor generation, which demands producing wrong answers that remain plausible enough to mislead students unfamiliar with the correct material. To structure their analysis, the researchers developed a taxonomy of reasoning strategies informed by educational learning science and patterns observed in actual LLM outputs.

Subject-Specific Differences in Reasoning#

The study applied this taxonomy to analyze LLM reasoning traces across mathematics and science multiple-choice questions. The results revealed a striking contrast:

• Math domain: Models demonstrated a more sophisticated, diagnostic-quality approach. They worked through the correct solution, identified where student errors typically occur, simulated those specific errors, and then selected plausible wrong answers as distractors.

• Science domain: Models relied more heavily on a surface-level strategy, generating distractors primarily based on semantic similarity to the correct answer rather than modeling genuine student misconceptions.

Key Failure Patterns#

The researchers identified two dominant failure modes that undermined distractor quality. Most commonly, models either struggled to generate the correct solution in the first place, or they discarded plausible distractor candidates during the selection phase. Both issues degraded the alignment between LLM-generated and human-authored distractors.

Anchoring Improves Performance#

A simple intervention proved effective: when the correct answer was included in the prompt, models improved their distractor alignment with human-written versions by 6.4% on average. This finding underscores how essential it is to anchor distractor generation to the ground truth, particularly for sustaining the model's grasp of where and how students typically go astray.