Persona steering can degrade AI essay quality and bias automated grading, study finds

A team of researchers has investigated how activation steering—a technique that customizes large language models' responses by manipulating internal representations—affects both answer quality and automated scoring in educational settings. Their findings raise important concerns about deploying personalized AI systems in classrooms.

How the study worked#

The team examined seven distinct character traits encoded as activation vectors across three different language models, testing them on the ASAP-SAS benchmark for short-answer evaluation. They measured both the quality of generated responses and how these persona-steered models scored student submissions.

Impact on answer generation#

The results consistently showed that steering personalities into models degraded answer quality overall. The damage was far from uniform: open-ended writing tasks in English Language Arts suffered far more than factual science questions. Interpretive and argumentative prompts proved particularly vulnerable, experiencing degradation up to 11 times larger than simpler tasks.

Systematic biases in grading#

When the personalized models took on the role of graders, they exhibited predictable—but troubling—patterns. Scorers steered toward "evil" or "impolite" personas marked papers more harshly, while those reflecting "good" or "optimistic" traits were more lenient. These calibration shifts appeared roughly 2.5–3 times stronger for ELA assessments than science questions. Mixture-of-experts architectures showed particularly extreme biases, with shifts approximately six times larger than traditional dense models.

Implications for education#

The study represents the first systematic examination of personality-based activation steering in academic generation and assessment tasks. The researchers emphasize that deploying such personalized models in real educational environments requires careful attention to both task type and model architecture, with calibration strategies tailored to avoid introducing bias into student evaluation.