Authorship Science Reveals What LLM Personalization Benchmarks Miss

Researchers evaluating large language model personalization—the task of making AI write in a specific person's unique style—have discovered that many existing benchmarks fail to measure what they claim to measure. By grounding their evaluation in authorship science theory, they've uncovered a significant performance gap that conventional metrics entirely miss.

The Evaluation Problem#

Stylistic personalization differs fundamentally from generic task adaptation. It requires an LLM to capture the distinctive writing patterns, word choices, and rhetorical habits that make one author distinguishable from another. Yet existing benchmarks lack foundation in actual authorship verification science—the discipline devoted to detecting who wrote a given text.

This theoretical gap matters because, as the research demonstrates, metric choice itself can determine conclusions. Without principled measurement standards, different evaluation methods offer wildly different pictures of the same methods' performance.

A Theory-Grounded Measurement Framework#

The team employed three distinct measurement traditions to evaluate four personalization approaches across 50 authors and 1,000 text generations:

• LUAR: A trained authorship verification model grounded in authorship science
• LLM-as-judge: An AI evaluator with decoupled trait matching
• Classical stylometrics: Function-word analysis, a time-tested technique from linguistics

Crucially, only LUAR provided calibrated baselines that give scores absolute meaning—a human ceiling of 0.756 and a cross-author floor of 0.626 that define realistic performance bounds.

The Hidden Authorship Gap#

When measured against these theory-grounded baselines, all four personalization methods scored far below the floor, ranging from 0.484 to 0.508. This reveals an authorship gap invisible to uncalibrated metrics: none of the methods reliably capture individual authorial style as well as random cross-author matching would suggest.

The three evaluation approaches produced near-zero pairwise correlations (|r| < 0.07), confirming that without theoretical grounding, the choice of metric determines what researchers conclude about the same underlying systems. An LLM judge might declare a clear winner while LUAR finds no meaningful differentiation.