LLMs excel at comparative emotion ratings, not direct scales

Automatically labeling datasets remains one of the biggest practical hurdles when building machine learning systems for specialized tasks. While large language models have made real strides in handling categorical annotation—think labeling text as positive or negative—they struggle more with tasks that demand numerical precision, such as predicting emotional intensity on a continuous scale.

The annotation challenge#

Researchers at the University of Stuttgart have identified a pattern that mirrors how humans themselves behave: when asked to rate emotions on a numerical scale, both people and LLMs tend to perform less reliably than when asked to make comparative judgments. This insight led the team to test whether comparative annotation methods—specifically best-worst scaling, where a model ranks items from best to worst—might work better for machine-generated labels.

Testing three approaches#

The study compared three different ways of using LLMs to annotate emotion intensity:

• Direct rating scales: asking the model to assign a number to represent emotional intensity
• Pairwise comparisons: having the model choose which of two texts expresses stronger emotion
• Best-worst scaling: instructing the model to rank multiple items by intensity in a single judgment

What the results showed#

Best-worst scaling produced the most consistent annotations. When the researchers then trained a transformer-based regression model on these automatically generated labels, the resulting system performed nearly as well as one trained on hand-annotated data. This suggests that the way annotation is framed—comparative rather than absolute—can meaningfully improve the quality of machine-generated training datasets.

The work opens a path for reducing human labor in creating regression datasets for domains where direct measurement proves difficult.