LLM-Powered Text Annotation Amplifies Racial Biases Across Thousands of Tasks

Large language models are now widely deployed to automatically label and evaluate text in research, content moderation, hiring, and other high-stakes domains. But a comprehensive investigation spanning 19 models and over 4 million annotation decisions reveals a troubling pattern: these systems reliably reproduce racial stereotypes when assessing written passages.

How Bias Enters the Annotation Process#

The research team tested two distinct approaches to probe stereotyping. In the first, they embedded names traditionally associated with different racial and ethnic groups into texts and asked models to rate various attributes. In the second, they presented identical sentences written in either Standard American English or African American Vernacular English, then measured how the dialect choice alone affected model judgments.

The results were remarkably consistent across nearly all 19 models examined.

Stereotypes by Identity and Dialect#

Name-based patterns: When texts contained names linked to Black individuals, 18 out of 19 models rated the content as more aggressive and more gossipy than the same texts with names associated with white individuals. Asian names triggered a different stereotype profile—described as a "bamboo ceiling"—where 17 of 19 models judged those individuals as more intelligent, while simultaneously rating them as less confident and less sociable by overwhelming margins (18 of 19). Arab names elicited cognitive elevation paired with interpersonal devaluation. Across all four minority groups tested, 19 of 19 models consistently rated individuals as less self-disciplined.

Dialect differences: When the same text was presented in African American Vernacular English versus Standard American English, all 19 models rated it as significantly less professional (average gap: -0.774 on relevant scales), less reflective of an educated speaker (-0.688), more toxic (18 of 19), and more angry (19 of 19). These gaps reveal that subtle linguistic cues alone—independent of content—activate stereotypical judgments.

The Exception That Points to a Deeper Problem#

One unexpected finding emerged around hiring decisions based on applicant names: fine-tuned models systematically favored minority-named candidates, suggesting that safety training can overcorrect in some domains. However, this outlier highlights rather than resolves the core issue—it shows that bias mitigation remains selective and inconsistent.