LLMs show demographic bias when left without explicit instructions on whose perspective to adopt

When large language models perform text annotation tasks without explicit demographic guidance, they gravitate toward particular demographic profiles—a finding that has important implications for how AI systems evaluate subjective content like politeness and offense.

The Problem with Silent Defaults#

Human annotators bring their own backgrounds to labeling tasks. A phrase like "yo, bro" might strike an elderly woman as rude, while a teenager might view it as friendly banter. This demographic variation in human judgment is a real phenomenon that affects how we should interpret and use labeled datasets. Yet when LLMs conduct annotations, we often don't know whose perspective—if anyone's in particular—they're adopting.

How Researchers Tested LLM Demographics#

A team of researchers took two complementary approaches. First, they examined what demographic profiles LLMs naturally replicate when given neutral annotation prompts. Then they compared the results against two other prompt conditions: prompts that included irrelevant demographic details ("you are an annotator who lives in house number 5") and prompts that explicitly specified demographics ("You are a 45-year-old man and an expert on politeness annotation").

The study used the POPQUORN dataset—a controlled corpus specifically designed to measure how human annotators with different demographic backgrounds rate the same text for politeness and offensiveness. This dataset had not previously been analyzed for LLM behavior.

What the Research Found#

When demographics were made explicit in prompts, the models showed measurable shifts in their annotations tied to gender, race, and age. This contradicts earlier work that found little to no demographic effect in LLM-based annotation, suggesting that demographic prompting can indeed steer model behavior in meaningful ways.

Why This Matters#

The findings highlight a gap between what we assume LLMs do (produce neutral, objective judgments) and what actually happens. Since subjective tasks like assessing politeness or offense inherently reflect values and perspectives, understanding which demographic lens an LLM defaults to becomes crucial for building fair and transparent AI systems. The research underscores the need to either explicitly acknowledge the demographic assumptions built into annotation systems or design them to capture demographic variation deliberately.