Study reveals homogeneity bias is unevenly rooted across LLMs—and methodology matters

A new study maps how large language models reproduce homogeneity bias—the tendency to characterize members of marginalized groups as more alike than those in dominant groups—revealing that this form of stereotyping is not uniformly robust across different models and settings.

The Research Scope#

Researchers evaluated seven open-weight instruction-tuned LLMs ranging from 7 to 20 billion parameters, testing how they portray different demographic groups across varying inference conditions. The team manipulated two key variables: the decoding parameters (temperature and top-p settings in a 5×5 grid) and the way identity was signaled to the model (either explicit demographic labels or names coded with racial associations).

What the Data Shows#

The findings were decidedly uneven. When using explicit demographic labels, Hispanic and Asian Americans emerged as consistently portrayed with greater internal homogeneity than White Americans in six of the seven models tested—and this effect persisted across all temperature and top-p combinations. However, portrayals of African Americans and gender-based homogeneity bias varied substantially from model to model, with some showing the bias in opposite directions.

A stricter, cell-level analysis confirmed that Hispanic and Asian homogeneity bias proved robust, while the weaker signals for African American and gender-based bias did not hold up under scrutiny across all models.

How Methodology Shapes Results#

The research took an additional step by repeating the experiment using a names-based approach, where group identity was signaled through racially distinctive surnames rather than labels. This shift revealed something striking: the names paradigm largely reproduced the Hispanic and Asian homogeneity bias observed with labels, but it produced the opposite pattern for African Americans. Models generated outputs showing Black-coded surnames as less homogeneous than White-coded names—a reversal entirely absent when using explicit labels.

This finding underscores a critical lesson: the methodology used to measure bias can fundamentally alter which biases are detected and even reverse their direction, suggesting that researchers and developers must carefully consider how they operationalize demographic categories when assessing model behavior.