New dataset reveals widespread "artificial hivemind" effect in language model outputs

A new study published on arXiv reveals a troubling tendency in language models: despite their apparent sophistication, they produce strikingly similar outputs across different systems when tasked with open-ended creative or brainstorming prompts. This phenomenon—termed the "Artificial Hivemind effect"—suggests that exposure to AI-generated content at scale could gradually flatten human thought and creativity.

The Infinity-Chat dataset#

Researchers introduced Infinity-Chat, a large-scale collection of 26,000 real-world, open-ended user queries drawn from actual interactions with AI systems. Unlike previous diversity studies limited to narrow tasks like random number generation, this dataset captures the full spectrum of how people actually engage with language models—from brainstorming sessions to creative writing requests. The researchers developed the first comprehensive taxonomy for categorizing open-ended prompts, organizing them into 6 broad categories that branch into 17 more specific subcategories.

The dataset includes 31,250 human annotations—25 independent evaluations per example—allowing researchers to understand both collective patterns and individual variation in how people respond to AI-generated text. This richness enabled a systematic investigation into how well AI systems align with the diverse preferences of real humans.

Homogeneity across and within models#

The analysis uncovered two interconnected problems. First, individual models exhibited intra-model repetition, consistently producing similar responses to varied prompts. More concerning, however, was the inter-model homogeneity: different language models, often built by different organizations with different architectures and training data, generated remarkably similar outputs. This convergence happens despite the apparent diversity of approaches in model development.

Misalignment with human variability#

A secondary finding highlighted a gap in AI quality metrics. Current approaches to evaluation—including reward models and AI judges used to score other models—perform worse when rating outputs for prompts that elicit differing preferences across individual human evaluators. While these systems maintain comparable overall quality scores, they struggle to capture the idiosyncratic nature of human taste, suggesting they may inadvertently reinforce homogeneity rather than preserve meaningful variation.