New benchmark exposes how language models struggle with non-Western kinship reasoning

A new research benchmark highlights a substantial weakness in large language models: they struggle considerably when reasoning about kinship systems that use cultural rules rather than pure biological logic.

Researchers at arXiv have introduced KinshipQA, a procedurally-generated evaluation suite designed to test how well LLMs can perform multi-hop reasoning across different kinship classification systems used by cultures around the world. The work, which evaluates six LLMs, reveals that models experience a dramatic accuracy drop when shifting from biological reasoning to culturally-marked categorization.

The Performance Gap#

Testing on seven distinct kinship systems—Eskimo, Sudanese, Hawaiian, Iroquois, Dravidian, Crow, and Omaha—the researchers found that LLMs achieve 40.9% lower accuracy when handling the five non-descriptive systems compared to biological composition tasks. This gap appears consistently across all tested models and grows more severe with reasoning depth. At 5–6 reasoning hops, accuracy on culturally-overridden kinship plummets to just 10.6%, while biological reasoning over identical chains maintains 58.6% accuracy.

Why Cultural Rules Are Harder#

The benchmark tests up to six hops of reasoning and includes a tunable simulator designed to prevent models from simply memorizing training examples. When humans are given explicit rule access, they reach 89% accuracy—showing the tasks are inherently solvable. LLMs, however, manage only 50.7%, suggesting the difficulty lies in how models apply cultural overrides rather than the logical complexity itself.

Two follow-up experiments point toward underlying causes. When researchers substituted kinship terms with invented words while keeping system labels intact, accuracy improved by 6.1%—implicating the influence of familiar English phrasing. A second probe found that explicitly prepending override rules helped models reason about "skewing systems" (Crow and Omaha) by 17.1%, but hurt performance on other systems, suggesting models lack an ingrained prior for when cultural rules should apply.

Broader Implications for Compositionality#

The findings demonstrate a meaningful limitation in how LLMs compose abstract reasoning across culturally-informed domains. The work challenges assumptions that passing simpler benchmarks like CLUTRR—which covers only one kinship system—reflects broader compositional reasoning capability. Code and evaluation materials are available on GitHub.