New benchmark tests LLMs' ability to infer complex string-transformation rules

A team of researchers has introduced PBEBench, a fresh approach to evaluating how well large language models can perform inductive reasoning tasks. Rather than assessing domain-specific skills like mathematics or coding, this benchmark isolates reasoning capability itself by framing problems as a sequence of string transformations—a concept borrowed from historical linguistics.

How the benchmark works#

The core task is elegantly simple: given input and output strings, models must infer a cascade of rewrite rules that transform the inputs into the outputs. Think of it as learning a pattern of transformations and then applying those rules consistently. The researchers built an automated pipeline that generates problems with adjustable difficulty levels, allowing large-scale evaluation without the risk of data contamination that plagues many other benchmarks.

Two versions of the benchmark were created: PBEBench-Lite for efficiently sorting models by capability level, and PBEBench itself, designed to match the complexity of programs that historical linguists construct when analyzing language change.

Performance gaps reveal limitations#

The findings paint a sobering picture of current model capabilities. Models equipped with test-time compute or long chain-of-thought (LCoT) reasoning substantially outperform those without these enhancements. Yet even the best-performing systems—including gpt-oss-120b, the strongest open-source model tested—struggle on harder instances. When cascade lengths reach 20 to 30 steps, solve rates plummet below 5%, falling far short of what would be needed for realistic linguistic reconstruction work.

Why this matters#

By abstracting away domain specifics, PBEBench provides a cleaner window into whether LLMs can actually reason through multi-step logical problems or merely recognize patterns within familiar domains. The gap between models with and without expensive reasoning techniques highlights that current approaches may rely more on scaling compute than on genuine improvements in reasoning architecture.