New Benchmark Exposes Large Language Models' Struggle With Telecom Technical Tables

Researchers have developed a new benchmark to test how well large language models handle one of engineering's core challenges: reading and interpreting dense technical tables from 3GPP telecom specifications.

The Problem#

Despite their growing adoption in telecommunications engineering, most LLMs struggle significantly with 3GPP standards—the formal documents that define modern mobile network technologies. Much of the critical technical detail in these specifications lives in complex tables, yet little has been known about how effectively current models can actually extract information from them.

Introducing TeleTables#

A new dataset called TeleTables aims to fill this gap. The benchmark includes:

• 2,220 real tables drawn from 13 different 3GPP specifications
• Four distinct table formats to reflect real-world diversity
• 500 human-verified multiple-choice questions ranging from straightforward lookup tasks to multi-step reasoning challenges

What the Evaluation Revealed#

Researchers tested 20 open-weight LLMs across different architecture types: general-purpose models, multimodal systems, reasoning-focused variants, and table-specialized approaches. The results exposed two critical performance bottlenecks.

Without the table in context: When asked to answer questions based on memorized knowledge alone, no general-purpose model exceeded 41% accuracy. Domain-specific training appears essential for even baseline competence.

With the table provided: Performance improved dramatically, with top models surpassing 90%. However, this strength proved fragile. As reasoning requirements deepened, the breadth of evidence needed expanded, and table structure grew more complex, accuracy fell systematically—with a 32.2 percentage point gap between the easiest and hardest reasoning tasks.

Key Takeaway#

Pre-training on table-specific data from other domains offered no consistent boost to telecom table performance. Instead, the findings point to the importance of robust reasoning capabilities as the real foundation for reliable interpretation of intricate technical documentation. For LLMs to become trustworthy tools in telecom engineering, they need both domain grounding and genuine reasoning depth.