ConcoLixir: Using LLMs as Discovery Oracles to Improve Python Concolic Testing

Researchers have introduced ConcoLixir, a system that augments Python concolic testing—a technique that blends concrete execution with symbolic constraint solving—by integrating large language models as "discovery oracles."

The Problem with Symbolic Testing#

Concolic testing is powerful for automated code exploration, but it hits predictable walls. Library calls can force symbolic variables to downgrade to concrete values, semantic operations like regex parsing and checksums resist symbolic solving, and exploration can get stuck on previously covered code paths.

How ConcoLixir Works#

Rather than replacing the symbolic solver or acting as a correctness oracle, the LLM plays a focused role: it serves as a discovery oracle that suggests promising directions when the symbolic engine falters.

The system operates in three modes:

• Generating initial seeds to kick off exploration
• Proposing concrete inputs when the solver reaches a dead end
• Targeting uncovered code when coverage plateaus

Every candidate input the LLM generates is executed concolically—meaning the actual execution results and path constraints guide all subsequent exploration. The system never trusts the LLM's reasoning directly; it validates everything through observed coverage and concrete constraint collection.

Results#

Testing across synthetic benchmarks, real-world code, and library targets, ConcoLixir improved mean line coverage over baseline concolic testing by 8.6, 15.1, and 17.0 percentage points respectively. Coverage gains were most pronounced near semantic barriers and library boundaries—exactly where traditional symbolic methods struggle. Remarkably, the entire evaluation cost just $1.63 in API charges.

The work demonstrates that bounded, symbolic-guided LLM assistance can meaningfully extend automated testing without requiring the solver itself to rely on neural networks.

Full paper on arXiv