New Method Trains AI Models to Catch Program Logic Bugs Through Competitive Code Variants

Large language models struggle with code understanding tasks that demand careful reasoning about how programs actually behave, rather than just pattern-matching syntax. Researchers have now developed a synthetic training method to address this gap by having AI agents play a reasoning game against each other.

The Semantic Inequivalence Game#

The approach, detailed in a new paper on arXiv, centers on what researchers call a semantic inequivalence game (SInQ). Two AI agents work in opposition: one creates variations of real code snippets that are semantically distinct (meaning they behave differently despite looking similar), while the other tries to find test inputs that expose where those variants diverge. This adversarial setup forces both agents to sharpen their understanding of program logic—the generator must craft genuinely different behaviors, and the evaluator must develop a keen eye for spotting them.

The synthetic training data produced by this process can then be used to fine-tune language models on tasks that typically expose their weaknesses in code reasoning.

Results Across Multiple Benchmarks#

The team evaluated the method on several code understanding tasks. On a Python builtin identifier swap benchmark—a notoriously difficult test where models must understand that swapping built-in function names changes program behavior—the approach achieved substantial improvements over baseline LLM performance.

Notably, the method also transferred to a cross-language vulnerability detection task. Models trained purely on Python data using this synthetic approach improved at spotting security vulnerabilities in C/C++ code, suggesting the reasoning skills learned are generalizable across languages.

Open Release#

The researchers have released both the code to reproduce their experiments and the synthetic training datasets, making the approach available for others to fine-tune their own models.

Paper: Program Semantic Inequivalence Game with Large Language Models (arXiv)