New Framework Generates Better Debug Logs by Learning from Program Execution

Software logging—the practice of recording system state during execution—is fundamental to debugging and maintenance. However, most current approaches to automatically generating log statements rely on static code analysis alone and produce their output in a single step, without accounting for how the program actually behaves at runtime. This limitation becomes especially problematic as large language models increasingly consume logs to assist with debugging and code repair.

Moving Beyond Static Analysis#

Existing log generation tools typically evaluate their results by comparing generated logs to logs written by developers, treating human-authored logs as the ultimate benchmark. This assumption breaks down in the LLM era, where logs serve purposes beyond human readability. A log might read well to a developer but fail to provide the information an AI model needs to localize defects or suggest repairs.

How ReLog Works#

Researchers have introduced ReLog, an iterative framework that takes a fundamentally different approach. Rather than static analysis, ReLog harnesses language models to generate logging statements, then executes the code, observes the results, and repeatedly refines the logs based on runtime information. This cycle continues until the logs better support downstream tasks like pinpointing bugs and generating fixes.

The team evaluated ReLog on a benchmark derived from Defects4J, a well-known collection of real software bugs. In direct scenarios where source code is available, ReLog achieved an F1 score of 0.520 and successfully repaired 97 defects. In indirect settings where code cannot be examined, it reached an F1 score of 0.408. Across both configurations, ReLog consistently surpassed baseline approaches.

Implications for Debugging#

The work demonstrates that logs should be optimized for their end use—whether that's helping humans understand code or assisting LLMs with repair tasks—rather than simply matching developer conventions. Testing across multiple language models showed the framework generalizes well, and ablation studies confirmed that both iterative refinement and compilation repair are essential components.