LLM-Generated Code Shows Signs of Degradation Under Long-Term Use

A study examining the long-term behavior of machine-learning-generated software has uncovered troubling patterns: applications built by large language models show measurable performance degradation when run continuously over extended periods, even when they pass initial functional tests.

The Research Gap#

While previous work has validated LLM-generated code for correctness, security, and basic robustness, researchers found virtually no examination of how these applications behave after days of sustained operation. This matters because real-world services run continuously, and small problems compound over time. The investigation addressed this blind spot by stress-testing AI-written applications under realistic conditions.

Testing Methodology#

Teams generated backend service applications in JavaScript, Python, and Rust using LLM platforms, based on scenarios from BaxBench (a benchmark suite). After validating functionality with standard tests, they subjected each application to 48-hour continuous workload execution, continuously measuring three key metrics:

• Memory usage — steady increase suggests potential leaks
• Response time — slowing requests indicate resource strain
• Throughput — declining capacity to handle requests

Statistical analysis using the Mann–Kendall test and Sen's slope estimator identified which trends were statistically significant across all tested application-environment pairs.

Findings: Memory as the Red Flag#

Memory consumption emerged as the strongest predictor of software aging. Most LLM-generated applications displayed statistically significant upward trends in memory use, pointing to common problems like unreleased resources and accumulating object references. Response times and throughput told a more mixed story, varying widely depending on the specific application and language, but memory degradation was consistent.

Static code analysis uncovered plausible explanations: the generated implementations contained patterns known to trigger gradual performance loss, such as improper cleanup of connections and cached data structures that grow without bounds.

A Larger Problem#

Notably, the researchers compared their findings against human-written reference implementations and discovered something unexpected: the aging patterns in LLM-generated code mirrored those found in manually developed systems. This suggests the issue is not unique to AI generation but reflects a broader challenge in application reliability that affects both humans and machines. However, the implication is clear: passing a functional test suite is insufficient assurance before deploying generated software into production environments where it will run continuously.