Code LLM Memorization Tests Break Down at Scale, Researchers Find

A new study from arXiv challenges the prevailing narrative around memorization in large language models built for code, finding that the methods used to diagnose this problem produce unreliable results as model scale increases.

The Problem with Existing Tests#

Researchers examined how well current diagnostic techniques actually work at detecting memorization—the degree to which models simply reproduce training examples rather than learning underlying patterns. Traditional approaches, such as encoder-style probes that introduce perturbations like synonym replacement or dead-code insertion, consistently fail to expose memorization in larger models, even when testing on datasets known to contain training data. Decoder-style probes relying on log probability measures show equally degraded performance.

This represents a fundamental crack in the foundation of recent memorization research, which has frequently concluded that code LLMs rely heavily on memorization. If the tools designed to measure this phenomenon don't work reliably, how confident should we be in those conclusions?

Separating Two Distinct Phenomena#

The key insight from the research involves distinguishing between two previously conflated concepts: representation load and memorization. Representation load refers to a model's ability to adapt its internal representations to handle different surface forms of the same problem—say, renaming variables or restructuring code slightly. Memorization, by contrast, means the model has simply stored and reproduced exact training examples.

By applying invertible mathematical transforms to numeric problems, the researchers isolated these factors independently. The findings revealed that larger encoders can successfully manage substantial representation load while still converging on correct solutions, even when the exact training example differed significantly.

Why This Matters for Real Applications#

The practical implications reach beyond academic metrics. In actual software engineering contexts, what matters is whether a model can produce functional code under varying conditions—not whether a specific solution appeared in the training set. If an LLM generates working code by generalizing from learned patterns (representation load) rather than retrieving memorized snippets, the distinction between the two becomes less consequential for real-world utility and generalizability in agentic AI systems.