What makes code examples work in AI prompts? Naming clarity beats formatting

Researchers have systematically analyzed why providing code examples alongside prompts helps large language models generate better code—and discovered that the answer lies less in sophisticated problem-solving insights and more in practical details like clear variable names and explicit input/output demonstrations.

How In-Context Learning Works#

In-Context Learning (ICL) is a technique where developers embed code examples directly in prompts to help LLMs learn from demonstrations. While this approach consistently improves code generation performance, the underlying mechanisms remained murky. The new research, published on arXiv, uses controlled experiments on competitive programming problems and real-world repository tasks to pinpoint what actually drives these improvements.

What Doesn't Help (Much)#

The study found that LLMs struggle to extract generalizable problem-solving strategies from provided code samples—whether those examples solve similar problems or come from existing repositories. Simply showing a working solution doesn't teach the model how to approach novel challenges. This suggests that reflection-based learning has significant limitations when applied to code generation.

What Actually Makes Examples Effective#

The research reveals that LLMs benefit substantially from explicit contextual information embedded in examples:

• Input and output demonstrations
• Required helper functions
• Namespace and library information

These concrete details appear to anchor the model's understanding in practical context rather than abstract patterns.

The Naming Effect#

Perhaps the most striking finding concerns variable naming. Through systematic mutations of test examples, researchers discovered that identifier clarity is far more critical than code formatting or low-level implementation choices. When descriptive variable names were replaced with opaque alternatives, performance dropped by as much as 30 percentage points—a dramatic decline driven by a single dimension of code style.

Importantly, LLMs prioritize semantic clarity over stylistic conventions. Adhering to surface-level naming patterns (like camelCase vs. snake_case) matters far less than ensuring that names actually convey what a variable represents.