AI
How confident should LLMs be when generating code? A study in uncertainty
Researchers find that language models express varying levels of certainty when completing code, with type systems and programming languages significantly shaping model confidence.
Language models trained on code are increasingly used to help developers write software faster. But how confident should these AI systems actually be in their suggestions? A new study examines this question by analyzing how uncertain code-generating large language models become across different programming contexts.
Measuring model confidence through perplexity#
The research measures model confidence using intrinsic metrics—specifically perplexity, entropy, and mutual information—that gauge how certain an LLM feels when predicting each token of code. These metrics are easier to compute than traditional downstream evaluations and may serve as early warning signals for hallucinated or incorrect code suggestions. The team evaluated code perplexity across multiple models and programming languages using 2,254 code files sampled from 881 GitHub projects.
Key findings: Languages and type systems matter#
The results reveal striking patterns. Strongly-typed languages like Java exhibit significantly lower perplexity—meaning the models feel more confident generating them—while dynamically-typed languages show higher uncertainty. Scripting languages tend to trigger even greater perplexity, with Shell consistently ranking as the highest across nearly all conditions.
This suggests that type systems provide structure that helps LLMs make more predictable choices. Language-specific rankings remain stable across different training corpora, indicating the pattern isn't tied to a particular dataset or evaluation approach.
The minimal role of comments#
Adding code comments to the evaluation corpus increased perplexity modestly, but the overall language ranking remained largely unchanged. This indicates that documentation doesn't substantially shift the fundamental confidence profiles LLMs exhibit for different programming languages.
Implications for code completion#
The findings help practitioners gauge whether LLM-based code completion is suitable for their projects. By understanding how language choice, model selection, and code characteristics influence model confidence, developers can better assess the reliability of AI-generated completions in context.
