AI
Linguistic patterns in prompts can predict LLM performance without running inference
A new study shows that measurable linguistic features of prompts—such as syntax and sentence structure—reliably forecast how well language models will perform on requirements engineering tasks.
Prompt wording matters far more than many practitioners realize. A single change in phrasing can dramatically shift the quality of outputs from large language models, yet teams often lack a reliable way to evaluate prompts before actually running them—leading to expensive trial-and-error cycles.
A new empirical study from arXiv shows that this guesswork may be avoidable. Researchers found that certain measurable linguistic properties of prompts can predict LLM performance on requirements classification tasks with reasonable accuracy, opening the door to low-cost prompt selection before inference.
The problem: wasted compute on bad prompts#
In software engineering workflows, prompts guide critical tasks like requirements analysis, code generation, and artifact synthesis. When a prompt is poorly formulated, the outputs become unreliable—but practitioners typically have no principled way to spot problems before submitting queries to the model. This forces teams to either run expensive LLM calls repeatedly or rely on intuition when refining prompts.
How the researchers tested it#
The team generated 9,000 linguistically controlled prompt variants by systematically varying 30 linguistic metrics across 100 seed prompts. These were evaluated using five open-source LLMs on 625 annotated requirements, with performance measured on binary classification tasks targeting F1 score, F2 score, precision, and recall.
Regression models were trained via stratified 10-fold cross-validation with permutation-based significance testing to ensure robust results.
What they found#
Linguistic features proved to be significant predictors of LLM performance across all measured targets, with R² values ranging from 0.38 to 0.42 (p < 0.05). Several patterns emerged:
- Syntactic and morphosyntactic features (like sentence structure and grammatical complexity) carried the strongest predictive signal
- Cross-LLM predictors included compound dependency distribution, conjunction density, and word/sentence length—suggesting all models are sensitive to domain vocabulary density and structural complexity
- Lexical variety showed little relevance as a quality dimension, contrary to some intuitions
