LLMs Show Promise but Fall Short as Standalone Goal Extractors for Requirements Engineering

Large language models are increasingly being applied to automate repetitive tasks in requirements engineering—the process of defining what software should do. A new paper from arXiv tests whether LLMs can automatically extract functional goals from software documentation, a key step in goal-oriented requirements engineering (GORE). The results suggest these tools work best as accelerators for human experts rather than full replacements.

The Three-Stage Extraction Pipeline#

The researchers designed a system that breaks down goal extraction into three sequential steps: actor identification (determining who or what performs actions), high-level goal extraction (capturing broad objectives), and low-level goal extraction (defining specific functional tasks). The pipeline chains multiple LLMs together, each equipped with carefully engineered prompts to guide the model through one phase of the process.

Key Technical Approaches Tested#

The team experimented with several prompting strategies to improve accuracy:

• Few-shot learning: providing in-context examples alongside the prompt to guide the model's output
• Zero-shot feedback loops: using a "generation-critic" mechanism where one LLM generates goals and another LLM critiques and refines them
• Similarity measurement: analyzing how closely in-context examples matched the input data to understand what helps performance

Results: Practical but Imperfect#

The system achieved 61% accuracy at low-level goal identification, the final stage where specificity matters most. The feedback-loop mechanism with zero-shot prompting outperformed standalone few-shot approaches. Interestingly, combining the feedback mechanism with few-shot examples did not improve results, suggesting that the bottleneck may lie in how the "critic" LLM is prompted rather than the number of examples provided.

Where This Heads Next#

The researchers concluded that their current approach is best suited to —helping human analysts work faster rather than replacing them entirely. They plan to integrate additional techniques including to ground extractions in actual documents and to make the model's reasoning more transparent. Refining both the quantity and quality of example shots is also expected to yield gains.