New Benchmark Tests How Well AI Models Handle Software Engineering Tasks Beyond Code

While large language models have proven adept at writing and understanding code, their ability to tackle the broader world of software engineering—requirements documents, issue descriptions, design discussions—has received far less scrutiny. A new benchmark called SELU (Software Engineering Language Understanding) fills that gap.

A Comprehensive Evaluation Framework#

Researchers have created what they describe as the first comprehensive benchmark for assessing how well LLMs handle non-code software engineering tasks. The benchmark encompasses 22 distinct natural language understanding exercises drawn from real development environments, including issue trackers and discussion forums. These tasks range across multiple problem types: classification (such as distinguishing functional from non-functional requirements), regression (estimating implementation effort), Named Entity Recognition (identifying key entities in documents), and Masked Language Modeling (predicting missing text).

Testing and Results#

The team evaluated the benchmark against 22 open-source models of varying sizes and architectures, along with two proprietary models tested through zero-shot and few-shot prompting approaches. Performance was measured using standard metrics including F1-macro, SMAPE, F1-micro, and accuracy scores, with statistical comparisons drawn via Bayesian signed-rank testing.

A striking finding emerged: fine-tuned models consistently outperformed other approaches, showing both strong average performance and minimal variance across different tasks. More unexpectedly, domain-specific adaptation—where models undergo additional pre-training on code—did not substantially improve results and sometimes actually hindered performance on tasks involving developer communication artifacts.

Implications for AI Development#

These findings suggest that a one-size-fits-all approach to LLM specialization for software engineering may be misguided. The results indicate that general fine-tuning strategies can be more effective than domain narrowing, particularly when the target work involves written communication between developers rather than executable code.