NLP System Automatically Links Security Requirements to Development Backlogs

Software teams working in heavily regulated industries face a persistent challenge: security requirements often remain implicit in product backlogs, buried within brief, unstructured task descriptions. A new research effort demonstrates how natural language processing can surface these hidden security concerns and connect them to applicable compliance rules.

The Problem#

Development teams managing continuous software delivery in regulated sectors—from finance to healthcare—must balance velocity with security compliance. However, backlog items typically lack explicit security annotations. Engineers must manually dig through vague descriptions to determine whether a task touches on sensitive operations, data handling, or system integrity. This leaves critical security gaps and makes it difficult to apply relevant compliance standards proactively.

The Solution#

Researchers have built an NLP-driven system that automates this detective work. The approach combines two components: a security-relevance classifier that identifies which backlog items have security implications, and a retrieval-augmented generation (RAG) pipeline that pulls applicable security requirements from policy documents and standards like the CIS Benchmarks.

The classifier was trained on a dataset of 288 backlog items that nine security practitioners manually reviewed and labeled for security relevance, achieving strong inter-rater agreement (Fleiss' κ = 0.787).

Performance Results#

The classifier achieved an F2 score of 0.774 on in-distribution test data—optimized for recall, meaning it errs toward flagging items as potentially security-relevant rather than missing real ones. When tested on five established benchmarks using zero-shot inference, it achieved a mean G-measure around 0.65, matching or exceeding most open-source and classical ML baselines.

A preliminary trial with two practitioners evaluated the full RAG pipeline against real enterprise backlogs, using company security policies and CIS Benchmarks as the knowledge source. Of 24 requirements retrieved by the system, 12 were rated as relevant (4/5 or higher) by the evaluators.

Why It Matters#

This work offers an early demonstration that automation can reduce the manual burden on security teams and help embed compliance thinking earlier in the development cycle. By flagging security-relevant tasks and surfacing applicable rules, the system supports what the authors term —treating security requirements discovery as an ongoing part of delivery, not a gate at the end.