Training Cybersecurity AI on a Fraction of the Data Typically Required

Achieving strong performance in security-focused AI doesn't require massive training datasets—a new study from arXiv demonstrates that careful domain adaptation can yield state-of-the-art cybersecurity models while dramatically cutting computational costs.

The Challenge of Specialized Security Models#

General-purpose large language models excel at natural language tasks but often miss the nuanced domain knowledge essential for analyzing threats, assessing vulnerabilities, and reviewing security documentation. Extending existing models with cybersecurity expertise has historically demanded enormous training runs. The researchers set out to prove that targeted, efficient adaptation could deliver better results with far fewer tokens.

How the Approach Works#

The team employed Domain-Adaptive Continuous Pretraining (DAP), a methodology that refines pretrained models on specialized data while distributing the computational load across multiple GPU clusters using a technique called Fully Sharded Data Parallel (FSDP). They adapted three models—Llama-3.1-8B, DeepSeek-R1-Distill-Qwen-14B, and Llama-3.3-70B-Instruct—using a curated corpus of 126 million words drawn from security standards, academic papers, and technical documentation.

The Results#

The Llama-3.3-70B-Instruct model after domain adaptation achieved the highest performance across all three security benchmarks:

• CTI-MCQ: 0.718 accuracy
• CyberMetric: 0.933 accuracy
• SecEval: 0.864 accuracy

Crucially, this was accomplished using just 118.8 million tokens—a 23-to-42-fold reduction compared to existing specialized security models like Llama-Primus-Base (trained on 2.77 billion tokens) and Foundation-Sec-8B (trained on 5 billion tokens).

Impact and Sustainability#

The approach underscores how leveraging scalable high-performance computing infrastructure enables efficient domain adaptation without the environmental or financial burden of large-scale retraining. The researchers argue this pathway supports practical security AI assistants for threat intelligence, vulnerability discovery, and documentation review—while aligning with broader goals around sustainable and responsible AI development.