New framework helps AI coding agents block sensitive data leaks through natural language policies

A research team has proposed PolicyGuard, a data loss prevention system designed to catch sensitive information in user prompts before they reach large language models used for coding tasks. The framework operates as a filter that sits between the user and the AI agent, using another LLM to evaluate whether incoming text violates security guidelines.

The Challenge with Current Approaches#

Existing data loss prevention solutions have significant limitations. Most rely on rigid, pattern-matching rules that struggle to adapt to new threats, require specialized retraining to modify their behavior, or depend on opaque vendor-managed classifiers that teams cannot customize to their own needs. These approaches offer little flexibility for organizations with unique security requirements.

How PolicyGuard Works#

The core innovation is the policy-as-prompt paradigm: instead of hard-coding rules or retraining models, security teams write plain-language policy documents that describe what should and should not be allowed. A non-engineer can edit these policies without touching code or machine learning pipelines. An LLM reads both the incoming user prompt and the policy file, then makes a classification decision about whether to block the message.

Performance Results#

Testing across 2,000 multilingual prompts showed the system achieved a 96.5% effective block rate—the rate at which forbidden content was correctly flagged—while maintaining a 3.0% false positive rate on a frozen test set of 927 prompts. On a separate hidden holdout dataset of 217 prompts, the system achieved perfect 100% accuracy.

Comparative experiments demonstrated that expressing policies in natural language significantly outperformed the same rules encoded in JSON format (McNemar chi-squared = 31.58, p < 0.001). The approach also substantially beat zero-shot classification attempts.

Portability Across Models#

A key finding was that the same policy configuration could be used across four different large language models, achieving block rates between 86.4% and 96.5% without any adjustments, suggesting the approach generalizes well across different LLM architectures.

The research paper has been submitted to arXiv and is currently in withdrawn status.