New Framework Tracks How LLMs Handle Harmful Requests—Revealing Compliance, Refusal, and Dangerous Escalations

Most safety assessments of large language models boil down to simple pass-fail metrics—does the model refuse, or does it comply? A new study from researchers including Mengya Hu takes a deeper view, examining how harm shifts between what a user asks for and what the model actually outputs.

The Gap in Current Safety Testing#

Traditional safety evaluations typically report binary outcomes: attack success rates, refusal rates, or binary harm classifications. These metrics obscure a crucial question: does a model's response actually reduce, maintain, or amplify the risk present in the original request? The research addresses this gap by analyzing paired human-labeled records of both prompts and responses across four harm categories—sexual content, self-harm, hate speech, and violence—each tagged at severity levels (Safe, Low, Medium, High).

Key Findings: Three Risk Trajectories#

The paired analysis reveals three distinct patterns:

• 61% of responses reduce harm compared to the prompt, suggesting the model successfully mitigates risk.
• 36% preserve severity, leaving the harm level unchanged.
• 3% escalate harm, introducing new risks where the original request posed less danger.

The escalation cases break down into two mechanisms: benign prompts that trigger unrequested harmful details, and answers that stay relevant to the request but veer into higher-severity territory than the prompt warranted.

Category-Specific Patterns#

Sexual content shows the highest persistence of harm in the sample, driven largely by direct compliance at the same severity level rather than drift from initially innocuous inputs. This pattern differs meaningfully across the other harm categories and points to distinct behavioral modes in how models handle different types of risky content.

The Helpfulness-Harmlessness Tradeoff#

A joint relevance analysis uncovers a telling tension: models that escalate harm by complying tend to stay focused and relevant to the user's intent, whereas refusals often fall back on generic disclaimers with minimal informational value. This suggests a tradeoff between being genuinely helpful and maintaining safety boundaries—a finding that complicates the simple framing of "safe" versus "unsafe" outputs.