Open-source LLMs debate to auto-label sensitive mental health and safety data

Labeling sensitive information like mental health indicators and risky online behavior is expensive and complex—especially when the same piece of content might need multiple labels at once. A new approach proposed by researchers tackles this challenge by orchestrating debates between open-source language models to automatically enrich such datasets.

The challenge with multi-label annotation#

Traditional annotation efforts struggle with multi-label classification: marking content that falls into several categories simultaneously. Mental health data, for instance, may need simultaneous labeling for different life events and symptoms. Online safety data requires identification of multiple risk factors. Crowdsourcing these annotations is costly, time-consuming, and may burden annotators with distressing content.

Confidence-aware debate framework#

The researchers developed Confidence-Aware Fine-Grained Debate (CFD), which models how human annotators collaborate. Rather than asking a single model for a single answer, the system deploys multiple open-source LLMs that communicate and reason together about labeling decisions. Each model shares its confidence levels during this process, allowing the group to reach more nuanced conclusions. This fine-grained confidence exchange proved key to robust performance.

New datasets and real-world testing#

The team created two expert-annotated resources: detailed labels for life events and symptoms in a mental health wellbeing dataset, and a new dataset focused on sharenting—parents sharing information about their children online—and associated safety concerns. Across these tasks, CFD outperformed competing enrichment strategies and showed that the quality and variability of confidence signals matter for effectiveness.

Impact on downstream performance#

When researchers integrated the automatically enriched labels into downstream applications—without additional training—system performance improved significantly. Depending on the task and how enrichment was formatted, gains reached as high as 9.9 Macro-F1 points. This suggests that thoughtful integration of enriched indicators can substantially boost real-world NLP systems in mental health and safety domains.