New Method Reduces Cost of Labeling Privacy Policies for AI Training

Building machine learning systems that can accurately interpret privacy policies has been hampered by the expense and complexity of assembling large, high-quality labeled datasets. A research team has introduced Calpric, a framework that brings together crowdsourced annotation, automated text processing, and active learning to tackle this problem efficiently.

The Challenge#

Training deep learning models on privacy policies requires substantial labeled training data, which traditionally demands significant investment. Expert annotators—such as lawyers or law students—can provide accurate labels but are costly to employ at scale. This creates a bottleneck that limits the breadth and depth of models trained on such documents.

How Calpric Works#

The system operates through three complementary strategies:

• Automated preparation: Smart segmentation and text selection break privacy policies into manageable chunks, making the labeling task accessible to non-expert annotators
• Crowdsourced annotation: Platform workers from services like Amazon Mechanical Turk can now label segments reliably, despite lacking legal training
• Active learning: Rather than labeling randomly, the approach strategically selects which samples to annotate, maximizing coverage of the input space with fewer labels

This combination reduces disagreement between annotators, cuts annotation costs to roughly $0.92–$1.71 per segment, and improves balance across data categories.

Results#

The approach produced a curated dataset containing 16,000 labeled privacy policy segments spanning nine data categories, each with balanced positive and negative examples. The resulting models perform with greater accuracy across a wider variety of data types and deliver more granular, fine-grained classifications than prior work in the field.