Crowdsourced Privacy Policy Labeling System Cuts Training Data Costs

Training deep learning models to understand privacy policies faces a major bottleneck: assembling large amounts of labeled training data is expensive and time-consuming. Researchers introduced Calpric, a system designed to tackle this problem by merging crowdsourcing, automated text processing, and active learning into a single pipeline.

How Calpric Works#

The approach combines three key strategies. First, automatic text selection and segmentation breaks privacy policy documents into manageable chunks, making it easier for non-expert crowdsourced workers to label them accurately. This preprocessing step is crucial—by simplifying what annotators see, untrained workers from platforms like Amazon Mechanical Turk performed comparably to law students while requiring less coordination overhead.

Second, active learning intelligently selects which samples to label, allowing the system to build comprehensive coverage of the input space with fewer examples. This cuts down the total labeling burden and improves data balance across categories.

Finally, the crowdsourcing framework brings these pieces together, enabling researchers to gather reliable annotations at scale without premium-cost expertise.

Results and Impact#

The full pipeline produced a dataset of 16,000 labeled privacy policy text segments across 9 data categories, with balanced positive and negative examples. Per-segment labeling costs came in at roughly $0.92–$1.71—a significant reduction compared to traditional annotation workflows. The resulting models achieved stronger accuracy across a broader range of data categories and generated more granular, fine-grained labels than earlier systems.

Research Context#

This research was submitted to the arXiv preprint server under the Computer Science/Computation and Language category. Note: the paper has since been withdrawn by the author.