Dice Loss Outperforms Cross-Entropy for Imbalanced NLP Tasks

Training on imbalanced datasets—where negative instances overwhelmingly dominate—remains a persistent challenge in natural language processing. A research paper on arXiv introduces dice loss as a solution, addressing a fundamental mismatch between how models are trained and how they are evaluated.

The Core Problem#

Most NLP tasks involving sequence tagging, entity recognition, and reading comprehension suffer from severe class imbalance. Background examples and easy-negative cases flood training datasets, yet standard cross-entropy loss treats all training examples equally. This creates friction: during training, every instance contributes uniformly to the objective, but at test time F1 scores emphasize correct predictions on the minority positive class. The model therefore optimizes toward a different goal than what evaluation metrics actually reward.

Dice Loss as an Alternative#

The proposed approach draws on the Sorensen-Dice coefficient and Tversky index—metrics that balance false positives and false negatives rather than pursuing accuracy above all. By switching from cross-entropy to dice loss, the training objective becomes more resilient to data imbalance. The researchers further enhance this method by dynamically weighting training examples to downweight easy-negative cases, reducing their outsized influence during training.

Empirical Results#

The method achieved state-of-the-art results across multiple benchmarks: part-of-speech tagging on CTB5, CTB6, and UD1.4; named entity recognition on CoNLL03, OntoNotes5.0, MSRA, and OntoNotes4.0. The approach also produced competitive performance on machine reading comprehension and paraphrase identification tasks. Analysis demonstrates that the weighted dice loss strategy narrows the gap between training objective and evaluation metric—aligning the model's learning signal with downstream performance.