New lightweight approach infuses sentiment knowledge into language models

A team of researchers has developed a lighter-weight method for incorporating sentiment understanding into pre-trained language models, moving away from the computationally expensive mask and reconstruction approaches used in prior work.

The two-task approach#

The researchers' method, called KESA, leverages two complementary sentiment-aware auxiliary tasks. The first task, sentiment word cloze, trains models to identify correct sentiment expressions within text when given the overall emotional tone as context. The second task, conditional sentiment prediction, works in reverse—it predicts the document's overall sentiment based on individual word-level emotional signals.

This bidirectional design ensures models learn from multiple angles. The researchers also tested different label combination strategies to harmonize distinct types of sentiment labels across tasks, based on the hypothesis that diverse information sources yield deeper semantic understanding.

Results and implementation#

Testing demonstrated that the approach consistently beat standard pre-trained models on sentence-level sentiment analysis benchmarks. Notably, KESA showed additive gains when combined with existing knowledge-enhanced training methods, suggesting the techniques are complementary rather than redundant.

The implementation prioritizes simplicity and efficiency compared to more complex post-training schemes, making the approach practical for real-world deployment. Code and data have been made publicly available to support reproducibility and adoption.