New Framework Uses Cognitive Comparison to Pick Best Model for Sentiment Analysis

Choosing the right machine learning model for sentiment analysis is challenging when multiple algorithms show different strengths across various evaluation metrics. A new framework attempts to address this by systematically comparing models using weighted criteria to determine which performs best for a given task.

The CPC-CMS Framework#

The Cognitive Pairwise Comparison Classification Model Selection (CPC-CMS) framework applies expert-informed decision-making to rate and compare machine learning classifiers. Rather than relying on a single performance measure, the system evaluates models across eight different criteria:

• Accuracy, precision, and recall — standard classification metrics
• F1-score and Specificity — harmonic mean and true negative rate
• Matthews Correlation Coefficient (MCC) and Cohen's Kappa — measures of agreement accounting for chance
• Efficiency — computational performance

The framework calculates relative weights for these criteria based on expert judgment, then builds a decision matrix that scores competing models against these weighted benchmarks.

Models Tested#

Researchers evaluated seven baseline classifiers ranging from classical algorithms to modern deep learning approaches:

• Traditional methods: Naive Bayes, Linear Support Vector Classification, Random Forest, Logistic Regression
• Gradient boosting: Extreme Gradient Boosting (XGBoost)
• Neural networks: Long Short-Term Memory (LSTM), ALBERT (a lightweight Transformer variant)

Results and Broader Application#

Testing on three public social media datasets, the framework identified ALBERT as the top performer when time constraints are ignored. However, when computational efficiency is factored in, performance rankings shift — no single model consistently outperforms others across all trade-offs.

The team validated their approach against other multi-criteria decision methods, including the Analytic Hierarchy Process, TOPSIS, and MOORA, with results remaining consistent despite differing aggregation values. The framework is designed to extend beyond sentiment analysis to other classification problems across different domains.