Conventional wisdom suggests that combining multiple forecasts reduces error and improves accuracy. But research into effort estimation for software projects reveals a more nuanced picture: merging predictions from different models can actually degrade performance, and any gains from ensemble methods are often marginal and statistically unreliable.
The Study Design#
Researchers examined four publicly available software-project datasets—Kitchenham (145 projects), Desharnais (77 projects), China (499 projects), and Maxwell (63 projects)—running 100 repeated train-test cycles on each. They combined three conventional modeling approaches: parametric regression, analogy-based estimation, and random forests, testing them individually and as ensembles using five different combination strategies.
The combination methods varied in sophistication, from simple equal weighting of all predictions to more advanced techniques like inverse-MSE weighting, nonnegative simplex combination, and Bates-Granger weighting with covariance shrinkage. All preprocessing and calibration were learned within training data to avoid contamination.
When Combining Fails#
The most striking finding emerged in the Kitchenham dataset, which included recorded human estimates alongside the algorithmic models. Equal weighting of all four sources caused mean test error to surge from 0.239 to 0.455—a significant degradation. The best alternative, a simplex rule that enforces constraint conditions, limited the damage to 0.250 but still underperformed the strongest individual model.
The problem stems from the dominance of one accurate source: when human judgment proved superior, averaging it with weaker statistical methods diluted its signal. Model errors across datasets showed substantial correlation (0.54 to 0.97), meaning the models tended to fail on the same projects, undermining the diversity that makes ensembles work.
Small and Uncertain Gains Elsewhere#
On the other three datasets, every combination rule lowered average error by 1.3 to 8.6%, with Bates-Granger shrinkage performing best. However, accounting for statistical overlap in the repeated splits revealed that these improvements were not reliably distinguishable from random noise. Only on a retrospective test using completion-year splits did ensemble methods retain consistent gains of about 5%.
The Bottom Line#
The researchers conclude that combining effort estimates is a conditional decision. Pooling predictions near a dominant forecaster can actively harm accuracy. Elsewhere, ensemble methods show only modest and uncertain benefits that disappear under rigorous statistical scrutiny. For practitioners, the implication is clear: before merging estimates, carefully assess whether your strongest individual model might be diluted rather than improved by the combination.