Active Learning and AI Transparency Cut Software Debt Detection Labeling Work in Half

Detecting architecture technical debt—the subtle structural compromises developers acknowledge in comments and issue trackers—remains one of the most laborious classification tasks in software engineering. A new study demonstrates how intelligent sampling and interpretable machine learning can dramatically reduce the annotation burden.

The Challenge of Hidden Technical Debt#

Developers often document architectural shortcuts and design decisions they'd like to revisit later. These self-admitted technical debts (SATD) live in code comments, commit messages, and project tracking systems. While spotting them manually is straightforward, scaling detection across large codebases demands thousands of expert judgments. The problem is especially acute for architecture-level debt, which tends to be abstract and highly context-dependent—harder to spot than simpler code smells or naming issues.

A Three-Layer Detection Strategy#

The research team built their approach on three complementary techniques. First, they refined an existing dataset of architecture debt examples from Jira issues and extracted a set of reliable keywords to use as an initial filter. This keyword-based pass identified over 103,000 candidate issues across 10 open-source projects, dramatically narrowing the search space.

Next, they applied active learning with multiple query strategies to pick the most informative samples for human review. The Breaking Ties strategy—which selects instances where the classifier is most uncertain—performed best, achieving an F1-score of 0.72 while reducing labeling effort by roughly half compared to random sampling.

Making Predictions Transparent#

To build confidence in automated classifications, the team applied explainability methods. Both SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) were tested. Expert review found both useful, though LIME was generally preferred for clarity and usability. This transparency layer helps practitioners understand why the model flagged a particular issue as architectural debt.

Why This Matters#

The combined approach cuts human annotation work substantially while maintaining strong detection performance. For organizations managing sprawling codebases, the ability to surface architectural compromises with less manual overhead opens the door to better tracking and remediation of technical debt at scale.