New Framework Reveals Significant Gaps in Computer Science Curriculum Coverage

Undergraduate computer science programs worldwide follow international curriculum standards that are revised roughly every decade, yet institutions have struggled to objectively assess how thoroughly they cover these guidelines or track shifts when new ones arrive. A team of researchers has now introduced a reusable measurement framework that addresses this long-standing challenge.

The Problem with Current Approaches#

Existing methods for evaluating curriculum alignment typically rely on broad topic modeling or manual expert review. These approaches suffer from several limitations: they often skip reliability checks, lack benchmarking against known standards, and typically measure coverage at only a single moment in time. This leaves program leaders without a dependable way to understand their actual compliance or plan for improvements.

A Multi-Stage Validation Pipeline#

The researchers designed a three-part process that separates candidate identification from confirmation. The method works as follows:

• Semantic retrieval uses automated tools to propose potential matches between courses and knowledge units from curriculum guidelines
• Large language model review validates each proposed match against explicit coverage criteria
• Expert validation provides human verification of the resulting curriculum map

The team benchmarked seven different retrieval approaches against expert judgments and discovered a critical finding: no automatic setup achieved satisfactory levels of both precision and recall. The best performer reached an F1 score of 0.55—and when tuned for better recall, it overstated actual coverage. This established that automated retrieval works best as a starting point for narrowing possibilities rather than as a final measurement tool.

Results from Real-World Testing#

Researchers applied the framework to one accredited computer science bachelor's degree program, measuring it against both the 2013 and 2023 versions of the international Computer Science Curricula standards. Two independent experts validated each mapping and reconciled their assessments, achieving substantial agreement (Cohen's kappa scores of 0.64 and 0.69).

The findings revealed significant coverage gaps under the newer 2023 standard: