New Benchmark Reveals Major Gaps in How AI Drug Discovery Models Are Evaluated

Generative AI models promise to accelerate drug discovery by automatically designing novel candidate compounds. Yet a fundamental problem persists: the metrics used to judge these models' performance often fail to predict whether their output actually works in the real world of medicinal chemistry.

Researchers from the HEDGEHOG project have now documented this disconnect. They tested 22 generative models designed for molecular discovery and found that standard evaluation metrics showed almost no correlation with what truly matters—how many proposed compounds survive practical drug development filters.

The Problem With Current Measures#

Existing benchmarks rely on metrics like diversity scores and similarity measures to rank molecular generators. These tests are fast to run and mathematically clean, but they tell an incomplete story. A model might score well on traditional metrics while producing compounds that are chemically infeasible, structurally flawed, or impossible to synthesize. This mismatch wastes computational resources and delays the identification of genuinely promising drug candidates.

A Practical Six-Stage Filter#

HEDGEHOG reframes evaluation as a real drug discovery workflow. The system filters candidate molecules through six successive stages:

• Preprocessing — cleaning and standardizing raw molecular data
• Physicochemical screening — checking properties like molecular weight, solubility, and lipophilicity
• Structural validation — flagging known chemical hazards and ensuring molecules are chemically sound
• Synthesis feasibility — assessing whether compounds can actually be made in the lab
• Molecular docking — simulating binding to the intended drug target (KRAS G12D in their test case)
• Pose and interaction analysis — examining whether the 3D shape and atomic contacts make sense

Real-World Test Results#

The researchers ran each of 22 models three times, requesting 1,000 candidate molecules per attempt. The results were sobering: after passing through all six stages, molecule survival ranged from zero to 127 compounds per run, depending on the model. Crucially, conventional metrics did not predict these outcomes. "None of the standard metrics showed a significant association with final survival after correction for multiple testing," the team found.