Expert study finds XAI methods helpful but limited for AI safety certification

As AI regulation tightens globally—notably with the EU AI Act on the horizon—certifying that machine learning systems are trustworthy and safe has become critical. Yet the opaque nature of modern ML models creates a fundamental obstacle to applying traditional certification approaches. A new study investigates whether explainable AI (XAI) methods might bridge this gap.

What the research found#

Researchers conducted 15 interviews with specialists in both XAI and certification to assess how well explainability tools could support safe AI development and formal approval. The findings painted a nuanced picture: XAI techniques can indeed serve as a practical tool during development, particularly for uncovering hidden biases and failure modes in models. However, their role in actual certification faces significant limits because regulatory approval demands complete and verifiable information about how systems work—something current XAI methods cannot reliably provide.

The certification challenge#

Certification of complex technical systems has long relied on thorough documentation, testing, and traceability. Machine learning's black-box problem disrupts this model: even when XAI methods generate insights, those insights may be incomplete, approximate, or difficult to validate independently. Regulators and auditors cannot easily verify that an explanation tool has captured all relevant aspects of a model's decision-making.

Implications for AI governance#

As regulatory frameworks like the EU AI Act begin to take effect, the gap between what XAI can deliver and what certification requires will likely become a practical bottleneck. The study suggests that while explainability remains valuable for internal development—catching bugs and biases before deployment—a broader shift in ML architecture, documentation standards, and oversight may be needed to meet certification demands.