AI
Study reveals tension between two faithfulness measures in AI reasoning explanations
Researchers find that optimizing large language models for one type of explanation integrity doesn't always improve another, suggesting AI transparency may require multiple complementary approaches.
A new research paper examines a fundamental challenge in understanding how large language models explain their reasoning: whether their explanations actually reflect what's happening inside the model, or merely what appears plausible to observers.
Two competing approaches to checking truthfulness#
Researchers distinguish between two ways of measuring whether chain-of-thought explanations (step-by-step reasoning shown by AI systems) genuinely represent the model's internal behavior. The contextual approach tests explanations by changing the inputs or the reasoning trace itself to see if the model's conclusions shift accordingly. The parametric approach directly examines the model's learned weights and internal knowledge to verify whether explanations align with what the model actually knows.
Until now, these two testing paradigms had been studied largely in isolation, with only informal comparisons between them.
A unified framework for testing both#
The researchers created FaithMATE, a single interface that allows them to optimize models toward either faithfulness paradigm and measure how improvements in one area translate to the other. They tested their approach across three different language models, two datasets, and six different faithfulness metrics.
Key findings reveal asymmetries#
The results showed that the two paradigms don't behave equally when models are optimized:
- Parametric gains transfer broadly: When researchers optimized models to improve parametric faithfulness, improvements appeared consistently across both testing approaches.
- Contextual improvements are uneven: Optimizing for contextual faithfulness produced more variable and unpredictable benefits.
- Metrics within a paradigm diverge: Even within the contextual approach alone, improving performance on one metric didn't reliably improve performance on others—suggesting each metric captures a different aspect of faithfulness.
