Explainable AI
Dataiku's Kiji Inspector Uses Sparse Autoencoders to Demystify AI Agent Decisions
An open-source framework combines sparse autoencoders and causal testing to reveal what's actually driving LLM agents' tool selection choices.
InfoQ
Deciphering why large language model agents make the decisions they do remains one of AI's thorniest problems. Current approaches—prompt engineering, behavioral testing, even chain-of-thought reasoning—rarely expose the underlying cause; instead, they offer plausible-sounding narratives that may or may not reflect the model's true reasoning.
Dataiku's 575 Lab, an open-source initiative focused on responsible AI, is tackling this opacity head-on with Kiji Inspector, a tool designed to illuminate exactly how agents choose among available functions or knowledge sources.
How Sparse Autoencoders Expose Hidden Logic#
The breakthrough rests on a three-decade-old technique reimagined for agent interpretability: sparse autoencoders. Where standard autoencoders compress information through a bottleneck, sparse autoencoders do the opposite—they expand hidden model activations into a wider feature space while forcing roughly 95% of those dimensions to remain inactive. This selective firing reveals which specific features the model actually relies on.
Here's the process:
- Extract activation patterns from a model layer during agent reasoning
- Expand those activations into 10,000 dimensions, forcing sparsity through regularization
- Reconstruct the original activation to minimize error
- Identify which of the ~668 active features matter most for tool selection decisions
The catch: you need access to open-source model weights. Kiji Inspector works with Nemotron, Gemma, and other open models—but not closed systems like OpenAI's.
Training on Contrastive Pairs#
Rather than training on generic prompts, the team built a million-sample dataset (500,000 paired examples) comparing correct versus incorrect tool choices in near-identical scenarios. A simple example: "How do I reset my password?" should route to a knowledge base, whereas "I tried to reset my password three times" might need a support ticket. Both requests look similar on the surface; the decision-making features differ sharply.
This contrastive design makes it possible to isolate which features actually distinguish between tool choices, versus mere correlations in the data.

