Netflix Releases Agentic Tool for Causal Inference With Built-In Peer Review

Netflix has released an open-source agentic system designed to streamline observational causal inference (OCI), a statistical method for understanding cause-and-effect relationships in data. Rather than automating analysis away from human experts, the workflow is built to remove tedious work while keeping decision-making power with the analyst.

How the System Works#

The workflow operates through an actor-critic loop. A human analyst begins by setting up a templated Jupyter notebook and an analysis plan. The actor agent then uses this plan to generate a specification, populate notebook parameters, and run the analysis. Once complete, a separate critic agent reviews the results and either approves them (with full confidence, with caveats, or rejects them), then suggests changes to the specification.

This two-agent design has a clear purpose: the actor handles execution while the critic catches mistakes and flags gaps that the actor might miss. Netflix framed the causal inference task as target trial emulation—essentially, finding the best-possible A/B test to answer the research question at hand.

Real-World Impact#

In a case study, Netflix demonstrated the workflow's value by analyzing how a new entertainment feature (such as games) affects user retention. A baseline approach using a standard LLM simply ran linear regression, producing an estimated effect. When the oci-agent workflow ran the same analysis, it produced a result that was "just 25% of the baseline"—flagging issues like early adopter bias and failed diagnostic tests that the simpler method had missed.

Rather than burying these findings, the system exposes every step: analysis plans, specifications, plots, and runnable notebooks that domain experts can inspect and reproduce later. Netflix calls this approach process audits—making reasoning transparent so analysts can catch errors before relying on conclusions.

Why This Matters#

Causal inference is notorious for hidden pitfalls. Analysts can easily introduce bias, apply the wrong statistical method, or misinterpret results. Netflix's design recognizes that agents are good at running repetitive checks and sensitivity analyses, but humans must retain authority over framing questions and judging whether results make sense. By automating drudgery and flagging potential issues through the critic agent, the workflow lowers barriers to rigorous analysis.

The system was tested against the Atlantic Causal Inference Conference (ACIC) competition dataset and proved competitive with benchmark systems. The is now available on GitHub, and Netflix published on their technical blog.