New comprehensive guide outlines the foundations of differentiable programming

Recent advances in artificial intelligence have been underpinned by a convergence of factors: scaling up models, accumulating larger training datasets, leveraging more powerful hardware, and harnessing the power of differentiable programming. This emerging computational approach makes it possible to automatically optimize program behavior by computing gradients through entire systems—even those containing branching logic and complex data structures.

What is differentiable programming?#

Differentiable programming represents a shift in how we design software for machine learning. Rather than simply applying differentiation tools to existing code, it emphasizes architecting programs with end-to-end gradient computation in mind. This allows practitioners to use gradient-based optimization methods to adjust program parameters directly, adapting the system's behavior based on data.

Unifying perspectives across disciplines#

The paradigm draws from multiple established fields: automatic differentiation (computing derivatives efficiently), graphical models (representing complex dependencies), optimization (finding good parameter values), and statistics (understanding data distribution). A key insight is the duality between optimization and probabilistic perspectives—viewing the same computational structures through both lenses reveals deep connections.

Probabilistic interpretation and uncertainty#

When programs are made differentiable, they implicitly induce probability distributions over their execution paths and outputs. This probabilistic layer opens a pathway to quantify uncertainty in a system's predictions and decisions, moving beyond point estimates toward more principled uncertainty measures.

Reference#

The full paper is available on arXiv.