Chain Rule (Probability)

Every joint distribution, no matter how complex, can be written as a product of conditional probabilities. The chain rule makes this factorization precise. Given events A₁ through Aₙ, their joint probability equals the probability of the first event, times the probability of the second given the first, times the probability of the third given the first two, and so on. No approximation is involved — the identity is exact, following directly from the definition of conditional probability applied repeatedly.

This seemingly simple algebraic fact has far-reaching consequences. It is the principle that allows Bayesian networks to represent high-dimensional distributions compactly, the mechanism behind autoregressive language models, and the foundation on which sequential Bayesian updating rests.

Derivation from Conditional Probability

The chain rule follows by repeatedly applying the definition of conditional probability. Recall that for any two events A and B with P(A) > 0:

For three events, apply the definition twice. First factor the joint into a pair:

The pattern extends to any number of variables by induction. At each step, one variable is peeled off the joint and turned into a conditional. The process terminates when a single marginal probability remains. Because each step is an exact application of the definition of conditional probability, the final product is an identity — it holds for all valid probability distributions.

The Two-Event Case and Bayes' Theorem

The simplest instance of the chain rule is the product rule for two events:

Setting these two factorizations equal and solving for P(A | B) yields Bayes' Theorem directly. In this sense the chain rule is logically prior to Bayes' Theorem — Bayes is what you get when you equate two different chain-rule decompositions of the same joint probability.

Connection to Bayesian Networks

A Bayesian network encodes a joint distribution over many variables as a directed acyclic graph. The chain rule guarantees that any joint distribution can be written as a product of conditionals. The network's structure then introduces conditional independence assumptions that simplify these factors.

In the full chain rule, each successive conditional depends on all previous variables. A Bayesian network exploits the graph structure to drop irrelevant parents — for example, C might depend only on A (not B), and D might depend only on B and C (not A). These simplifications are not approximations; they are assertions about the structure of the problem. When they hold, the number of parameters needed to specify the joint distribution drops dramatically.

This is why the chain rule is sometimes called the "engine" of Bayesian networks. Without it there would be no principled way to decompose a joint distribution into local, modular factors that can be estimated and reasoned about independently.

Role in Sequential Bayesian Updating

When data arrive sequentially — observations x₁, x₂, …, xₙ — the chain rule provides the mathematical justification for updating beliefs one observation at a time:

If the observations are conditionally independent given the parameter θ (the standard i.i.d. assumption), every conditioning set reduces to just θ, and the likelihood simplifies to a product of identical terms. But even when observations are dependent — as in time series, spatial data, or sequential experiments — the chain rule shows how to build the full likelihood from one-step-ahead conditionals.

This decomposition is the basis of online learning algorithms, particle filters, and Kalman filters. At each time step the agent absorbs one new observation, updates its posterior, and carries that posterior forward as the prior for the next step. The mathematical legitimacy of this "update and propagate" strategy rests entirely on the chain rule.

Autoregressive Models and Language

Modern autoregressive language models generate text by sampling one token at a time, each conditioned on all previous tokens. This is a direct application of the chain rule:

No independence assumptions are made — each token can depend on the entire preceding context. The model learns each conditional factor P(wₜ | w₁,…,wₜ₋₁) from data using a neural network. The chain rule guarantees that the product of these learned conditionals defines a valid joint distribution over sequences, regardless of the quality of the learned factors.

Continuous Variables

The chain rule extends naturally to continuous random variables by replacing probabilities with probability densities:

This form is essential in multivariate statistics and probabilistic modeling. Hierarchical Bayesian models, for instance, express complex joint densities over parameters and hyperparameters as cascading conditionals — hyperpriors feeding into priors feeding into likelihoods — each factor specified independently. The chain rule guarantees their product defines a coherent joint distribution.

A Worked Example: Medical Screening

A patient undergoes two independent diagnostic tests for a condition. The chain rule structures the joint probability of the disease status and both test results:

The conditional independence assumption — that the two tests provide independent information given the true disease status — allows the third factor to drop its dependence on Test₁. This is precisely the kind of structural simplification that Bayesian networks formalize. Without the chain rule to begin with, there would be no principled starting point from which to introduce such simplifications.

Information-Theoretic Perspective

Taking logarithms of both sides of the chain rule yields an additive decomposition of information. The joint entropy (or log-probability) decomposes into a sum of conditional entropies:

This identity is the foundation of information theory's connection to probability. It tells us that the total uncertainty in a collection of variables equals the uncertainty in the first, plus the residual uncertainty in the second after knowing the first, and so on. Mutual information, KL divergence, and all the other information-theoretic quantities used in Bayesian model comparison ultimately trace back to this decomposition.

"Probability theory is nothing but common sense reduced to calculation." — Pierre-Simon Laplace, Théorie analytique des probabilités (1812)

Example: Predicting Equipment Failure in a Factory

A manufacturing plant monitors two sequential warning indicators on its assembly line: a temperature sensor (T) and a vibration sensor (V). Historical data show that 5% of production runs develop a critical fault (F). When a fault is developing, the temperature sensor triggers 80% of the time. When the temperature sensor has triggered and a fault is developing, the vibration sensor also triggers 90% of the time.

Setting Up the Chain

We want P(T triggers, V triggers, Fault) — the joint probability that both alarms fire and a real fault is present. The chain rule decomposes this neatly:

Plugging in the Numbers

So 3.6% of production runs will see both sensors fire during a genuine fault. The chain rule let us build this joint probability from three simpler conditional relationships, each of which we could estimate from maintenance logs independently. Without the chain rule, we would need to observe all three events simultaneously in enough cases to estimate the joint probability directly — a much harder task when faults are rare.