The concept of a hyperparameter arises naturally from the hierarchical structure of Bayesian models. When a model parameter θ is given a prior distribution p(θ | α), the quantities α that parameterize this prior are called hyperparameters. They sit one level above the model parameters in the inferential hierarchy, governing how the prior behaves — how concentrated it is, where it centers its mass, and what shape it takes. The name reflects this hierarchy: "hyper" (from Greek, meaning "above" or "beyond") indicates that these parameters operate at a higher level than the ordinary model parameters.
In non-hierarchical Bayesian analysis, hyperparameters are typically fixed by the analyst based on prior knowledge, convenience, or convention. In hierarchical (multi-level) models, they may themselves receive prior distributions — called hyperpriors — and be estimated from the data, adding another layer to the inferential structure.
Parameters: θᵢ | α ~ p(θᵢ | α) (prior, governed by hyperparameter α)
Hyperparameters: α ~ p(α) (hyperprior)
Joint Posterior p(θ, α | x) ∝ [∏ᵢ p(xᵢ | θᵢ)] · [∏ᵢ p(θᵢ | α)] · p(α)
Examples Across Common Models
Hyperparameters appear throughout Bayesian analysis. In the Beta-Binomial model, the success probability θ has a Beta(α, β) prior, and α and β are the hyperparameters. Their values control whether the prior is concentrated near 0 (small α, large β), near 1 (large α, small β), or diffuse (both small). In Normal models, a prior μ ~ Normal(μ₀, σ₀²) has hyperparameters μ₀ (prior mean) and σ₀² (prior variance). In Dirichlet-Multinomial models, the concentration parameter α governs how peaked or flat the prior distribution over category probabilities is.
In Gaussian process models, the kernel hyperparameters — length scale, signal variance, and noise variance — control the smoothness, amplitude, and noise level of the function prior. These are often estimated by maximizing the marginal likelihood (type II maximum likelihood or empirical Bayes).
Three approaches exist for handling hyperparameters: (1) Fixed: choose values based on domain knowledge or convention (e.g., α = β = 1 for a uniform Beta prior). (2) Empirical Bayes: estimate hyperparameters by maximizing the marginal likelihood p(x | α) = ∫ p(x | θ) p(θ | α) dθ. This is fast but underestimates uncertainty. (3) Fully Bayesian: place a hyperprior on α and integrate it out. This is the most principled approach but requires additional computation.
The Role in Hierarchical Models
Hyperparameters reach their full importance in hierarchical Bayesian models, where data are organized into groups — patients within hospitals, students within schools, measurements within experiments. Each group has its own parameter θᵢ, and these group-level parameters are modeled as draws from a common distribution governed by hyperparameters. The hierarchical structure allows partial pooling: groups with little data borrow strength from the overall population, while groups with abundant data are estimated more independently.
The degree of pooling is controlled by the hyperparameters — specifically, by the ratio of the within-group variance to the between-group variance (the hyperparameter). When the between-group variance is small (a strong hyperparameter), groups are pooled heavily toward the grand mean. When it is large, groups are estimated nearly independently. The data determine this ratio through the posterior on the hyperparameters, producing an adaptive balance between complete pooling and no pooling.
θᵢ | μ, τ² ~ Normal(μ, τ²) (group-level parameters)
μ ~ Normal(m, s²) (hyperprior on the grand mean)
τ² ~ InvGamma(a, b) (hyperprior on between-group variance)
Sensitivity Analysis
Because hyperparameters control the prior, the posterior can be sensitive to their values — especially when data are sparse. Sensitivity analysis involves systematically varying the hyperparameters and observing how the posterior changes. If the conclusions are robust across a reasonable range of hyperparameters, confidence in the analysis increases. If conclusions are highly sensitive, this signals that the data are not sufficiently informative to overwhelm the prior, and the choice of hyperparameters must be defended carefully.
In practice, weakly informative hyperparameters — those that spread prior mass broadly but exclude physically impossible or implausible values — are often recommended as a compromise between full non-informativeness and strong prior specification. Andrew Gelman and colleagues have advocated for this approach in applied Bayesian modeling, particularly for variance parameters where the choice of hyperprior can substantially affect the posterior.
Hyperparameters in Machine Learning
The term "hyperparameter" has been borrowed by the machine learning community to refer to any quantity set before training — learning rates, regularization strengths, network architectures, batch sizes. While the usage is related, the Bayesian meaning is more precise: a hyperparameter is specifically a parameter of a prior distribution. The machine-learning usage is broader but less formally grounded. Bayesian optimization, a technique for tuning machine learning hyperparameters, uses Gaussian process priors on the objective function — bringing the full Bayesian hierarchy to bear on a problem that began with borrowed terminology.
"The choice of hyperparameters is not separate from the model — it is part of the model. Making hyperparameters explicit, and giving them priors, is one of the great advantages of the Bayesian approach." — Dennis Lindley, Bayesian Statistics: A Review (1972)
Example: How Hyperparameters Shape the Prior
A data scientist models customer conversion rates using a Beta(α, β) prior. She considers four hyperparameter settings and observes 18 successes in 25 trials.
→ Posterior: Beta(19, 8) → Mean: 0.704, 95% CI: [0.52, 0.88]
Jeffreys — Beta(0.5, 0.5): Prior mean = 0.500, Prior var = 0.1250
→ Posterior: Beta(18.5, 7.5) → Mean: 0.712, 95% CI: [0.53, 0.89]
Skewed — Beta(2, 5): Prior mean = 0.286, Prior var = 0.0255
→ Posterior: Beta(20, 12) → Mean: 0.625, 95% CI: [0.47, 0.78]
Concentrated — Beta(10, 10): Prior mean = 0.500, Prior var = 0.0119
→ Posterior: Beta(28, 17) → Mean: 0.622, 95% CI: [0.49, 0.76]
The Uniform and Jeffreys hyperparameters produce similar results because both encode minimal prior information — the 25 data points dominate. The Skewed prior (α=2, β=5) pulls the posterior toward lower values, reflecting a prior expectation that conversion rates tend to be low. The Concentrated prior (α=10, β=10) narrows the credible interval because it adds 20 pseudo-observations. These differences illustrate how hyperparameters are not mere technical details — they encode substantive assumptions about the problem.