Deep Ensembles¶

Deep ensembles quantify predictive uncertainty by training multiple independently initialized neural networks and averaging their predictions at inference time. Deep-UQ now exposes five ensemble variants so the same core idea can be used for plain regression, regression with input-dependent noise, classification, multi-output regression, and multi-output regression with predicted noise.

Why Use Deep Ensembles¶

Deep ensembles remain one of the strongest practical UQ baselines for neural networks because they do not require a variational posterior, Hessian approximation, or sampler over weights. The uncertainty signal comes from the spread across independently trained models.

Two ideas underpin the method family:

1. Independent models started from different random initializations settle into different local solutions.
2. Aggregating those solutions approximates model averaging and exposes epistemic uncertainty through prediction disagreement.

The classic ensemble argument goes back to Hansen and Salamon (1990), while the modern deep-learning formulation used here follows Lakshminarayanan, Pritzel, and Blundell (2017).

Shared Setup¶

Let \(\mathcal{D}=\{(x_i, y_i)\}_{i=1}^N\) and let \(M\) denote the number of ensemble members. Each member has parameters $ heta^{(m)}$ and predictive mapping \(f_{ heta^{(m)}}(x)\).

Each model is trained independently:

At prediction time, the ensemble combines member outputs instead of trusting any single fitted network.

1. DeepEnsembleRegressor¶

DeepEnsembleRegressor is the plain regression ensemble. Each member predicts a single deterministic output \(f_{ heta^{(m)}}(x)\) and is trained with mean squared error:

The ensemble predictive mean is

and the epistemic variance used in Deep-UQ is the member-wise sample variance (with unbiased=False in the implementation):

This method is appropriate when the observation noise is negligible, already accounted for in the dataset, or not the quantity of interest. In that case the ensemble variance is interpreted as model uncertainty only.

Deep-UQ interface: DeepEnsembleRegressor.predict_uq(x) returns mean, epistemic_var, and total_var=epistemic_var.

Scientific notebook: DeepEnsemble_AdvectionDiffusionReaction1D_Tutorial.ipynb

Primary references: Hansen and Salamon (1990); Lakshminarayanan et al. (2017).

2. HeteroscedasticDeepEnsembleRegressor¶

HeteroscedasticDeepEnsembleRegressor augments each member so that it predicts a mean and an input-dependent variance:

Deep-UQ implements a Gaussian negative log-likelihood with a diagonal variance head. Writing \(s_{ heta^{(m)}}(x)=\log \sigma^2_{ heta^{(m)}}(x)\), each member minimizes

The ensemble predictive mean remains

while the total predictive variance is decomposed into epistemic and aleatoric terms:

This is the right variant when the data contain input-dependent measurement noise. In Deep-UQ, the predicted variance is clamped below by min_variance for numerical stability.

Deep-UQ interface: predict_uq(x) returns mean, epistemic_var, aleatoric_var, and total_var.

Scientific notebook: HeteroscedasticDeepEnsemble_AdvectionDiffusionReaction1D_Tutorial.ipynb

Primary references: Nix and Weigend (1994) for Gaussian mean/variance prediction; Kendall and Gal (2017) for the epistemic/aleatoric decomposition; Lakshminarayanan et al. (2017) for the ensemble construction.

3. DeepEnsembleClassifier¶

DeepEnsembleClassifier targets classification. Each member predicts logits \(z_{ heta^{(m)}}(x) \in \mathbb{R}^C\) and is trained with cross-entropy:

where

The ensemble predictive probability vector is the average of member probabilities:

Deep-UQ also reports a per-class probability variance,

which acts as a direct disagreement proxy. This is especially useful near scientific safety/failure boundaries where member classifiers disagree about the class region.

Deep-UQ interface: predict_uq(x) returns probs, probs_var, and sets mean=probs for consistency with the package-wide UQResult contract.

Scientific notebook: DeepEnsemble_Elasticity2D_Classification_Tutorial.ipynb

Primary references: Hansen and Salamon (1990); Lakshminarayanan et al. (2017).

4. MultiOutputDeepEnsembleRegressor¶

MultiOutputDeepEnsembleRegressor generalizes the plain regressor to vector outputs. Each member predicts

for example displacement and stress, or any other coupled set of scientific outputs.

Training still uses MSE,

and Deep-UQ aggregates the mean componentwise:

The package currently reports a diagonal epistemic covariance approximation, not a full output covariance matrix:

That choice keeps the interface simple and aligns with the existing UQResult contract. It is appropriate when the main question is uncertainty magnitude per output channel rather than the full cross-output covariance.

Deep-UQ interface: predict_uq(x) returns mean, epistemic_var, and total_var=epistemic_var, each shaped like the vector output.

Scientific notebook: MultiOutputDeepEnsemble_ElasticBar1D_Tutorial.ipynb

Primary references: Lakshminarayanan et al. (2017). The multi-output form implemented in Deep-UQ is a direct vector-valued extension of the same ensemble averaging principle.

5. HeteroscedasticMultiOutputDeepEnsembleRegressor¶

HeteroscedasticMultiOutputDeepEnsembleRegressor combines the previous two extensions: each member predicts a vector mean together with a diagonal vector variance:

The member loss is the diagonal multivariate Gaussian negative log-likelihood, implemented channelwise in Deep-UQ:

Deep-UQ again returns a diagonal covariance decomposition:

This variant is the most complete regression-side ensemble in the current package: it captures member disagreement and input-dependent per-output noise at the same time.

Deep-UQ interface: predict_uq(x) returns mean, epistemic_var, aleatoric_var, and total_var, each matching the output shape.

Scientific notebook: HeteroscedasticMultiOutputDeepEnsemble_Transport2D_Tutorial.ipynb

Primary references: Nix and Weigend (1994); Kendall and Gal (2017); Lakshminarayanan et al. (2017). As with the plain multi-output regressor, this is a package-level extension of the same ensemble principle with diagonal Gaussian heads.

Deep-UQ Interfaces¶

Available classes:

• DeepEnsembleRegressor
• HeteroscedasticDeepEnsembleRegressor
• DeepEnsembleClassifier
• MultiOutputDeepEnsembleRegressor
• HeteroscedasticMultiOutputDeepEnsembleRegressor
• DeepEnsembleWrapper (backward-compatible alias of DeepEnsembleRegressor)

Recommended Scientific Examples¶

References¶

1. Hansen, L. K., & Salamon, P. (1990). Neural Network Ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993-1001. DOI: 10.1109/34.58871
2. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Advances in Neural Information Processing Systems 30. NeurIPS proceedings
3. Nix, D. A., & Weigend, A. S. (1994). Estimating the Mean and Variance of the Target Probability Distribution. Proceedings of the 1994 IEEE International Conference on Neural Networks. DOI: 10.1109/ICNN.1994.374138
4. Kendall, A., & Gal, Y. (2017). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? Advances in Neural Information Processing Systems 30. NeurIPS proceedings