A single ML model is just one opinion. It learned from one arrangement of training data, made specific decisions about what patterns matter, and internalized the noise in that particular dataset alongside the signal.
What if, instead of trusting that one opinion, you trained dozens of models on slightly different versions of the same data and took a vote?
That's the core idea behind Bootstrap Aggregation, usually called Bagging. It doesn't try to build a perfect model. It builds many imperfect models with deliberate variation between them, then combines their predictions into something more stable than any individual model could produce.
What "Bootstrap" Actually Means
Before the machine learning part, there's a statistical concept worth understanding.
A bootstrap sample is a new dataset created by sampling with replacement from the original dataset. The resulting sample has the same number of rows as the original, but it's not identical. Because each observation is drawn independently with replacement, some rows from the original dataset will appear multiple times, and others won't appear at all.
Here's a small example. Say your original training set has five examples:
Original: [A, B, C, D, E]
A bootstrap sample of size 5 might look like:
Bootstrap 1: [A, A, C, D, D] (B and E omitted; A and D appear twice)
Bootstrap 2: [B, C, C, E, E] (A and D omitted; C and E appear twice)
Bootstrap 3: [A, B, B, D, E] (C omitted; B appears twice)
Each bootstrap sample represents a slightly different picture of the same underlying data. The statistical theory behind this goes back to work on estimating properties of a distribution from a sample, but for our purposes the key point is this: by sampling with replacement, you get genuine variation between samples without needing more data.
For a dataset with n observations, any given bootstrap sample will omit roughly 1/e of the original observations on average, which works out to about 36.8%. Those omitted observations become useful later.
The Bagging Pipeline
The full process looks like this:
Original Training Dataset
|
+----------+----------+----------+
| | | |
Bootstrap 1 Bootstrap 2 Bootstrap 3 ...
| | | |
Model 1 Model 2 Model 3 ...
| | | |
+----------+----------+----------+
|
Aggregation
|
Final Prediction
Bootstrap sampling: Create k bootstrap samples from the original training set. Each sample has the same number of rows as the original but is different due to sampling with replacement.
Train independently: Train one model on each bootstrap sample. These models are trained completely independently of each other. This is important: they don't communicate, don't share gradients, and don't adjust based on each other's performance.
Generate predictions: For a new input, run it through all k models and collect their predictions.
Aggregate: Combine those predictions into one.
For regression, aggregation typically means averaging the numerical outputs:
Final prediction = (pred_1 + pred_2 + ... + pred_k) / k
For classification, it typically means majority voting:
Final prediction = most common class among (pred_1, pred_2, ..., pred_k)
Why Does Bagging Actually Work?
This is the important part. Bagging isn't magic. It works because of a specific property of aggregating independent, diverse predictions.
Start with the concept of variance in the context of a model. A high-variance model is sensitive to the specific training data it saw. Train it on one dataset, and it learns one set of patterns. Train it on a slightly different dataset, and it might make quite different predictions. Decision trees with no depth limit are a classic example: they'll fit their training data nearly perfectly but their predictions can shift dramatically if the training data changes slightly.
Now consider what happens when you average multiple high-variance predictions.
If the errors made by individual models are uncorrelated (or at least not perfectly correlated), their random mistakes tend to cancel out when averaged. One model overestimates on a particular region of the input space; another underestimates. Their average is closer to the truth than either individual prediction.
Formally, if you have k models each with variance sigma squared and zero covariance between their errors, the variance of their average is:
Var(average) = sigma^2 / k
As k increases, the variance of the ensemble prediction shrinks. This is the same reason that averaging many noisy measurements of a quantity gives you a better estimate than relying on a single noisy measurement.
The important caveat is the "uncorrelated" part. If all your models make the same mistakes because they're too similar to each other, averaging them doesn't help much. The formula for the average variance when models have pairwise correlation rho is:
Var(average) = rho * sigma^2 + (1 - rho) * sigma^2 / k
When rho approaches 1 (models are identical), the variance reduction from k models disappears. This is why the bootstrap sampling step matters: it deliberately creates variation between the training sets, which creates variation between the models, which reduces correlation between their errors.
Bagging primarily reduces variance. It does not reliably reduce bias. If all your models are systematically wrong in the same direction, averaging them will still give you a systematically wrong answer.
A Concrete Example
Say you're classifying whether an email is spam or not, and you train five small decision trees using bagging. For a particular email, they each make a prediction:
Model 1: Spam
Model 2: Not Spam
Model 3: Spam
Model 4: Spam
Model 5: Not Spam
Majority vote: Spam appears three times, Not Spam appears twice. Final prediction: Spam.
For a regression example, say you're predicting house prices and five models return:
Model 1: $412,000
Model 2: $395,000
Model 3: $428,000
Model 4: $407,000
Model 5: $418,000
Average: ($412,000 + $395,000 + $428,000 + $407,000 + $418,000) / 5 = $412,000
Any one of these models might be somewhat off. Their average tends to be closer to the true value than most of the individual predictions, assuming the individual errors aren't all biased in the same direction.
Why Not Just Train the Same Model Repeatedly?
The obvious question: why go through the trouble of bootstrap sampling? Why not just train the same model multiple times on the same data?
If your model is deterministic and you train it on the same dataset, you'll get the same model every time. Averaging five copies of the same model gives you exactly that model, with no variance reduction at all.
Bootstrap sampling is what creates the necessary diversity. Each sample contains a different subset of the training examples, with different duplications. A model trained on Bootstrap 1 never sees some of the original examples and sees others multiple times. A model trained on Bootstrap 2 has a different combination of repeated and missing examples. The result is k genuinely different models that learned slightly different things.
A secondary benefit: because each model is trained independently, the training process can be parallelized. Train k models simultaneously across k machines or k CPU cores. This is not possible with methods where models must be trained sequentially.
Out-of-Bag Samples
Remember that each bootstrap sample omits roughly 36.8% of the original training examples. Those omitted examples are called out-of-bag (OOB) samples for that particular model.
Since the model was never trained on its out-of-bag examples, those examples can be used to evaluate the model's performance on data it hasn't seen. For each original training example, it was out-of-bag for some fraction of the k models. You can collect predictions from those models (the ones that didn't train on this example) and use them to estimate the model's generalization error.
Aggregating these OOB predictions across the entire training set gives you an OOB error estimate: an estimate of how well the ensemble performs on unseen data, without needing to hold out a separate validation set.
This is a useful property. In situations where you want to use as much data as possible for training, OOB evaluation lets you get a generalization estimate without sacrificing any training examples to a held-out set.
OOB error is not always equivalent to k-fold cross-validation or other evaluation strategies, and the two can diverge depending on the dataset and model. But it's a practically useful tool that comes essentially for free when you're already doing bagging.
Bagging and Random Forests
Random Forest is one of the most successful applications of the bagging idea, but it's not identical to bagging.
Standard bagging: take bootstrap samples, train full models independently on each, aggregate predictions.
Random Forest: also uses bootstrap samples and independent training, but adds one more source of randomness. At each split point in each decision tree, instead of considering all available features, it considers only a random subset of features. This additional randomization makes the trees even more different from each other.
Why does this help? In standard bagging with decision trees, if there's one very strong predictor in the dataset, most of the trees will use that predictor near the top of each tree. The resulting trees will be more correlated with each other than bootstrap sampling alone would suggest, because they're all making similar early splits. Restricting feature selection at each split breaks this correlation and lets other predictors contribute meaningfully across different trees.
Random Forest therefore benefits from two distinct sources of diversity: bootstrap sampling (different training examples) and random feature selection (different features considered at each split). The result is typically lower correlation between trees and better ensemble performance than standard bagging with decision trees.
Bagging:
Bootstrap samples + independent training + aggregation
Random Forest:
Bootstrap samples + independent training + random feature subsets + aggregation
Bagging vs. Boosting
Boosting is another ensemble method, but it operates on entirely different principles.
| Bagging | Boosting | |
|---|---|---|
| Training order | Independent, parallel | Sequential, each model trained on results of previous |
| Data weighting | Each model sees a bootstrap sample | Examples weighted by difficulty; hard examples emphasized |
| Primary effect | Reduces variance | Reduces bias (and variance in some formulations) |
| Model relationship | Each model stands alone | Each model corrects residuals of the previous ensemble |
| Examples | Random Forest | AdaBoost, Gradient Boosting, XGBoost |
In boosting, model 2 is trained on what model 1 got wrong. Model 3 is trained on what the combination of model 1 and model 2 got wrong. Each new model is guided by the failures of the previous ones, which makes the ensemble increasingly accurate on hard examples.
Bagging makes models independent on purpose. Boosting makes models dependent on purpose.
Both can produce strong ensembles, but they're solving different problems. Bagging is most useful when your base model is high-variance and unstable. Boosting can sometimes help when your base model is too simple (high-bias), though this depends significantly on the specific boosting algorithm and configuration.
When Bagging Helps (and When It Doesn't)
Bagging is most effective when the base model is a high-variance, low-bias learner. Deep decision trees are the canonical example: they overfit individual training sets aggressively, and bagging provides substantial stability gains.
If your base model is already stable (low-variance), bagging provides little benefit. A linear regression model trained on reasonable data will produce nearly the same predictions regardless of small variations in the training set. Averaging many nearly identical linear models doesn't give you much.
Bagging also doesn't fix bias. If your model consistently underestimates a quantity because it lacks the capacity to fit the true relationship, averaging more such models will still underestimate.
The conditions where bagging provides meaningful improvement:
- High-variance base models
- Datasets where different bootstrap samples genuinely change model behavior
- Sufficient diversity between the resulting models The conditions where bagging helps less:
- Already-stable base models
- Highly correlated base models despite bootstrap sampling
- Problems where bias is the primary issue
The Deeper Intuition
Training one model is like asking one person for directions. They might know the area well, or they might give you confidently wrong advice based on a bad experience they had once.
Asking ten people who've each explored slightly different parts of the area gives you more coverage. Their individual answers might vary, but their collective answer tends to be more reliable than any single person's, especially if they each have genuine knowledge rather than all reading from the same map.
Bagging does the same thing with models. It doesn't make any individual model better. It creates genuine variation through bootstrap sampling, exploits the statistical property that averaging independent noisy estimates reduces variance, and produces an ensemble whose stability comes from the diversity of its components.
The robustness isn't in any single tree or any single decision. It emerges from the agreement across models that each learned from a slightly different slice of the data.
Top comments (0)