Many machine learning models perform well on training data but behave unpredictably when the input data changes slightly. This “instability” is often caused by high variance—the model learns patterns that are too dependent on the specific sample it was trained on. Bootstrap Aggregation, commonly called bagging, is a practical technique to reduce variance and improve generalisation. It works by training multiple models on different bootstrapped (resampled) versions of the dataset and then averaging their predictions. For learners building strong ML foundations through data science classes in Pune, bagging is an essential idea because it connects statistics, model behaviour, and real-world reliability in a clear way.
What Bagging Is and Why It Reduces Variance
Bagging is based on the bootstrap: a resampling method where you create multiple datasets by sampling with replacement from the original dataset. Each bootstrapped dataset has the same size as the original, but some rows appear multiple times and some may be missing.
The bagging workflow looks like this:
- Create many bootstrapped datasets from the original training data.
- Train the same base model on each dataset independently.
- Combine predictions:
- Regression: average predictions.
- Classification: majority vote (or average probabilities).
Why does this help? If a single model is sensitive to small changes in the data (high variance), its predictions can swing. But if you train many such models on slightly different samples, their errors are less likely to be identical. Averaging tends to cancel out random fluctuations, producing a more stable final prediction. This is a key statistical insight taught early in data science classes in Pune, especially when discussing bias–variance trade-offs.
Bagging vs Single Models: Where Instability Shows Up
High-variance models typically include:
- Deep decision trees
- K-nearest neighbours (depending on k and noise)
- Complex neural networks (in smaller datasets)
- Models with many interaction terms
Decision trees are the classic example. A small change in the training set can lead to a completely different tree structure. Bagging addresses this by building many trees and combining them.
However, bagging is not primarily designed to reduce bias (systematic error from underfitting). If your base model is too simple to capture the relationship in the data, averaging many simple models will still be simple. Bagging shines when the base model is reasonably expressive but unstable.
How Bagging Works in Practice
Step 1: Bootstrapped samples
Suppose you have 10,000 training rows. Bagging might generate 100 bootstrapped samples, each created by randomly selecting 10,000 rows with replacement. Each sample contains duplicates and omits some rows.
Step 2: Train multiple base learners
You train 100 models—often called “base learners”—one per bootstrap sample. These models can be trained concurrently, which makes bagging friendly for modern compute setups.
Step 3: Aggregate predictions
For regression, the final prediction is the average of the 100 outputs. For classification, the most common approach is majority vote or averaging probabilities and selecting the most likely class.
This simple pipeline improves robustness without requiring complex optimisation tricks. Many practitioners first encounter it during applied ML work after studying ensemble methods in data science classes in Pune.
Random Forest: The Most Popular Bagging Extension
Random Forest is bagging applied to decision trees with an additional idea: feature randomness. When splitting a node, each tree considers only a random subset of features instead of all features. This makes trees less correlated with one another, which further improves the variance reduction effect.
In plain terms:
- Bagging reduces variance by averaging many models trained on different data samples.
- Random Forest goes further by also reducing similarity between trees through random feature selection.
This is why Random Forest often performs strongly out of the box, especially on tabular datasets.
When Bagging Helps and When It Doesn’t
Bagging is most helpful when:
- Your model is unstable and shows high variance.
- Your dataset has noise, and you want smoother decision boundaries.
- You want improved accuracy without heavy feature engineering.
- You need better reliability under data shifts that are small but frequent.
Bagging may not be ideal when:
- The base model is already stable (for example, linear regression with regularisation).
- You need interpretability from a single transparent model.
- Training many models is too expensive for your latency or compute budget.
Also, bagging does not automatically fix data leakage, poor target definitions, or biased samples. If the training data is flawed, bagging will average flawed models.
Practical Tips for Using Bagging Well
- Choose a high-variance base model: Bagging delivers the biggest gains with models like decision trees.
- Use enough estimators: More models usually improve stability up to a point, after which gains flatten.
- Control complexity: For trees, set depth or minimum samples per leaf to avoid extreme overfitting.
- Evaluate properly: Use cross-validation or a clean holdout set to confirm generalisation improvements.
- Watch latency: Aggregating many models can increase prediction time; balance performance and speed.
These are the kinds of trade-offs that become clearer with project-based practice, which is why ensemble tuning is often a module in data science classes in Pune.
Conclusion
Bootstrap Aggregation (bagging) is a straightforward yet powerful method for reducing model variance. By training multiple models on bootstrapped subsets of the data and averaging their predictions, bagging produces more stable and reliable results than a single high-variance model. It is the foundation of widely used methods like Random Forest and remains a practical choice for improving generalisation on real-world datasets. If you are building strong ensemble intuition through data science classes in Pune, mastering bagging will help you design models that perform consistently—not just on one dataset, but across the messy variations that real data always brings.