|
|
{% extends "layout.html" %}
|
|
|
|
|
|
{% block content %}
|
|
|
<!DOCTYPE html>
|
|
|
<html lang="en">
|
|
|
<head>
|
|
|
<meta charset="UTF-8">
|
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
|
<title>Study Guide: Bagging (Bootstrap Aggregating)</title>
|
|
|
|
|
|
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
|
|
|
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
|
|
<style>
|
|
|
|
|
|
body {
|
|
|
background-color: #ffffff;
|
|
|
color: #000000;
|
|
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
|
|
font-weight: normal;
|
|
|
line-height: 1.8;
|
|
|
margin: 0;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
.container {
|
|
|
max-width: 800px;
|
|
|
margin: 0 auto;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
h1, h2, h3 {
|
|
|
color: #000000;
|
|
|
border: none;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
h1 {
|
|
|
text-align: center;
|
|
|
border-bottom: 3px solid #000;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 30px;
|
|
|
font-size: 2.5em;
|
|
|
}
|
|
|
|
|
|
h2 {
|
|
|
font-size: 1.8em;
|
|
|
margin-top: 40px;
|
|
|
border-bottom: 1px solid #ddd;
|
|
|
padding-bottom: 8px;
|
|
|
}
|
|
|
|
|
|
h3 {
|
|
|
font-size: 1.3em;
|
|
|
margin-top: 25px;
|
|
|
}
|
|
|
|
|
|
|
|
|
strong {
|
|
|
font-weight: 900;
|
|
|
}
|
|
|
|
|
|
|
|
|
p, li {
|
|
|
font-size: 1.1em;
|
|
|
border-bottom: 1px solid #e0e0e0;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 10px;
|
|
|
}
|
|
|
|
|
|
|
|
|
li:last-child {
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
ol {
|
|
|
list-style-type: decimal;
|
|
|
padding-left: 20px;
|
|
|
}
|
|
|
|
|
|
ol li {
|
|
|
padding-left: 10px;
|
|
|
}
|
|
|
|
|
|
|
|
|
ul {
|
|
|
list-style-type: none;
|
|
|
padding-left: 0;
|
|
|
}
|
|
|
|
|
|
ul li::before {
|
|
|
content: "•";
|
|
|
color: #000;
|
|
|
font-weight: bold;
|
|
|
display: inline-block;
|
|
|
width: 1em;
|
|
|
margin-left: 0;
|
|
|
}
|
|
|
|
|
|
|
|
|
pre {
|
|
|
background-color: #f4f4f4;
|
|
|
border: 1px solid #ddd;
|
|
|
border-radius: 5px;
|
|
|
padding: 15px;
|
|
|
white-space: pre-wrap;
|
|
|
word-wrap: break-word;
|
|
|
font-family: "Courier New", Courier, monospace;
|
|
|
font-size: 0.95em;
|
|
|
font-weight: normal;
|
|
|
color: #333;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
.story-bagging {
|
|
|
background-color: #f0f9ff;
|
|
|
border-left: 4px solid #0d6efd;
|
|
|
margin: 15px 0;
|
|
|
padding: 10px 15px;
|
|
|
font-style: italic;
|
|
|
color: #555;
|
|
|
font-weight: normal;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
.story-bagging p, .story-bagging li {
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
.example-bagging {
|
|
|
background-color: #f3f8fe;
|
|
|
padding: 15px;
|
|
|
margin: 15px 0;
|
|
|
border-radius: 5px;
|
|
|
border-left: 4px solid #4dabf7;
|
|
|
}
|
|
|
|
|
|
.example-bagging p, .example-bagging li {
|
|
|
border-bottom: none !important;
|
|
|
}
|
|
|
|
|
|
|
|
|
.quiz-section {
|
|
|
background-color: #fafafa;
|
|
|
border: 1px solid #ddd;
|
|
|
border-radius: 5px;
|
|
|
padding: 20px;
|
|
|
margin-top: 30px;
|
|
|
}
|
|
|
.quiz-answers {
|
|
|
background-color: #f3f8fe;
|
|
|
padding: 15px;
|
|
|
margin-top: 15px;
|
|
|
border-radius: 5px;
|
|
|
}
|
|
|
|
|
|
|
|
|
table {
|
|
|
width: 100%;
|
|
|
border-collapse: collapse;
|
|
|
margin: 25px 0;
|
|
|
}
|
|
|
th, td {
|
|
|
border: 1px solid #ddd;
|
|
|
padding: 12px;
|
|
|
text-align: left;
|
|
|
}
|
|
|
th {
|
|
|
background-color: #f2f2f2;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
|
|
|
@media (max-width: 768px) {
|
|
|
body, .container {
|
|
|
padding: 10px;
|
|
|
}
|
|
|
h1 { font-size: 2em; }
|
|
|
h2 { font-size: 1.5em; }
|
|
|
h3 { font-size: 1.2em; }
|
|
|
p, li { font-size: 1em; }
|
|
|
pre { font-size: 0.85em; }
|
|
|
table, th, td { font-size: 0.9em; }
|
|
|
}
|
|
|
</style>
|
|
|
</head>
|
|
|
<body>
|
|
|
|
|
|
<div class="container">
|
|
|
<h1>🌳 Study Guide: Bagging (Bootstrap Aggregating)</h1>
|
|
|
|
|
|
<h2>🔹 1. Introduction</h2>
|
|
|
<div class="story-bagging">
|
|
|
<p><strong>Story-style intuition: The Wisdom of Crowds</strong></p>
|
|
|
<p>Imagine you want to guess the number of jellybeans in a giant jar. If you ask one person, their guess might be way off. They might be an expert, or they might be terrible at guessing. Their prediction has high <strong>variance</strong>. But what if you ask 100 different people and take the average of all their guesses? The final averaged guess is almost always much closer to the true number than any single individual's guess. This is the "wisdom of crowds" effect. <strong>Bagging</strong> applies this same logic to machine learning. Instead of trusting one complex model (one expert guesser), we train many models on slightly different perspectives of the data and combine their predictions to get a more stable and accurate result.</p>
|
|
|
</div>
|
|
|
<p><strong>Bagging</strong>, short for <strong>Bootstrap Aggregating</strong>, is a powerful ensemble machine learning technique. Its primary goal is to reduce the variance of a model, thereby preventing overfitting and improving its stability. It works by training multiple instances of the same base model on different random subsets of the training data and then aggregating their predictions.</p>
|
|
|
|
|
|
<h2>🔹 2. How Bagging Works</h2>
|
|
|
<p>The process of Bagging is a straightforward three-step method.</p>
|
|
|
|
|
|
<ol>
|
|
|
<li><strong>Bootstrap Sampling:</strong> This is the "B" in Bagging. We create multiple new training datasets from our original dataset. Each new dataset is created by <strong>sampling with replacement</strong>.
|
|
|
<div class="example-bagging"><p><strong>Example:</strong> If our original dataset is `[A, B, C, D]`, a bootstrap sample might be `[B, A, D, B]`. Notice that 'B' was picked twice and 'C' was not picked at all. Each bootstrap sample is the same size as the original dataset.</p></div>
|
|
|
</li>
|
|
|
<li><strong>Train Models in Parallel:</strong> We train a separate instance of the same base model (e.g., a Decision Tree) on each of the bootstrap samples. Since these models are independent of each other, they can all be trained at the same time (in parallel).</li>
|
|
|
<li><strong>Aggregate Predictions:</strong> Once all models are trained, we use them to make predictions on new, unseen data. The final prediction is an aggregation of all the individual model predictions.
|
|
|
<ul>
|
|
|
<li><strong>For Regression (predicting a number):</strong> We take the <strong>average</strong> of all predictions.</li>
|
|
|
<li><strong>For Classification (predicting a category):</strong> We take a <strong>majority vote</strong>.</li>
|
|
|
</ul>
|
|
|
</li>
|
|
|
</ol>
|
|
|
|
|
|
<h2>🔹 3. Mathematical Concept</h2>
|
|
|
<p>The aggregation step is what combines the "wisdom" of the individual models. For a new data point \(x\), and \(m\) trained models:</p>
|
|
|
<ul>
|
|
|
<li><strong>Regression:</strong> The final prediction is the mean of the individual predictions.
|
|
|
<p>$$ \hat{y} = \frac{1}{m} \sum_{i=1}^{m} f_i(x) $$</p>
|
|
|
</li>
|
|
|
<li><strong>Classification:</strong> The final prediction is the class that receives the most votes.
|
|
|
<p>$$ \hat{y} = \text{majority\_vote}\{f_1(x), ..., f_m(x)\} $$</p>
|
|
|
</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 4. Key Points</h2>
|
|
|
<ul>
|
|
|
<li><strong>Reduces Variance:</strong> This is the primary benefit. By averaging the outputs, the random errors and quirks of individual models tend to cancel each other out, leading to a much more stable final prediction.</li>
|
|
|
<li><strong>Best with Unstable Models:</strong> Bagging is most effective when used with high-variance, low-bias models. Decision Trees are the perfect example: a single deep decision tree is very prone to overfitting (high variance), but a bagged ensemble of them is very robust.</li>
|
|
|
<li><strong>Parallelizable:</strong> Each model in the ensemble is trained independently, making Bagging very efficient on multi-core processors.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 5. Advantages & Disadvantages</h2>
|
|
|
<table>
|
|
|
<thead>
|
|
|
<tr>
|
|
|
<th>Advantages</th>
|
|
|
<th>Disadvantages</th>
|
|
|
</tr>
|
|
|
</thead>
|
|
|
<tbody>
|
|
|
<tr>
|
|
|
<td>✅ Significantly <strong>reduces overfitting</strong> and variance.</td>
|
|
|
<td>❌ <strong>Increased Computational Cost:</strong> You have to train multiple models instead of just one, which takes more time and resources.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td>✅ Often leads to a major <strong>improvement in accuracy</strong> and stability.</td>
|
|
|
<td>❌ <strong>Loss of Interpretability:</strong> It's easy to understand and visualize a single decision tree, but it's very difficult to interpret the combined logic of 100 different trees.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td>✅ Can be applied to almost any type of base model (e.g., trees, SVMs, neural networks).</td>
|
|
|
<td>❌ Less effective for models that are already stable and have low variance (like Linear Regression).</td>
|
|
|
</tr>
|
|
|
</tbody>
|
|
|
</table>
|
|
|
|
|
|
<h2>🔹 6. Python Implementation (Beginner Example)</h2>
|
|
|
<div class="story-bagging">
|
|
|
<p>In this example, we'll compare a single, complex Decision Tree to a Bagging ensemble of many Decision Trees. We expect the single tree to overfit and perform perfectly on the training data but poorly on the test data. The Bagging classifier should be more robust and perform well on both.</p>
|
|
|
</div>
|
|
|
<pre><code>
|
|
|
from sklearn.datasets import make_classification
|
|
|
from sklearn.model_selection import train_test_split
|
|
|
from sklearn.tree import DecisionTreeClassifier
|
|
|
from sklearn.ensemble import BaggingClassifier
|
|
|
from sklearn.metrics import accuracy_score
|
|
|
|
|
|
# --- 1. Create a Sample Dataset ---
|
|
|
X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
|
|
|
n_redundant=0, random_state=42)
|
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
|
|
|
|
|
|
# --- 2. Train a Single Decision Tree (High Variance Model) ---
|
|
|
single_tree = DecisionTreeClassifier(random_state=42)
|
|
|
single_tree.fit(X_train, y_train)
|
|
|
y_pred_tree = single_tree.predict(X_test)
|
|
|
print(f"Single Decision Tree Accuracy: {accuracy_score(y_test, y_pred_tree):.2%}")
|
|
|
|
|
|
# --- 3. Train a Bagging Ensemble of Decision Trees ---
|
|
|
# We create an ensemble of 100 decision trees.
|
|
|
bagging_clf = BaggingClassifier(
|
|
|
base_estimator=DecisionTreeClassifier(random_state=42),
|
|
|
n_estimators=100,
|
|
|
random_state=42
|
|
|
)
|
|
|
bagging_clf.fit(X_train, y_train)
|
|
|
y_pred_bagging = bagging_clf.predict(X_test)
|
|
|
print(f"Bagging Classifier Accuracy: {accuracy_score(y_test, y_pred_bagging):.2%}")
|
|
|
</code></pre>
|
|
|
|
|
|
<h2>🔹 7. Applications</h2>
|
|
|
<ul>
|
|
|
<li><strong>Random Forest:</strong> The most famous application of Bagging. A Random Forest is an ensemble of decision trees that uses Bagging for data sampling and adds an extra layer of randomness by also selecting a random subset of features for each tree.</li>
|
|
|
<li><strong>Medical Diagnosis:</strong> Combining the opinions of multiple diagnostic models to make a more reliable prediction about a patient's condition.</li>
|
|
|
<li><strong>Fraud Detection:</strong> Training multiple models on different subsets of transaction data to create a more robust fraud detection system.</li>
|
|
|
</ul>
|
|
|
|
|
|
<div class="quiz-section">
|
|
|
<h2>📝 Quick Quiz: Test Your Knowledge</h2>
|
|
|
<ol>
|
|
|
<li><strong>What does "Bootstrap Aggregating" mean?</strong></li>
|
|
|
<li><strong>What is the main goal of Bagging? Does it primarily reduce bias or variance?</strong></li>
|
|
|
<li><strong>If you were using Bagging for a regression problem to predict house prices, how would you calculate the final prediction from your ensemble of models?</strong></li>
|
|
|
<li><strong>Why is Bagging not very effective when used with a simple model like Linear Regression?</strong></li>
|
|
|
</ol>
|
|
|
<div class="quiz-answers">
|
|
|
<h3>Answers</h3>
|
|
|
<p><strong>1.</strong> <strong>Bootstrap</strong> refers to creating random subsamples of the data with replacement. <strong>Aggregating</strong> refers to combining the predictions of the models trained on these subsamples (e.g., by averaging or voting).</p>
|
|
|
<p><strong>2.</strong> The main goal of Bagging is to <strong>reduce variance</strong>. It helps to stabilize unstable models that are prone to overfitting.</p>
|
|
|
<p><strong>3.</strong> You would take the <strong>average</strong> of the price predictions from all the individual models in the ensemble.</p>
|
|
|
<p><strong>4.</strong> Linear Regression is a low-variance (stable) model. Its predictions don't change drastically even when the training data is slightly modified. Since Bagging's main strength is reducing variance, it provides little benefit to an already stable model.</p>
|
|
|
</div>
|
|
|
</div>
|
|
|
|
|
|
<h2>🔹 Key Terminology Explained</h2>
|
|
|
<div class="story-bagging">
|
|
|
<p><strong>The Story: Decoding the Jellybean Guesser's Strategy</strong></p>
|
|
|
</div>
|
|
|
<ul>
|
|
|
<li>
|
|
|
<strong>Ensemble Method:</strong>
|
|
|
<br>
|
|
|
<strong>What it is:</strong> A machine learning technique where multiple models (often called "weak learners") are trained and their predictions are combined to achieve better performance than any single model alone.
|
|
|
<br>
|
|
|
<strong>Story Example:</strong> Instead of relying on one expert jellybean guesser, you assemble a "committee" or <strong>ensemble</strong> of 100 guessers.
|
|
|
</li>
|
|
|
<li>
|
|
|
<strong>Bootstrap Sampling:</strong>
|
|
|
<br>
|
|
|
<strong>What it is:</strong> A resampling method that involves drawing random samples from a dataset *with replacement*.
|
|
|
<br>
|
|
|
<strong>Story Example:</strong> To give each of your 100 guessers a slightly different perspective, you show each one a different random handful of jellybeans from the jar (and you put the beans back each time). This is <strong>bootstrap sampling</strong>.
|
|
|
</li>
|
|
|
<li>
|
|
|
<strong>Variance (in Models):</strong>
|
|
|
<br>
|
|
|
<strong>What it is:</strong> A measure of how much a model's predictions would change if it were trained on a different subset of the data. High variance means the model is unstable and sensitive to the specific training data it sees (i.e., it overfits).
|
|
|
<br>
|
|
|
<strong>Story Example:</strong> A single, overconfident "expert" guesser has high <strong>variance</strong>; their guess might be very different if they saw a slightly different handful of jellybeans. The averaged guess of the crowd has low variance.
|
|
|
</li>
|
|
|
</ul>
|
|
|
|
|
|
</div>
|
|
|
|
|
|
</body>
|
|
|
</html>
|
|
|
{% endblock %}
|
|
|
|
