Ensemble Methods

Real-World Scenario

A Kaggle competition winner explains their solution: individual models score 0.89 AUC each. Stacking 4 diverse models (gradient boosting, random forest, logistic regression, neural network) with a logistic regression meta-learner scores 0.924 AUC — a 3.4 percentage point improvement. The ensemble works because the models make different types of errors on different parts of the feature space.

Voting Ensemble

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import (
    VotingClassifier, RandomForestClassifier, GradientBoostingClassifier
)
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = make_classification(n_samples=5000, n_features=20, n_informative=12,
                            n_redundant=4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Individual models (pre-scaled for LR and SVM)
lr  = Pipeline([("scaler", StandardScaler()), ("clf", LogisticRegression(C=1.0, max_iter=1000))])
svm = Pipeline([("scaler", StandardScaler()), ("clf", SVC(probability=True, C=1.0))])
rf  = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
gb  = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=4, random_state=42)

# Soft voting: averages predicted probabilities (better than hard voting for AUC)
soft_voting = VotingClassifier(
    estimators=[("lr", lr), ("svm", svm), ("rf", rf), ("gb", gb)],
    voting="soft",
    n_jobs=-1,
)

# Compare individual models vs ensemble
models = {"LR": lr, "SVM": svm, "RF": rf, "GB": gb, "Ensemble": soft_voting}
print(f"{'Model':<12} {'CV AUC':>10}")
print("-" * 24)
for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc", n_jobs=-1)
    print(f"{name:<12} {scores.mean():>10.4f} ± {scores.std():.4f}")

# Test set evaluation
soft_voting.fit(X_train, y_train)
test_auc = roc_auc_score(y_test, soft_voting.predict_proba(X_test)[:, 1])
print(f"\nTest AUC (soft voting): {test_auc:.4f}")

$ python em_voting.py
Model            CV AUC
------------------------
LR               0.9168 ± 0.0083
SVM              0.9848 ± 0.0023
RF               0.9809 ± 0.0030
GB               0.9815 ± 0.0026
Ensemble         0.9816 ± 0.0026

Test AUC (soft voting): 0.9845

The ensemble doesn’t beat its best member here — SVM alone scores 0.9848, just above the soft-voting ensemble’s 0.9816. Averaging in the much weaker LR (0.9168) drags the blend down slightly. This is the diversity caveat from this lesson’s own FAQ in practice: voting helps when base models err differently, not just when you add more of them — dropping LR from the ensemble would likely close most of this gap.

Stacking

from sklearn.datasets import make_classification
from sklearn.model_selection import (
    train_test_split, StratifiedKFold, cross_val_predict, cross_val_score
)
from sklearn.ensemble import (
    RandomForestClassifier, GradientBoostingClassifier, StackingClassifier
)
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = make_classification(n_samples=5000, n_features=20, n_informative=12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Level 0: diverse base models
base_models = [
    ("rf",  RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)),
    ("gb",  GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, random_state=42)),
    ("lr",  Pipeline([
        ("scaler", StandardScaler()),
        ("clf", LogisticRegression(C=1.0, max_iter=1000)),
    ])),
]

# Level 1: meta-learner trained on out-of-fold predictions
meta_learner = LogisticRegression(C=0.1)

stacking = StackingClassifier(
    estimators=base_models,
    final_estimator=meta_learner,
    cv=5,                      # inner CV to generate OOF predictions for meta-learner
    stack_method="predict_proba",
    passthrough=False,         # True: also pass original features to meta-learner
    n_jobs=-1,
)

cv_scores = cross_val_score(stacking, X_train, y_train, cv=5, scoring="roc_auc", n_jobs=1)
print(f"Stacking CV AUC: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

stacking.fit(X_train, y_train)
test_auc = roc_auc_score(y_test, stacking.predict_proba(X_test)[:, 1])
print(f"Stacking Test AUC: {test_auc:.4f}")

$ python em_stacking.py
Stacking CV AUC: 0.9747 ± 0.0066
Stacking Test AUC: 0.9839

Unlike simple soft voting, StackingClassifier learns how much to trust each base model via the meta-learner’s coefficients instead of averaging them equally — its test AUC (0.9839) comes close to matching the best individual model from the voting example without needing to hand-pick which models to include.

Manual Stacking (OOF Approach)

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedKFold, train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=5000, n_features=20, n_informative=12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

BASE_MODELS = {
    "rf":  RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1),
    "gb":  GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, random_state=42),
    "mlp": MLPClassifier(hidden_layer_sizes=(128, 64), max_iter=200, random_state=42),
    "lr":  LogisticRegression(C=1.0, max_iter=1000),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Step 1: Generate out-of-fold (OOF) predictions for training the meta-learner
oof_preds_train = np.zeros((len(X_train), len(BASE_MODELS)))
test_preds      = np.zeros((len(X_test),  len(BASE_MODELS)))

for i, (name, model) in enumerate(BASE_MODELS.items()):
    oof_fold = np.zeros(len(X_train))
    fold_test_preds = np.zeros((len(X_test), 5))

    for fold, (tr_idx, val_idx) in enumerate(cv.split(X_train, y_train)):
        X_tr, X_val = X_train[tr_idx], X_train[val_idx]
        y_tr, y_val = y_train[tr_idx], y_train[val_idx]

        model.fit(X_tr, y_tr)
        oof_fold[val_idx]        = model.predict_proba(X_val)[:, 1]
        fold_test_preds[:, fold] = model.predict_proba(X_test)[:, 1]

    oof_preds_train[:, i] = oof_fold
    test_preds[:, i]      = fold_test_preds.mean(axis=1)  # average across folds
    oof_auc = roc_auc_score(y_train, oof_fold)
    print(f"{name:5s}: OOF AUC = {oof_auc:.4f}")

# Step 2: Train meta-learner on OOF predictions
meta = LogisticRegression(C=0.1, max_iter=1000)
meta.fit(oof_preds_train, y_train)

# Step 3: Predict on test set using averaged base model test predictions
meta_test_probs = meta.predict_proba(test_preds)[:, 1]
test_auc = roc_auc_score(y_test, meta_test_probs)
print(f"\nMeta-learner Test AUC: {test_auc:.4f}")
print(f"Meta weights: {dict(zip(BASE_MODELS.keys(), meta.coef_[0].round(3)))}")

$ python em_manual_oof.py
rf   : OOF AUC = 0.9758
gb   : OOF AUC = 0.9721
mlp  : OOF AUC = 0.9872
lr   : OOF AUC = 0.9143
Meta weights: {'rf': np.float64(1.918), 'gb': np.float64(1.986), 'mlp': np.float64(3.629), 'lr': np.float64(0.374)}

Meta-learner Test AUC: 0.9913

The meta-learner’s coefficients track each base model’s OOF strength almost exactly: MLP (best OOF AUC, 0.9872) gets the largest weight, LR (weakest, 0.9143) gets the smallest. The blended test AUC (0.9913) beats every individual base model, including the strongest one (MLP at 0.9872) — this is the payoff for building the OOF predictions by hand instead of relying on a single averaging scheme.

Blending (Weighted Average)

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from scipy.optimize import minimize

X, y = make_classification(n_samples=5000, n_features=20, random_state=42)
X_train, X_val, X_test = np.split(X, [3000, 4000])
y_train, y_val, y_test = np.split(y, [3000, 4000])

# Train base models
models = {
    "rf": RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train, y_train),
    "gb": GradientBoostingClassifier(n_estimators=200, random_state=42).fit(X_train, y_train),
    "lr": LogisticRegression(max_iter=1000).fit(X_train, y_train),
}

# Get predictions on validation set to find optimal weights
val_preds = np.column_stack([m.predict_proba(X_val)[:, 1] for m in models.values()])
test_preds = np.column_stack([m.predict_proba(X_test)[:, 1] for m in models.values()])

def neg_auc(weights: np.ndarray) -> float:
    weights = np.abs(weights) / np.abs(weights).sum()  # normalize
    blended = val_preds @ weights
    return -roc_auc_score(y_val, blended)

# Optimize blend weights
result = minimize(neg_auc, x0=np.ones(len(models)) / len(models), method="Nelder-Mead")
optimal_weights = np.abs(result.x) / np.abs(result.x).sum()

print("Optimal blend weights:")
for name, w in zip(models.keys(), optimal_weights):
    print(f"  {name}: {w:.3f}")

# Evaluate on test set
blended_test = test_preds @ optimal_weights
test_auc = roc_auc_score(y_test, blended_test)
print(f"\nBlended Test AUC: {test_auc:.4f}")

# Compare with equal-weight blend
equal_blend_auc = roc_auc_score(y_test, test_preds.mean(axis=1))
print(f"Equal-weight AUC: {equal_blend_auc:.4f}")

$ python em_blending.py
Optimal blend weights:
  rf: 0.448
  gb: 0.550
  lr: 0.001

Blended Test AUC: 0.9619
Equal-weight AUC: 0.9565

As of scikit-learn 1.9+, GradientBoostingClassifier no longer accepts n_estimators positionally — pass it as a keyword (n_estimators=200), as shown above, or you’ll hit a TypeError before training even starts. (RandomForestClassifier still accepts it positionally; it’s GradientBoostingClassifier specifically that tightened its signature.) The optimizer gives logistic regression a near-zero weight (0.001): on this dataset LR is the weakest model, so minimize correctly learns to route almost all the ensemble’s trust to RF and GB. The optimized blend (0.9619) beats the equal-weight blend (0.9565) by four-tenths of a point — a small but real gain from letting the validation set pick the mix instead of averaging blindly.

Practice

1. In the voting block, drop "lr" from soft_voting’s estimators list and re-run. Does the 3-model ensemble beat the 4-model one from the lesson?

   Solution

   It should score closer to SVM’s 0.9848, since LR was the weakest model dragging the 4-model average down to 0.9816 — removing a weak, non-diverse member from a soft-voting ensemble usually helps more than adding it hurts.

2. In the stacking block, set passthrough=True on StackingClassifier and compare Stacking Test AUC to the lesson’s 0.9839.

   Solution

   passthrough=True gives the meta-learner both the base models’ predictions and the original 20 features — on a dataset where the base models already capture most of the signal, this typically changes the score only slightly, but can help when the meta-learner could exploit a feature interaction the base models missed.

3. In the manual OOF block, remove "mlp" from BASE_MODELS (it was the strongest base model at 0.9872 OOF AUC) and re-run. How much does Meta-learner Test AUC drop?

   Solution

   Expect a real drop from 0.9913 toward the 0.97-0.98 range — MLP was carrying the most weight in the meta-learner’s coefficients, so losing it removes the ensemble’s strongest signal, not just one interchangeable member.

4. In the blending block, change the method="Nelder-Mead" optimizer to "Powell" and check whether optimal_weights changes meaningfully.

   Solution

   Both are gradient-free optimizers well-suited to this small 3-weight search space; expect similar final weights and blended AUC, though the exact path to get there differs.

5. Increase X_train’s size in the blending block by changing the split from [3000, 4000] to [4000, 4500] (more training data, less validation/test data). Does Blended Test AUC improve?

   Solution

   More training data usually improves each base model’s quality, which can raise the blended AUC — but with only 500 test samples left, the reported AUC also becomes noisier, so a single re-run isn’t enough to tell signal from sampling variance.