Machine Learning Cross-Validation

Real-World Scenario

A data scientist trains a fraud detection model with a 95% accuracy on the test set. But the dataset has 1000 frauds out of 100,000 transactions. Their single test split happened to capture most frauds in training. Stratified 5-fold CV with the right metrics reveals the true picture: 72% recall on fraud. Without proper cross-validation, they would have shipped a broken model.

K-Fold Cross-Validation

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import KFold, StratifiedKFold, cross_validate
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
import pandas as pd

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model",  RandomForestClassifier(n_estimators=100, random_state=42)),
])

# StratifiedKFold preserves class proportions — always use for classification
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

results = cross_validate(
    pipe, X, y,
    cv=cv,
    scoring=["accuracy", "f1", "roc_auc"],
    return_train_score=True,   # detect overfitting: train >> test = overfit
    n_jobs=-1,
)

for metric in ["accuracy", "f1", "roc_auc"]:
    train = results[f"train_{metric}"]
    test  = results[f"test_{metric}"]
    print(f"{metric:10s}: train={train.mean():.4f}±{train.std():.4f}  "
          f"test={test.mean():.4f}±{test.std():.4f}")

$ python cv_kfold.py
accuracy  : train=1.0000±0.0000  test=0.9561±0.0123
f1        : train=1.0000±0.0000  test=0.9651±0.0097
roc_auc   : train=1.0000±0.0000  test=0.9885±0.0083

Train scores are a perfect 1.0000 on every fold — an unpruned Random Forest memorizes its training set completely. The gap to the test scores (0.96-0.99) is the real signal here: it’s small enough that this forest generalizes well despite the training-set overfit, which is exactly what return_train_score=True is for — spotting when that gap gets big.

Leave-One-Out and Other Strategies

from sklearn.model_selection import LeaveOneOut, ShuffleSplit, RepeatedStratifiedKFold
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = load_iris(return_X_y=True)
model = KNeighborsClassifier(n_neighbors=5)

# Leave-One-Out — use every sample as a test set once
# Very high variance but useful for tiny datasets (< 50 samples)
loo = LeaveOneOut()
loo_scores = cross_val_score(model, X, y, cv=loo, scoring="accuracy")
print(f"LOO Accuracy: {loo_scores.mean():.4f}")

# ShuffleSplit — random train/test splits, repeated
ss = ShuffleSplit(n_splits=10, test_size=0.2, random_state=42)
ss_scores = cross_val_score(model, X, y, cv=ss, scoring="accuracy")
print(f"ShuffleSplit Accuracy: {ss_scores.mean():.4f} ± {ss_scores.std():.4f}")

# Repeated StratifiedKFold — more stable estimate via multiple repetitions
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)
rskf_scores = cross_val_score(model, X, y, cv=rskf, scoring="accuracy", n_jobs=-1)
print(f"Repeated 5-fold Accuracy: {rskf_scores.mean():.4f} ± {rskf_scores.std():.4f}")

$ python cv_strategies.py
LOO Accuracy: 0.9667
ShuffleSplit Accuracy: 0.9700 ± 0.0277
Repeated 5-fold Accuracy: 0.9647 ± 0.0262

All three land within a point of each other on iris (150 samples, well-separated classes) — the strategies mostly differ in how much variance they report, not in the mean. LOO’s 150 single-sample test folds give a stable mean but no usable std (each fold’s accuracy is 0 or 1); ShuffleSplit and repeated k-fold both expose a real ±0.02-0.03 spread you’d never see from one static split.

Time-Series Cross-Validation

Standard k-fold leaks future data into the past — a fatal error for time series. Use walk-forward (expanding or sliding window) splits instead.

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import root_mean_squared_error

rng = np.random.default_rng(42)
n = 500

# Simulated time series: sales with trend + seasonality + noise
dates = pd.date_range("2022-01-01", periods=n, freq="D")
trend = np.linspace(100, 200, n)
seasonality = 20 * np.sin(np.arange(n) * 2 * np.pi / 30)
noise = rng.normal(0, 5, n)
sales = trend + seasonality + noise

# Create lag features (previous days as predictors)
df = pd.DataFrame({"sales": sales}, index=dates)
for lag in [1, 7, 14, 30]:
    df[f"lag_{lag}"] = df["sales"].shift(lag)
df = df.dropna()

X = df.drop("sales", axis=1).values
y = df["sales"].values

# TimeSeriesSplit: each fold uses only past data for training
tscv = TimeSeriesSplit(n_splits=5, gap=0)

fold_rmses = []
for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]

    model = GradientBoostingRegressor(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    rmse = root_mean_squared_error(y_test, model.predict(X_test))
    fold_rmses.append(rmse)
    print(f"Fold {fold+1}: train_size={len(train_idx):4d}  test_size={len(test_idx):3d}  RMSE={rmse:.2f}")

print(f"\nMean RMSE: {np.mean(fold_rmses):.2f} ± {np.std(fold_rmses):.2f}")

$ python cv_timeseries.py
Fold 1: train_size=  80  test_size= 78  RMSE=9.79
Fold 2: train_size= 158  test_size= 78  RMSE=7.90
Fold 3: train_size= 236  test_size= 78  RMSE=9.81
Fold 4: train_size= 314  test_size= 78  RMSE=8.61
Fold 5: train_size= 392  test_size= 78  RMSE=12.01

Mean RMSE: 9.62 ± 1.40

Each fold’s training set only contains days before its test window — fold 1 trains on 80 days and tests on the next 78, fold 5 trains on 392 and tests on the final 78. RMSE doesn’t monotonically improve with more training data (fold 5 is actually the worst); the noise term in the simulated series is large enough that which 30-day season the test window lands in matters more than how much history the model saw.

Nested Cross-Validation

Nested CV provides an unbiased performance estimate when both tuning and evaluation happen on the same dataset.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import (
    StratifiedKFold, RandomizedSearchCV, cross_val_score
)
from scipy.stats import uniform, randint
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

# Inner loop: hyperparameter tuning
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=1)

# Outer loop: performance estimation
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

param_dist = {
    "n_estimators":  randint(50, 300),
    "max_depth":     randint(2, 6),
    "learning_rate": uniform(0.01, 0.2),
}

# The inner RandomizedSearchCV is the estimator in the outer CV
tuned_model = RandomizedSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_dist, n_iter=20, cv=inner_cv,
    scoring="roc_auc", n_jobs=-1, random_state=42,
)

# Nested CV: outer loop runs 5 times, each time inner loop tunes on that fold's train set
nested_auc = cross_val_score(tuned_model, X, y, cv=outer_cv, scoring="roc_auc", n_jobs=-1)

print(f"Nested CV AUC: {nested_auc.mean():.4f} ± {nested_auc.std():.4f}")
print(f"Per fold:      {nested_auc.round(4)}")

# Compare with non-nested (optimistically biased)
non_nested = RandomizedSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_dist, n_iter=20, cv=inner_cv,
    scoring="roc_auc", n_jobs=-1, random_state=42,
)
non_nested.fit(X, y)
print(f"\nNon-nested best CV AUC: {non_nested.best_score_:.4f}  "
      f"(optimistic bias: {non_nested.best_score_ - nested_auc.mean():.4f})")

$ python cv_nested.py
Nested CV AUC: 0.9916 ± 0.0059
Per fold:      [0.9984 0.9843 0.9864 0.9983 0.9906]

Non-nested best CV AUC: 0.9913  (optimistic bias: -0.0003)

On this dataset the “optimistic bias” is essentially zero — breast cancer is a clean, well-separated 569-sample dataset, so RandomizedSearchCV’s inner 3-fold tuning doesn’t have much room to overfit its own validation folds. Nested CV’s real value shows up on smaller or noisier datasets, where the inner loop can pick hyperparameters that happen to fit its particular folds rather than the underlying pattern; here it mainly confirms that the non-nested estimate was already trustworthy, at the cost of 5x more compute (one full hyperparameter search per outer fold instead of one).

Learning Curves — Diagnose with Data

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import learning_curve, StratifiedKFold
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
model = GradientBoostingClassifier(n_estimators=100, random_state=42)
cv    = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

train_sizes, train_scores, val_scores = learning_curve(
    model, X, y,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=cv,
    scoring="roc_auc",
    n_jobs=-1,
)

print("Learning curve (AUC):")
print(f"{'Train size':>12} {'Train AUC':>12} {'Val AUC':>12} {'Gap':>8}")
for size, tr, val in zip(train_sizes, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    gap = tr - val
    status = "⚠ overfit" if gap > 0.05 else "✓"
    print(f"{size:12d} {tr:12.4f} {val:12.4f} {gap:8.4f} {status}")

# Interpretation:
# - High gap between train and val → overfitting (add data or regularize)
# - Both low → underfitting (more complex model)
# - Val plateaus → more data won't help (hit model capacity ceiling)

$ python cv_learningcurve.py
Learning curve (AUC):
  Train size    Train AUC      Val AUC      Gap
          45       1.0000       0.9145   0.0855 ⚠ overfit
          91       1.0000       0.9522   0.0478 ✓
         136       1.0000       0.9815   0.0185 ✓
         182       1.0000       0.9864   0.0136 ✓
         227       1.0000       0.9878   0.0122 ✓
         273       1.0000       0.9893   0.0107 ✓
         318       1.0000       0.9892   0.0108 ✓
         364       1.0000       0.9918   0.0082 ✓
         409       1.0000       0.9911   0.0089 ✓
         455       1.0000       0.9929   0.0071 ✓

Only the smallest training slice (45 samples) crosses the 0.05 overfit threshold — with that little data, GradientBoostingClassifier fits the training folds almost perfectly while validation lags well behind. By 91 samples the gap is already under control, and val AUC keeps climbing slowly all the way to the full 455-sample training size, meaning more data would still help slightly rather than the curve having flattened out.

Practice

1. In the k-fold block, drop StandardScaler from the pipeline (fit RandomForestClassifier directly on X). Does removing it change the CV scores?

   Solution

   Scores stay essentially the same — tree-based models split on raw feature thresholds and don’t care about feature scale, unlike the linear/SVM/KNN models in the supervised-learning lesson where scaling matters a lot.

2. Change LeaveOneOut in the strategies block to KFold(n_splits=10) and compare how long each takes to run (time.perf_counter()), plus how the reported accuracy changes.

   Solution

   KFold(n_splits=10) finishes far faster than LOO’s 150 individual fits, and the mean accuracy is close to LOO’s — the practical case for LOO is when you truly can’t spare samples for validation, not for typical-sized datasets.

3. In the time-series block, add gap=7 to TimeSeriesSplit and re-run. What does the gap parameter protect against?

   Solution

   gap=7 inserts a 7-day buffer between each fold’s train and test windows, dropping the last 7 training days closest to the test window — this simulates a real deployment lag (e.g., you won’t have yesterday’s sales figures as a feature) and prevents the lag features from leaking information that’s too close to the test period.

4. In the nested CV block, shrink the dataset to a random 100-row subsample of load_breast_cancer (np.random.default_rng(0).choice(len(X), 100, replace=False)) and re-run. Does the “optimistic bias” number change?

   Solution

   With only 100 rows split across 3 inner folds (≈33 rows each), the inner search has far less data to tune on, so it’s more likely to pick hyperparameters that overfit those specific folds — expect the non-nested estimate to run noticeably higher than the nested one now, unlike the near-zero gap on the full 569-row dataset.

5. In the learning curve block, swap GradientBoostingClassifier for DecisionTreeClassifier(max_depth=None) (unpruned). How does the train/val gap compare across training sizes?

   Solution

   An unpruned tree memorizes every training set size perfectly (train AUC pinned at 1.0) while validation AUC stays lower and noisier than the gradient boosting curve — a textbook high-variance learner that needs max_depth limited or an ensemble wrapper to close the gap.