Scikit-Learn Advanced Model Selection

Choosing between models is itself a fitting step, and it can overfit like any other. Every example here ran with scikit-learn 1.9.1 on generated data, so the right answer is known.

Validation Curves: Where Overfitting Starts

validation_curve scores one hyperparameter across a range, on both training and validation folds.

from sklearn.datasets import make_classification
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=1500, n_features=20, n_informative=6,
                           flip_y=0.1, random_state=0)
depths = [1, 2, 4, 6, 8, 12, 20]
train, val = validation_curve(DecisionTreeClassifier(random_state=0), X, y,
                              param_name="max_depth", param_range=depths, cv=5)

print("depth  train   valid   gap")
for d, tr, va in zip(depths, train.mean(axis=1), val.mean(axis=1)):
    print(f"{d:5d}  {tr:.3f}  {va:.3f}  {tr - va:.3f}")

$ python validation_curve.py
depth  train   valid   gap
    1  0.769  0.768  0.001
    2  0.792  0.764  0.028
    4  0.861  0.817  0.044
    6  0.918  0.793  0.125
    8  0.960  0.789  0.171
   12  0.992  0.779  0.213
   20  1.000  0.773  0.227

Training accuracy climbs steadily with depth until the tree memorises everything at depth 20. Validation accuracy peaks at depth 4 (0.817) and then slides, and the gap between the two columns widens from almost nothing to 0.23. The peak of the validation column is the depth to use; the growing gap is overfitting made visible.

Learning Curves: Will More Data Help?

learning_curve retrains on growing slices of the data.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import learning_curve

X, y = make_classification(n_samples=3000, n_features=20, n_informative=10, random_state=1)
for model in [LogisticRegression(max_iter=1000), RandomForestClassifier(random_state=0)]:
    sizes, train, val = learning_curve(model, X, y, cv=5, n_jobs=-1,
                                       train_sizes=np.linspace(0.1, 1.0, 4))
    print(type(model).__name__)
    for n, tr, va in zip(sizes, train.mean(axis=1), val.mean(axis=1)):
        print(f"  {n:5d} rows  train {tr:.3f}  valid {va:.3f}")

$ python learning_curve.py
LogisticRegression
    240 rows  train 0.896  valid 0.863
    960 rows  train 0.865  valid 0.868
   1680 rows  train 0.871  valid 0.872
   2400 rows  train 0.875  valid 0.874
RandomForestClassifier
    240 rows  train 1.000  valid 0.891
    960 rows  train 1.000  valid 0.923
   1680 rows  train 1.000  valid 0.928
   2400 rows  train 1.000  valid 0.935

Logistic regression’s training and validation scores meet around 0.87 by 960 rows and stay there: the model is limited by its straight-line form, not by data, so more rows won’t help. The random forest scores 1.000 on its own training data at every size, yet its validation score is still rising at 2400 rows, so more data would probably help it further.

Grouped Rows Leak Across Folds

Each patient here contributes 10 visits, and the label is fixed per patient. The features carry a patient-specific fingerprint but no real signal about the label.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, cross_val_score

rng = np.random.default_rng(0)
patients = np.repeat(np.arange(100), 10)               # 100 patients x 10 visits
fingerprint = rng.normal(size=(100, 5))[patients]       # same per patient
X = fingerprint + rng.normal(scale=0.1, size=(1000, 5))
y = rng.integers(0, 2, 100)[patients]                   # label fixed per patient

model = RandomForestClassifier(random_state=0, n_jobs=-1)
kf = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0)).mean()
gkf = cross_val_score(model, X, y, cv=GroupKFold(5), groups=patients).mean()
print(f"KFold (visits mixed):       {kf:.3f}")
print(f"GroupKFold (by patient):    {gkf:.3f}")

$ python group_leak.py
KFold (visits mixed):       0.985
GroupKFold (by patient):    0.509

With plain KFold, about eight of each patient’s ten visits land in training, and the forest simply recognises the patient’s fingerprint: 98.5% accuracy on labels that carry no learnable signal. GroupKFold keeps every patient’s visits together, so validation patients are genuinely new, and accuracy drops to a coin flip, which is the truth.

Time Series: Don’t Train on the Future

A random walk has no predictable direction, but neighbouring days look alike. Shuffled folds exploit that.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

rng = np.random.default_rng(0)
t = np.arange(1000).reshape(-1, 1)                # the only feature: day number
y = np.cumsum(rng.normal(size=1000))             # random walk

model = RandomForestRegressor(random_state=0, n_jobs=-1)
shuffled = cross_val_score(model, t, y, cv=KFold(5, shuffle=True, random_state=0)).mean()
forward = cross_val_score(model, t, y, cv=TimeSeriesSplit(5)).mean()
print(f"shuffled KFold R2:    {shuffled:.3f}")
print(f"TimeSeriesSplit R2:   {forward:.3f}")

$ python time_leak.py
shuffled KFold R2:    0.995
TimeSeriesSplit R2:   -0.568

Shuffled folds let the forest look up the neighbouring days on both sides of each validation day, so R² is near perfect. With TimeSeriesSplit it must predict days after everything it trained on, and a random walk can’t be forecast from the day number, so R² goes negative: worse than predicting the mean. Only the second number matches how the model would be used.

Out-of-Fold Predictions

cross_val_predict gives every row a prediction from a model that never saw it.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import cross_val_predict

X, y = make_classification(n_samples=3000, weights=[0.85, 0.15], random_state=42)
model = HistGradientBoostingClassifier(random_state=0)

oof = cross_val_predict(model, X, y, cv=5, method="predict_proba")[:, 1]
insample = model.fit(X, y).predict_proba(X)[:, 1]

print(f"positive rate {y.mean():.3f}")
print(f"out-of-fold  ROC AUC {roc_auc_score(y, oof):.4f}   AP {average_precision_score(y, oof):.4f}")
print(f"in-sample    ROC AUC {roc_auc_score(y, insample):.4f}   AP {average_precision_score(y, insample):.4f}")

$ python oof.py
positive rate 0.154
out-of-fold  ROC AUC 0.9814   AP 0.9683
in-sample    ROC AUC 1.0000   AP 1.0000

In-sample, the model scores a perfect 1.0 on both metrics because it is grading rows it was trained on. The out-of-fold scores are the honest ones, and because every row has one, you can compute any metric, curve or threshold over all 3000 rows without holding data back. Average precision is the more informative number when positives are rare, since its baseline is the 15% positive rate rather than 0.5.

Practice

from sklearn.datasets import make_classification
from sklearn.model_selection import KFold, StratifiedKFold

X, y = make_classification(n_samples=500, weights=[0.95, 0.05], random_state=0)
for cv in [KFold(5, shuffle=True, random_state=0), StratifiedKFold(5, shuffle=True, random_state=0)]:
    rates = [f"{y[val].mean():.3f}" for _, val in cv.split(X, y)]
    print(f"{type(cv).__name__:16s} {rates}")

$ python p1_stratified.py
KFold            ['0.050', '0.030', '0.040', '0.090', '0.050']
StratifiedKFold  ['0.050', '0.050', '0.050', '0.050', '0.060']

import numpy as np
from sklearn.model_selection import GroupKFold, KFold

patients = np.repeat(np.arange(100), 10)
X = np.zeros((1000, 1))
splits = {"KFold": KFold(5, shuffle=True, random_state=0).split(X),
          "GroupKFold": GroupKFold(5).split(X, groups=patients)}
for name, split in splits.items():
    overlaps = [len(set(patients[tr]) & set(patients[va])) for tr, va in split]
    print(f"{name:10s} patients in both train and validation: {overlaps}")

$ python p2_group_check.py
KFold      patients in both train and validation: [88, 93, 85, 92, 87]
GroupKFold patients in both train and validation: [0, 0, 0, 0, 0]

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

X = np.arange(20).reshape(-1, 1)
for i, (tr, va) in enumerate(TimeSeriesSplit(n_splits=4, gap=1).split(X)):
    print(f"fold {i}: train {tr.min()}-{tr.max()}  validate {va.min()}-{va.max()}")

$ python p3_tss_ranges.py
fold 0: train 0-2  validate 4-7
fold 1: train 0-6  validate 8-11
fold 2: train 0-10  validate 12-15
fold 3: train 0-14  validate 16-19