Scikit-Learn Hyperparameter Tuning

Hyperparameters are the settings you choose before training, such as tree depth or learning rate. Every example here ran with scikit-learn 1.9.1 (Optuna 5.0.0 for one section), on the built-in breast cancer dataset unless noted.

Grid Search

GridSearchCV tries every combination in a grid and scores each one with cross-validation.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

grid = {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 20]}
search = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=0),
                      grid, cv=5, scoring="roc_auc", n_jobs=-1)
search.fit(X_train, y_train)

print("combinations x folds:", len(search.cv_results_["params"]), "x 5")
print("best params:", search.best_params_)
print(f"best CV AUC: {search.best_score_:.4f}")
print(f"test AUC:    {search.score(X_test, y_test):.4f}")

$ python grid.py
combinations x folds: 9 x 5
best params: {'max_depth': 6, 'min_samples_leaf': 5}
best CV AUC: 0.9894
test AUC:    0.9917

Nine combinations times five folds is 45 fits, plus one final refit of the winner on all training rows. The search never saw the test set, and the test set is scored only once, at the end, so no choice was made by looking at it. That is what keeps the test AUC an honest estimate; score it after every tweak and it becomes one more thing you tuned against.

Randomized Search

RandomizedSearchCV samples n_iter combinations, and it can sample from continuous distributions instead of fixed lists.

from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import RandomizedSearchCV

X, y = load_breast_cancer(return_X_y=True)
dist = {
    "learning_rate": loguniform(0.01, 0.5),
    "max_leaf_nodes": randint(4, 64),
    "min_samples_leaf": randint(5, 60),
    "l2_regularization": loguniform(1e-4, 10),
}
search = RandomizedSearchCV(HistGradientBoostingClassifier(random_state=0), dist,
                            n_iter=25, cv=5, scoring="roc_auc", random_state=0, n_jobs=-1)
search.fit(X, y)
print(f"best CV AUC: {search.best_score_:.4f}")
for k, v in search.best_params_.items():
    print(f"  {k:18s} {v:.4g}")

$ python random_search.py
best CV AUC: 0.9952
  l2_regularization  0.0001262
  learning_rate      0.2598
  max_leaf_nodes     19
  min_samples_leaf   52

Four parameters, two of them continuous, would be impossible to grid exhaustively; 25 random draws cover the space well enough to find a strong model. loguniform spreads samples evenly on a log scale, which suits parameters whose plausible values range over several orders of magnitude, such as l2_regularization from 0.0001 to 10.

best_score_ Is Optimistic

The best of many scores is partly luck. On random labels, where the honest accuracy is 0.5, compare the search’s own best score with nested cross-validation.

import numpy as np
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(0)
X = rng.standard_normal((100, 10))
y = rng.integers(0, 2, 100)                    # labels are coin flips

grid = {"n_neighbors": list(range(1, 40)), "weights": ["uniform", "distance"]}
search = GridSearchCV(KNeighborsClassifier(), grid, cv=5).fit(X, y)
nested = cross_val_score(GridSearchCV(KNeighborsClassifier(), grid, cv=5), X, y, cv=5)

print("configurations tried:", len(search.cv_results_["params"]))
print(f"search.best_score_:  {search.best_score_:.3f}")
print(f"nested CV estimate:  {nested.mean():.3f}")

$ python optimism.py
configurations tried: 78
search.best_score_:  0.590
nested CV estimate:  0.490

With 78 configurations to choose from, the search found one that happened to fit these folds’ noise, and reported 0.59 on data with nothing to learn. Nested cross-validation runs the whole search again inside each outer fold and scores it on rows the search never saw, which brings the estimate back to about a coin flip. Report the nested number, or a single held-out test score, not best_score_.

Successive Halving

HalvingRandomSearchCV starts many candidates on a small budget and gives more samples only to the survivors of each round.

from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.experimental import enable_halving_search_cv  # noqa: F401
from sklearn.model_selection import HalvingRandomSearchCV

X, y = load_breast_cancer(return_X_y=True)
dist = {"learning_rate": loguniform(0.01, 0.5), "max_leaf_nodes": randint(4, 64)}
search = HalvingRandomSearchCV(HistGradientBoostingClassifier(random_state=0), dist,
                               n_candidates=48, factor=3, cv=5, random_state=0)
search.fit(X, y)

for i, (n, r) in enumerate(zip(search.n_candidates_, search.n_resources_)):
    print(f"round {i}: {n:2d} candidates x {r:3d} samples")
print(f"best CV accuracy: {search.best_score_:.4f}")

$ python halving.py
round 0: 48 candidates x  20 samples
round 1: 16 candidates x  60 samples
round 2:  6 candidates x 180 samples
round 3:  2 candidates x 540 samples
best CV accuracy: 0.9740

Round 0 screens all 48 candidates on only 20 samples each, and every round keeps the best third and triples their samples. Only 2 candidates ever see 540 rows, so the search costs far less than giving all 48 the full data. The trade-off is that a candidate that only shines with lots of data can be dropped early.

Bayesian Optimisation with Optuna

Optuna (pip install optuna) proposes each new trial based on how earlier trials scored.

import optuna
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

optuna.logging.set_verbosity(optuna.logging.WARNING)
X, y = load_breast_cancer(return_X_y=True)

def objective(trial):
    model = HistGradientBoostingClassifier(
        learning_rate=trial.suggest_float("learning_rate", 0.01, 0.5, log=True),
        max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 4, 64),
        random_state=0)
    return cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=25)
print(f"best CV AUC: {study.best_value:.4f}")
print("best params:", {k: round(v, 4) for k, v in study.best_params.items()})

$ python optuna_search.py
best CV AUC: 0.9949
best params: {'learning_rate': 0.4337, 'max_leaf_nodes': 27}

Optuna reaches the same level as the random search above in the same 25 trials. On a dataset this easy most reasonable settings score above 0.99, so smarter sampling has little room to help; its advantage shows on slow models and larger search spaces, where each saved trial matters.

Early Stopping Instead of Tuning n_estimators

For boosting, the number of trees doesn’t need a search: early stopping ends training when a held-out slice stops improving.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

model = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True,
                                       validation_fraction=0.1, n_iter_no_change=20,
                                       random_state=0).fit(X_train, y_train)
print(f"trees built: {model.n_iter_} of a possible 1000")
print(f"test accuracy: {model.score(X_test, y_test):.4f}")

$ python early_stopping.py
trees built: 66 of a possible 1000
test accuracy: 0.9561

The model stopped after 66 trees because 20 rounds in a row brought no improvement on the internal 10% validation slice. Early stopping picks the tree count per fit automatically, so searches can focus on learning rate and tree shape instead.

Practice

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

X, y = load_breast_cancer(return_X_y=True)
grid = {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 20]}
search = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=0),
                      grid, cv=5, scoring="roc_auc", n_jobs=-1).fit(X, y)

res = pd.DataFrame(search.cv_results_)
cols = ["param_max_depth", "param_min_samples_leaf", "mean_test_score", "std_test_score"]
print(res.sort_values("rank_test_score")[cols].head(3).round(4).to_string(index=False))

$ python p1_results.py
param_max_depth  param_min_samples_leaf  mean_test_score  std_test_score
              6                       1           0.9912          0.0068
           None                       5           0.9911          0.0071
              6                       5           0.9908          0.0071

import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(0)
X = rng.standard_normal((100, 10))
y = rng.integers(0, 2, 100)

grids = {"1 config": {"n_neighbors": [5]},
         "78 configs": {"n_neighbors": list(range(1, 40)),
                        "weights": ["uniform", "distance"]}}
for name, grid in grids.items():
    s = GridSearchCV(KNeighborsClassifier(), grid, cv=5).fit(X, y)
    print(f"{name:10s} best_score_ {s.best_score_:.3f}")

$ python p2_one_config.py
1 config   best_score_ 0.400
78 configs best_score_ 0.590

from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import RandomizedSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), SVC())
print([name for name in pipe.named_steps])
search = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e2),
                                   "svc__gamma": loguniform(1e-4, 1e-1)},
                            n_iter=20, cv=5, random_state=0).fit(X, y)
print(f"best CV accuracy {search.best_score_:.4f}")
print({k: round(float(v), 4) for k, v in search.best_params_.items()})

$ python p3_pipeline_search.py
['standardscaler', 'svc']
best CV accuracy 0.9737
{'svc__C': 14.6859, 'svc__gamma': 0.0039}