Scikit-Learn Classification

Classification predicts a category: spam or not, fraud or not, which of three species. Every example here ran with scikit-learn 1.9.1, mostly on the built-in breast cancer dataset (569 tumours, 30 features, benign or malignant).

A Logistic Regression Baseline

Logistic regression is fast and interpretable, which makes it the right first model.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), digits=3))
print(f"ROC AUC: {roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]):.4f}")

$ python baseline.py
              precision    recall  f1-score   support

           0      0.976     0.976     0.976        42
           1      0.986     0.986     0.986        72

    accuracy                          0.982       114
   macro avg      0.981     0.981     0.981       114
weighted avg      0.982     0.982     0.982       114

ROC AUC: 0.9954

In this dataset class 0 is malignant and class 1 is benign. Recall for class 0 is the share of malignant tumours the model caught, which matters more here than overall accuracy. ROC AUC uses the probabilities rather than the labels, so it measures how well the model ranks malignant above benign across every possible threshold.

Random Forest and Feature Importance

A random forest averages many decision trees trained on random subsets of rows and features.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target)

rf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
print(f"test accuracy: {rf.score(X_test, y_test):.4f}")

top = np.argsort(rf.feature_importances_)[::-1][:5]
for i in top:
    print(f"  {data.feature_names[i]:22s} {rf.feature_importances_[i]:.3f}")

$ python random_forest.py
test accuracy: 0.9561
  worst perimeter        0.133
  worst area             0.128
  worst concave points   0.108
  mean concave points    0.094
  worst radius           0.091

No scaling was needed, since trees split on thresholds. feature_importances_ measures how much each feature reduced impurity across all trees. It is a quick first look, but it favours high-cardinality features and spreads credit across correlated ones, such as the several “worst” size measurements here.

Comparing Classifiers Fairly

A single split can flatter one model by luck. Cross-validation scores each model on five different held-out folds.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
models = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "forest": RandomForestClassifier(random_state=0, n_jobs=-1),
    "hist_gb": HistGradientBoostingClassifier(random_state=0),
    "svm_rbf": make_pipeline(StandardScaler(), SVC()),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for name, model in models.items():
    s = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")
    print(f"{name:9s} AUC {s.mean():.4f} +/- {s.std():.4f}")

$ python compare.py
logistic  AUC 0.9955 +/- 0.0056
forest    AUC 0.9923 +/- 0.0070
hist_gb   AUC 0.9940 +/- 0.0053
svm_rbf   AUC 0.9957 +/- 0.0046

All four are close, and the spread between folds is about as large as the gaps between models. On a small, clean dataset like this, the simple linear model is as good as the ensembles. Gradient boosting tends to pull ahead on larger, messier tables. SVC works with scoring="roc_auc" without probability=True because the scorer falls back to its decision_function.

predict vs predict_proba

predict returns labels; predict_proba returns the probabilities behind them.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X, y)

rows = X[[0, 19, 541]]
print("labels:       ", model.predict(rows))
print("P(benign):    ", model.predict_proba(rows)[:, 1].round(3))

$ python proba.py
labels:        [0 1 0]
P(benign):     [0.    0.926 0.454]

The first two rows are confident calls, while the third sits close to the 0.5 boundary. The label hides that difference; the probability exposes it, which is what you need to route uncertain cases to a human or to move the threshold.

The Accuracy Trap on Imbalanced Data

When positives are rare, accuracy stops meaning much. A baseline that always predicts the majority class shows how much.

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import accuracy_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

for model in [DummyClassifier(strategy="most_frequent"),
              HistGradientBoostingClassifier(random_state=0)]:
    pred = model.fit(X_train, y_train).predict(X_test)
    print(f"{type(model).__name__:31s} accuracy {accuracy_score(y_test, pred):.3f}"
          f"  recall(positive) {recall_score(y_test, pred):.3f}")

$ python imbalance_trap.py
DummyClassifier                 accuracy 0.945  recall(positive) 0.000
HistGradientBoostingClassifier  accuracy 0.957  recall(positive) 0.473

The dummy scores 94.5% accuracy while catching zero positives. The real model beats it by barely a point of accuracy, and its recall shows it still misses about half of the rare class, a weakness accuracy alone would hide. On imbalanced problems, report recall, precision or F1 for the class you care about.

Tuning the Decision Threshold

The default 0.5 threshold is rarely best for a rare class. TunedThresholdClassifierCV picks a threshold by cross-validation on the training rows only.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import f1_score
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split

X, y = make_classification(n_samples=5000, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

base = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train)
tuned = TunedThresholdClassifierCV(HistGradientBoostingClassifier(random_state=0),
                                   scoring="f1", cv=5).fit(X_train, y_train)

print(f"threshold 0.50  test F1 {f1_score(y_test, base.predict(X_test)):.3f}")
print(f"threshold {tuned.best_threshold_:.2f}  test F1 {f1_score(y_test, tuned.predict(X_test)):.3f}")

$ python threshold.py
threshold 0.50  test F1 0.547
threshold 0.07  test F1 0.603

The tuned threshold was chosen without ever seeing the test rows, so its test F1 is an honest estimate. Picking the threshold that maximises F1 on the test set itself, a common shortcut, would report a number the model can’t reproduce on new data.

Practice

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

for cw in [None, "balanced"]:
    pred = LogisticRegression(class_weight=cw).fit(X_train, y_train).predict(X_test)
    print(f"class_weight={str(cw):8s} recall {recall_score(y_test, pred):.3f}"
          f"  precision {precision_score(y_test, pred):.3f}")

$ python p1_balanced.py
class_weight=None     recall 0.364  precision 0.667
class_weight=balanced recall 0.782  precision 0.232

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
print(confusion_matrix(y_test, model.predict(X_test)))

$ python p2_confusion.py
[[41  1]
 [ 1 71]]

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split

X, y = make_classification(n_samples=5000, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

tuned = TunedThresholdClassifierCV(HistGradientBoostingClassifier(random_state=0),
                                   scoring="balanced_accuracy", cv=5).fit(X_train, y_train)
pred = tuned.predict(X_test)
print(f"threshold {tuned.best_threshold_:.3f}")
print(f"recall {recall_score(y_test, pred):.3f}  precision {precision_score(y_test, pred):.3f}")

$ python p3_recall_threshold.py
threshold 0.010
recall 0.727  precision 0.494