Supervised Learning Algorithms

Real-World Scenario

A bank needs to predict loan defaults. They evaluate 5 algorithms to find the best trade-off between predictive power, interpretability for regulators, and training speed. The comparison reveals that HistGradientBoosting gives the best AUC, but logistic regression is chosen for the core model because regulators require feature-weight explainability.

Linear Models

from sklearn.datasets import make_classification, make_regression
from sklearn.linear_model import (
    LogisticRegression, Ridge, Lasso, ElasticNet, SGDClassifier
)
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, roc_auc_score
import numpy as np

# Classification dataset
X, y = make_classification(n_samples=5000, n_features=20, n_informative=10,
                            n_redundant=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Linear models require feature scaling
scaler  = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# Logistic Regression — interpretable, fast, strong baseline
lr = LogisticRegression(
    C=1.0,           # inverse of regularization strength — smaller = more regularization
    penalty="l2",    # L2 ridge regularization (use "l1" for feature selection)
    max_iter=1000,
    solver="lbfgs",
    random_state=42,
)
lr.fit(X_train_s, y_train)
lr_auc = roc_auc_score(y_test, lr.predict_proba(X_test_s)[:, 1])
print(f"Logistic Regression AUC: {lr_auc:.4f}")

# Feature importance from coefficients
feature_importance = np.abs(lr.coef_[0])
top_features = np.argsort(feature_importance)[::-1][:5]
print(f"Top 5 features by |coefficient|: {top_features}")

# SGD Classifier — same math, scales to millions of samples
sgd = SGDClassifier(loss="log_loss", penalty="l2", alpha=1e-4,
                    max_iter=100, random_state=42)
sgd.fit(X_train_s, y_train)
sgd_auc = roc_auc_score(y_test, sgd.predict_proba(X_test_s)[:, 1])
print(f"SGD Classifier AUC:      {sgd_auc:.4f}")

$ python sl_linear.py
Logistic Regression AUC: 0.9103
Top 5 features by |coefficient|: [17  9  6 13  3]
SGD Classifier AUC:      0.8980

The two models land within a percentage point of each other — expected, since both fit a linear decision boundary and differ mainly in how they optimize it (closed-form solver vs. stochastic gradient steps). On scikit-learn 1.9+, this also prints a FutureWarning that penalty is deprecated in favor of l1_ratio; the code above still runs, but a newer scikit-learn release will require the replacement.

Decision Trees and Ensembles

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (
    RandomForestClassifier,
    GradientBoostingClassifier,
    HistGradientBoostingClassifier,
    AdaBoostClassifier,
    ExtraTreesClassifier,
)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = make_classification(n_samples=5000, n_features=20, n_informative=10,
                            random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

algorithms = {
    "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=200, max_features="sqrt",
                                             random_state=42, n_jobs=-1),
    "Extra Trees":   ExtraTreesClassifier(n_estimators=200, random_state=42, n_jobs=-1),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                                     max_depth=4, random_state=42),
    "HistGradientBoosting": HistGradientBoostingClassifier(max_iter=200, learning_rate=0.1,
                                                            max_depth=4, random_state=42),
}

print(f"{'Algorithm':<25} {'AUC':>8} {'CV AUC':>10}")
print("-" * 45)
for name, clf in algorithms.items():
    clf.fit(X_train, y_train)
    test_auc = roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1])
    cv_auc   = cross_val_score(clf, X, y, cv=5, scoring="roc_auc", n_jobs=-1).mean()
    print(f"{name:<25} {test_auc:>8.4f} {cv_auc:>10.4f}")

$ python sl_ensembles.py
Algorithm                      AUC     CV AUC
---------------------------------------------
Decision Tree               0.8902     0.8934
Random Forest               0.9883     0.9869
Extra Trees                 0.9910     0.9895
Gradient Boosting           0.9875     0.9881
HistGradientBoosting        0.9888     0.9879

A single decision tree trails every ensemble by roughly ten AUC points — it’s the clearest demonstration that averaging many weak, decorrelated trees beats any one of them. Extra Trees edges out the others here because its extra split-point randomization reduces variance further on this synthetic dataset; on real tabular data the boosted methods usually pull ahead instead.

How Gradient Boosting Works

import numpy as np
from sklearn.tree import DecisionTreeRegressor

class SimpleGradientBoosting:
    """Minimal gradient boosting for binary classification — shows the core algorithm."""

    def __init__(self, n_estimators=50, learning_rate=0.1, max_depth=3):
        self.n_estimators  = n_estimators
        self.learning_rate = learning_rate
        self.max_depth     = max_depth
        self.trees         = []
        self.F0            = 0.0   # initial prediction (log-odds of mean)

    def _sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

    def fit(self, X: np.ndarray, y: np.ndarray) -> "SimpleGradientBoosting":
        # Initial prediction: log-odds of the base rate
        p = y.mean()
        self.F0 = np.log(p / (1 - p))

        F = np.full(len(y), self.F0)   # running prediction (log-odds space)

        for _ in range(self.n_estimators):
            # Gradient of log-loss = residuals (actual - predicted probability)
            p_hat    = self._sigmoid(F)
            residuals = y - p_hat   # pseudo-residuals

            # Fit a regression tree to the residuals
            tree = DecisionTreeRegressor(max_depth=self.max_depth)
            tree.fit(X, residuals)

            # Update running predictions
            F += self.learning_rate * tree.predict(X)
            self.trees.append(tree)

        return self

    def predict_proba(self, X: np.ndarray) -> np.ndarray:
        F = np.full(len(X), self.F0)
        for tree in self.trees:
            F += self.learning_rate * tree.predict(X)
        probs = self._sigmoid(F)
        return np.column_stack([1 - probs, probs])

    def predict(self, X: np.ndarray) -> np.ndarray:
        return (self.predict_proba(X)[:, 1] >= 0.5).astype(int)


# Verify it works
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

custom_gb = SimpleGradientBoosting(n_estimators=100, learning_rate=0.1, max_depth=3)
custom_gb.fit(X_train, y_train)
auc = roc_auc_score(y_test, custom_gb.predict_proba(X_test)[:, 1])
print(f"Custom GradBoost AUC: {auc:.4f}")

$ python sl_gbfromscratch.py
Custom GradBoost AUC: 0.9529

100 trees of depth 3, each fit to the previous round’s residuals, gets within a couple points of scikit-learn’s tuned GradientBoostingClassifier from the block above — the whole algorithm really is just “repeatedly fit a small tree to what the last round got wrong.”

Support Vector Machines

from sklearn.svm import SVC, SVR
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = make_classification(n_samples=2000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# SVMs require feature scaling
scaler    = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# RBF kernel SVM — the standard choice for non-linear data
svm = SVC(
    C=1.0,             # regularization — smaller C = wider margin, more misclassification
    kernel="rbf",      # radial basis function kernel
    gamma="scale",     # kernel coefficient — "scale" = 1/(n_features * X.var())
    probability=True,  # enables predict_proba (slower training)
    random_state=42,
)
svm.fit(X_train_s, y_train)
svm_auc = roc_auc_score(y_test, svm.predict_proba(X_test_s)[:, 1])
print(f"SVM RBF AUC: {svm_auc:.4f}")

# Tune C and gamma
param_grid = {"C": [0.1, 1, 10], "gamma": ["scale", "auto", 0.01]}
search = GridSearchCV(
    SVC(kernel="rbf", probability=True),
    param_grid, cv=3, scoring="roc_auc", n_jobs=-1
)
search.fit(X_train_s, y_train)
print(f"Best params: {search.best_params_}")
print(f"Best CV AUC: {search.best_score_:.4f}")

# When NOT to use SVM
# - n_samples > 100,000: too slow (O(n²) to O(n³))
# - You need feature importances: SVMs don't provide them natively
# - Multiclass with many classes: use GBM or RF instead

$ python sl_svm.py
SVM RBF AUC: 0.9656
Best params: {'C': 1, 'gamma': 0.01}
Best CV AUC: 0.9515

The grid search’s best CV score (0.9515) is slightly below the single untuned split’s score (0.9656) — a reminder that a single train/test split is noisier than 3-fold CV, not that tuning made things worse. On scikit-learn 1.9+, probability=True also prints a FutureWarning pointing at CalibratedClassifierCV as the replacement; it still works here but is scheduled for removal in 1.11.

K-Nearest Neighbors

from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = make_classification(n_samples=2000, n_features=20, random_state=42)

scaler = StandardScaler()
X_s    = scaler.fit_transform(X)

# Find optimal K using cross-validation
k_values = [1, 3, 5, 7, 11, 21, 51]
for k in k_values:
    knn = KNeighborsClassifier(n_neighbors=k, metric="minkowski", p=2, n_jobs=-1)
    scores = cross_val_score(knn, X_s, y, cv=5, scoring="roc_auc")
    print(f"K={k:3d}: AUC={scores.mean():.4f} ± {scores.std():.4f}")

# KNN weaknesses:
# - Slow inference: O(n) per prediction for brute force
# - Curse of dimensionality: distance loses meaning in high-d space
# - No feature importance
# Use when: small dataset, non-linear decision boundary, interpretability needed

$ python sl_knn.py
K=  1: AUC=0.8020 ± 0.0237
K=  3: AUC=0.8884 ± 0.0101
K=  5: AUC=0.9178 ± 0.0038
K=  7: AUC=0.9248 ± 0.0067
K= 11: AUC=0.9360 ± 0.0082
K= 21: AUC=0.9454 ± 0.0077
K= 51: AUC=0.9522 ± 0.0058

AUC climbs steadily from K=1 through K=51 on this dataset — the opposite of the usual “small K overfits, large K underfits” story, because K=1’s decision boundary here is noisy enough that averaging over more neighbors keeps helping well past the range most guides suggest. This is exactly why K should be chosen by cross-validation on your own data rather than a rule of thumb.

Practice

1. In the linear models block, change n_informative from 10 to 20 (all features informative, none redundant) and re-run. Does the logistic regression AUC go up or down, and why?

   Solution

   AUC goes up — with n_redundant=5 removed and every feature informative, the model has less noise to separate signal from, so both LogisticRegression and SGDClassifier score higher than the 0.91/0.90 in the lesson.

2. Add BaggingClassifier (from sklearn.ensemble, wrapping a DecisionTreeClassifier) to the algorithms dict in the ensembles block and compare its AUC to Random Forest’s.

   Solution

   from sklearn.ensemble import BaggingClassifier
   algorithms["Bagging"] = BaggingClassifier(
       DecisionTreeClassifier(max_depth=5), n_estimators=200, random_state=42, n_jobs=-1
   )

   Bagging alone (no feature subsampling) usually scores a bit below Random Forest, since Random Forest’s per-split feature randomness decorrelates the trees further.

3. In SimpleGradientBoosting, change learning_rate from 0.1 to 0.5 with the same 100 estimators. Does AUC improve, and what does that suggest about learning rate and number of trees trading off against each other?

   Solution

   A higher learning rate reaches a similar or slightly lower AUC in fewer effective boosting rounds — each tree corrects more of the residual, so the model converges faster but can also overshoot and overfit if n_estimators stays high.

4. Run the KNN loop again with metric="manhattan" instead of minkowski/p=2 (Euclidean). Does the ranking of K values change?

   Solution

   knn = KNeighborsClassifier(n_neighbors=k, metric="manhattan", n_jobs=-1)

   The overall AUC-vs-K trend (higher K keeps helping) stays the same; Manhattan distance mainly matters when features have very different scales or when outliers should count less, both of which are already handled here by StandardScaler.

5. Time SVC(kernel="rbf") training on n_samples=2000 vs. n_samples=20000 using %timeit or time.perf_counter(). How does training time scale?

   Solution

   Training time grows much faster than linearly (SVM training is roughly quadratic to cubic in the number of samples), which is exactly the “don’t use SVM past ~100k rows” guidance in the lesson’s comments.