Classical ML Depth

Tabular problems still dominate industry ML, so this round goes deep on trees and ensembles. The questions are all variations on: what does this do to bias and variance, and why.

The setup

import numpy as np, time
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (RandomForestClassifier, BaggingClassifier,
                              GradientBoostingClassifier, ExtraTreesClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, accuracy_score

X, y = make_classification(n_samples=12_000, n_features=25, n_informative=10,
                           n_redundant=5, class_sep=0.9, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.3, random_state=42, stratify=y)
print(f"train {Xtr.shape}  test {Xte.shape}  positive rate {y.mean():.2f}")

train (8400, 25)  test (3600, 25)  positive rate 0.50

”Why does a single tree overfit?”

print(f"{'max_depth':>10} {'leaves':>8} {'train acc':>10} {'test acc':>9} {'gap':>7}")
for d in (2, 5, 10, 20, None):
    t = DecisionTreeClassifier(max_depth=d, random_state=42).fit(Xtr, ytr)
    tr = accuracy_score(ytr, t.predict(Xtr))
    te = accuracy_score(yte, t.predict(Xte))
    print(f"{str(d):>10} {t.get_n_leaves():>8} {tr:>10.4f} {te:>9.4f} {tr-te:>7.4f}")

         2        4     0.7420    0.7386  0.0034
         5       32     0.8404    0.8103  0.0301
        10      554     0.9433    0.8281  0.1152
        20     2489     0.9998    0.8106  0.1892
      None     2601     1.0000    0.8103  0.1897

100% training accuracy, 81% test. The mechanism is worth stating precisely: “An unpruned tree keeps splitting until every leaf is pure, so with 8,400 rows it can memorise the training set — 2,601 leaves for 8,400 rows is roughly three rows per leaf. Every leaf is then fitting noise, and the decision boundary is a staircase around individual points."

"How does a random forest fix it?”

results = {}
for name, model in [
    ("single tree (depth 20)", DecisionTreeClassifier(max_depth=20, random_state=42)),
    ("bagging (no feature sub)", BaggingClassifier(
        DecisionTreeClassifier(max_depth=20, random_state=42), n_estimators=200, random_state=42)),
    ("random forest", RandomForestClassifier(n_estimators=200, max_depth=20, random_state=42, n_jobs=-1)),
    ("extra trees", ExtraTreesClassifier(n_estimators=200, max_depth=20, random_state=42, n_jobs=-1)),
]:
    m = model.fit(Xtr, ytr)
    results[name] = (accuracy_score(ytr, m.predict(Xtr)),
                     accuracy_score(yte, m.predict(Xte)),
                     roc_auc_score(yte, m.predict_proba(Xte)[:, 1]))

print(f"{'model':<26} {'train':>7} {'test':>7} {'AUC':>7}")
for k, (tr, te, auc) in results.items():
    print(f"{k:<26} {tr:>7.4f} {te:>7.4f} {auc:>7.4f}")

model                        train    test     AUC
single tree (depth 20)      0.9998  0.8106  0.8107
bagging (no feature sub)    0.9998  0.8756  0.9438
random forest               0.9998  0.8869  0.9530
extra trees                 0.9998  0.8842  0.9524

Read it as two separate effects. Bagging alone takes test accuracy from 0.811 to 0.876 — averaging many high-variance models cancels their errors, because each bootstrap sample produces a different tree and the noise-fitting is uncorrelated. Random forest adds a second decorrelation — a random subset of features at each split — for another point.

“Averaging reduces variance in proportion to how uncorrelated the members are. Bootstrap sampling gets you part of the way; if one feature is very strong, every tree splits on it first and the trees stay correlated, so max_features forces them apart. That is the whole difference between bagging and a random forest, and it is why Extra Trees randomises the split threshold too.”

Note that training accuracy stayed at 0.9998 — the ensemble still memorises the training data, which is why train accuracy is useless as a diagnostic here and out-of-bag or cross-validated error is not:

rf_oob = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42, n_jobs=-1).fit(Xtr, ytr)
print(f"OOB score {rf_oob.oob_score_:.4f}   test accuracy {accuracy_score(yte, rf_oob.predict(Xte)):.4f}")

OOB score 0.8861   test accuracy 0.8894

Out-of-bag error is a free validation estimate — each tree is scored on the ~37% of rows its bootstrap sample excluded. Mentioning it is a nice detail.

”Bagging or boosting?”

print(f"{'n_estimators':>13} {'RF test':>9} {'GB test':>9} {'GB train':>9}")
for n in (5, 25, 100, 400):
    rf = RandomForestClassifier(n_estimators=n, random_state=42, n_jobs=-1).fit(Xtr, ytr)
    gb = GradientBoostingClassifier(n_estimators=n, learning_rate=0.1,
                                    max_depth=3, random_state=42).fit(Xtr, ytr)
    print(f"{n:>13} {accuracy_score(yte, rf.predict(Xte)):>9.4f} "
          f"{accuracy_score(yte, gb.predict(Xte)):>9.4f} "
          f"{accuracy_score(ytr, gb.predict(Xtr)):>9.4f}")

            5    0.8394    0.8022    0.8093
           25    0.8747    0.8611    0.8752
          100    0.8858    0.8908    0.9134
          400    0.8886    0.8956    0.9648

The contrast to draw:

The key asymmetry, said plainly: “Adding trees to a random forest converges and stops helping — it will not overfit. Adding trees to a boosted model keeps reducing training error and will eventually overfit, which is why boosting needs early stopping and a forest does not.”

Watch it happen:

from sklearn.model_selection import train_test_split as tts
Xf, Xv, yf, yv = tts(Xtr, ytr, test_size=0.2, random_state=1, stratify=ytr)

gb = GradientBoostingClassifier(n_estimators=1500, learning_rate=0.1, max_depth=5,
                                random_state=42).fit(Xf, yf)
val_auc = [roc_auc_score(yv, p[:, 1]) for p in gb.staged_predict_proba(Xv)]
best = int(np.argmax(val_auc))
print(f"best validation AUC {max(val_auc):.4f} at {best+1} trees")
print(f"AUC at 1500 trees   {val_auc[-1]:.4f}")
for n in (50, 200, best+1, 800, 1500):
    print(f"  {n:>5} trees → val AUC {val_auc[n-1]:.4f}")

best validation AUC 0.9541 at 213 trees
AUC at 1500 trees   0.9418
    50 trees → val AUC 0.9407
   200 trees → val AUC 0.9539
   213 trees → val AUC 0.9541
   800 trees → val AUC 0.9476
  1500 trees → val AUC 0.9418

Peaks at 213 and declines to 0.9418. staged_predict_proba walking the ensemble is the direct demonstration, and the practical form is n_iter_no_change:

gb_es = GradientBoostingClassifier(n_estimators=1500, learning_rate=0.1, max_depth=5,
                                   validation_fraction=0.2, n_iter_no_change=25,
                                   random_state=42).fit(Xtr, ytr)
print(f"early stopping chose {gb_es.n_estimators_} trees, "
      f"test AUC {roc_auc_score(yte, gb_es.predict_proba(Xte)[:,1]):.4f}")

early stopping chose 241 trees, test AUC 0.9548

“How would you tune it?”

The order matters, and saying the order is the answer:

print(f"{'lr':>6} {'trees for equal fit':>21} {'test AUC':>9} {'fit time':>9}")
for lr in (0.3, 0.1, 0.03):
    t0 = time.perf_counter()
    m = GradientBoostingClassifier(n_estimators=2000, learning_rate=lr, max_depth=3,
                                   validation_fraction=0.2, n_iter_no_change=30,
                                   random_state=42).fit(Xtr, ytr)
    print(f"{lr:>6} {m.n_estimators_:>21} "
          f"{roc_auc_score(yte, m.predict_proba(Xte)[:,1]):>9.4f} "
          f"{time.perf_counter()-t0:>8.1f}s")

   0.3                   118    0.9497      1.4s
   0.1                   347    0.9531      3.9s
  0.03                  1104    0.9542     11.9s

Learning rate and tree count trade off directly — a third of the rate needs roughly three times the trees for the same fit, with a small accuracy gain and 8× the time. So:

1. Fix a low learning rate (0.05-0.1) and let early stopping pick n_estimators.
2. Tune capacity: max_depth (3-8) or num_leaves for LightGBM.
3. Tune sampling: subsample, colsample_bytree — these regularise and speed things up.
4. Tune regularisation: min_child_weight, reg_lambda, reg_alpha.

“I would use random search or Bayesian optimisation rather than grid search — with six parameters, a grid spends most of its budget on dimensions that do not matter, and random search covers the important ones more densely for the same number of fits."

"Is feature importance trustworthy?”

This is the trap question in this round.

from sklearn.inspection import permutation_importance

# add a high-cardinality noise feature and a duplicate of a real one
rng = np.random.default_rng(0)
X_aug = np.column_stack([Xtr, rng.normal(size=len(Xtr)), Xtr[:, 0] + rng.normal(0, .01, len(Xtr))])
Xte_aug = np.column_stack([Xte, rng.normal(size=len(Xte)), Xte[:, 0] + rng.normal(0, .01, len(Xte))])
names = [f"f{i}" for i in range(25)] + ["pure_noise", "copy_of_f0"]

rf = RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1).fit(X_aug, ytr)

imp = sorted(zip(names, rf.feature_importances_), key=lambda t: -t[1])[:6]
print("impurity-based (train):")
for n_, v in imp:
    print(f"  {n_:<12} {v:.4f}")

perm = permutation_importance(rf, Xte_aug, yte, n_repeats=10, random_state=42, n_jobs=-1)
print("\npermutation (held-out test):")
for i in np.argsort(-perm.importances_mean)[:6]:
    print(f"  {names[i]:<12} {perm.importances_mean[i]:.4f} ± {perm.importances_std[i]:.4f}")
print(f"\n  pure_noise impurity {rf.feature_importances_[25]:.4f}  "
      f"permutation {perm.importances_mean[25]:+.4f}")

impurity-based (train):
  f11          0.0904
  f4           0.0871
  f0           0.0605
  copy_of_f0   0.0592
  f19          0.0548
  pure_noise   0.0281

permutation (held-out test):
  f11          0.0641 ± 0.0071
  f4           0.0598 ± 0.0064
  f19          0.0349 ± 0.0048
  f2           0.0281 ± 0.0043
  f0           0.0102 ± 0.0029
  copy_of_f0   0.0097 ± 0.0031

  pure_noise impurity 0.0281  permutation -0.0004

Three findings to name:

1. Pure noise scores 0.0281 on impurity importance — sixth out of 27 — because a continuous random feature offers many split points and can always reduce impurity slightly on the training data. Permutation importance gives it −0.0004: correctly nothing.
2. f0 and its near-duplicate split the credit (0.0605 and 0.0592). Neither looks important, though together they are. Correlated features always do this.
3. Permutation importance has an uncertainty estimate, which impurity importance does not.

“Impurity importance is computed on training data and is biased towards high-cardinality features. I use permutation importance on a held-out set for a global picture, and SHAP when I need per-prediction attribution — with the caveat that SHAP is expensive and that neither method is causal. High importance means the model relies on it, not that it drives the outcome.”

That last sentence is the one that gets remembered.

”When would you not use boosting?”

lr_pipe = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
gb_final = GradientBoostingClassifier(n_estimators=300, random_state=42).fit(Xtr, ytr)

for name, m in [("logistic regression", lr_pipe), ("gradient boosting", gb_final)]:
    t0 = time.perf_counter()
    for _ in range(100):
        m.predict_proba(Xte[:1])
    latency = (time.perf_counter() - t0) / 100 * 1000
    print(f"{name:<22} AUC {roc_auc_score(yte, m.predict_proba(Xte)[:,1]):.4f}   "
          f"single-row latency {latency:.3f} ms")

logistic regression    AUC 0.9294   single-row latency 0.081 ms
gradient boosting      AUC 0.9531   single-row latency 1.942 ms

24× the latency for 2.4 points of AUC. Whether that is worth it is a product question, and saying so is the point:

• Interpretability is a hard requirement — credit decisions, clinical use, anywhere a regulator asks “why was this person rejected”.
• Latency budget is tight — a linear model is a dot product.
• Very few rows — under a few thousand, a regularised linear model is often as good and far more stable.
• Extrapolation is needed — trees cannot predict outside the range of the training targets:

Xr = np.linspace(0, 10, 200).reshape(-1, 1)
yr = 3 * Xr.ravel() + 1
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
rf_r = RandomForestRegressor(n_estimators=50, random_state=0).fit(Xr, yr)
lin_r = LinearRegression().fit(Xr, yr)
future = np.array([[15.0], [20.0]])
print(f"true values at x=15, 20: {3*15+1}, {3*20+1}")
print(f"random forest predicts:  {rf_r.predict(future).round(2)}")
print(f"linear regression:       {lin_r.predict(future).round(2)}")

true values at x=15, 20: 46, 61
random forest predicts:  [30.42 30.42]
linear regression:       [46. 61.]

The forest predicts 30.42 for both — the mean of its highest leaf — because a tree cannot produce a value outside the training range. For a trending time series or any extrapolation, that is disqualifying, and it is a favourite follow-up.

Quick answers worth rehearsing

Gini vs entropy: near-identical in practice; Gini is slightly cheaper (no logarithm). Choosing between them is not where accuracy comes from.

How trees handle missing values: CART surrogate splits, LightGBM and XGBoost learn a default direction per split from the data. That is a genuine advantage over linear models, which need imputation.

XGBoost vs LightGBM vs CatBoost: XGBoost is level-wise and the most mature; LightGBM is leaf-wise, much faster on large data, and more prone to overfitting on small data; CatBoost handles categorical features natively with ordered target statistics and needs the least tuning.

Class imbalance in trees: class_weight='balanced' or scale_pos_weight, plus a threshold chosen from costs — and prefer that to resampling, which discards data or invents it.

The scoring

Practice

None  2601 leaves  train 1.0000  test 0.8103

bagging       0.8756
random forest 0.8869

best 0.9541 at 213 trees;  0.9418 at 1500

pure_noise  impurity 0.0281   permutation -0.0004

Next: deep learning questions — what interviewers actually ask outside research roles.