XGBoost and LightGBM

XGBoost and LightGBM are the two most widely used gradient boosting libraries outside scikit-learn, and both provide scikit-learn-compatible estimators. Every example here ran with XGBoost 3.4.1, LightGBM 4.7.0 and scikit-learn 1.9.1 (pip install xgboost lightgbm).

XGBoost with Early Stopping

In XGBoost 2 and later, early stopping is a constructor argument, and the validation data goes to fit.

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10_000, n_features=30, n_informative=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=5,
                          early_stopping_rounds=50, eval_metric="auc", random_state=42)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)

print(f"best iteration: {model.best_iteration} of 2000")
print(f"test AUC: {roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]):.4f}")

$ python xgb_early.py
best iteration: 738 of 2000
test AUC: 0.9931

Training was allowed 2000 rounds but stopped 50 rounds after the validation AUC last improved, at round 738. predict_proba automatically uses the model as of the best iteration, so no extra argument is needed. The test set played no part in choosing when to stop, so its AUC is an honest estimate.

Old Tutorial Code Breaks on XGBoost 2+

Many older tutorials read best_ntree_limit or pass ntree_limit to predict. Both were removed in XGBoost 2.0.

import xgboost as xgb
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, random_state=0)
booster = xgb.train({"objective": "binary:logistic"}, xgb.DMatrix(X, label=y),
                    num_boost_round=20)
try:
    print(booster.best_ntree_limit)
except AttributeError as e:
    print("AttributeError:", e)
try:
    booster.predict(xgb.DMatrix(X), ntree_limit=10)
except TypeError as e:
    print("TypeError:", e)

print(booster.predict(xgb.DMatrix(X), iteration_range=(0, 10))[:3].round(3))

$ python xgb_old_api.py
AttributeError: 'Booster' object has no attribute 'best_ntree_limit'
TypeError: Booster.predict() got an unexpected keyword argument 'ntree_limit'
[0.898 0.967 0.967]

Both failures are real errors on current XGBoost (here 3.4), not deprecation warnings, so code copied from an older tutorial crashes. The replacement for predicting with only the first N trees is iteration_range=(0, N), which also works on the scikit-learn estimators.

LightGBM with Early Stopping

LightGBM’s scikit-learn estimator takes early stopping as a callback. Since LightGBM 4.7 the validation data goes in eval_X and eval_y; the older eval_set still works but warns.

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10_000, n_features=30, n_informative=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

model = lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31,
                           random_state=42, verbose=-1)
model.fit(X_tr, y_tr, eval_X=X_val, eval_y=y_val, eval_metric="auc",
          callbacks=[lgb.early_stopping(50, verbose=False)])

print(f"best iteration: {model.best_iteration_} of 2000")
print(f"test AUC: {roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]):.4f}")

$ python lgb_early.py
best iteration: 428 of 2000
test AUC: 0.9941

LightGBM stopped at round 428 with a test AUC in the same range as XGBoost. The callback monitors the metric named in eval_metric on the eval_X/eval_y data, and best_iteration_ (with the trailing underscore, like other fitted attributes) records where it stopped.

Three Boosters, One Comparison

All three follow the scikit-learn API, so one loop compares accuracy and training time.

import timeit
import lightgbm as lgb
import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=20_000, n_features=40, n_informative=20, random_state=0)
models = {
    "sklearn HGB": HistGradientBoostingClassifier(max_iter=300, random_state=0),
    "XGBoost": xgb.XGBClassifier(n_estimators=300, max_depth=6, random_state=0),
    "LightGBM": lgb.LGBMClassifier(n_estimators=300, random_state=0, verbose=-1),
}
for name, m in models.items():
    auc = cross_val_score(m, X, y, cv=3, scoring="roc_auc").mean()
    secs = min(timeit.repeat(lambda: m.fit(X, y), number=1, repeat=3))
    print(f"{name:12s} CV AUC {auc:.4f}   fit {secs:.2f}s (best of 3)")

$ python compare_boosters.py
sklearn HGB  CV AUC 0.9944   fit 5.86s (best of 3)
XGBoost      CV AUC 0.9943   fit 3.41s (best of 3)
LightGBM     CV AUC 0.9945   fit 2.25s (best of 3)

The three AUCs are identical to three decimal places, so on this data the choice is about speed and features, not accuracy. On this machine LightGBM trained fastest and scikit-learn’s booster slowest, but the ordering and ratios depend on the CPU, the thread count and the data shape; time them on your own workload before switching libraries for speed.

Native Categorical Features

When the target depends on a category, LightGBM can split on the category directly. Here default risk depends on which of 30 regions a customer lives in, and the models are limited to single-split stumps to make the difference visible.

import lightgbm as lgb
import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 5000
region = rng.integers(0, 30, n)
risky = np.isin(region, rng.choice(30, 10, replace=False))     # 10 risky regions
y = (rng.random(n) < np.where(risky, 0.6, 0.2)).astype(int)

X_code = pd.DataFrame({"region": region, "income": rng.normal(50, 10, n)})
X_cat = X_code.assign(region=X_code["region"].astype("category"))

for n_trees in [1, 50]:
    model = lgb.LGBMClassifier(n_estimators=n_trees, num_leaves=2, random_state=0, verbose=-1)
    for name, X in [("integer", X_code), ("category", X_cat)]:
        auc = cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
        print(f"{n_trees:2d} stump(s), region as {name:8s} CV AUC {auc:.4f}")

$ python lgb_categorical.py
 1 stump(s), region as integer  CV AUC 0.5218
 1 stump(s), region as category CV AUC 0.6997
50 stump(s), region as integer  CV AUC 0.6292
50 stump(s), region as category CV AUC 0.7025

With integer codes, one split can only separate regions below a threshold from those above, but the 10 risky regions are scattered across the range, so a single stump barely beats a coin flip (0.52). As a category, one split can send all 10 risky regions left and the rest right, reaching 0.70 immediately. With 50 stumps the integer model narrows the gap by stacking many threshold splits, but it still trails the one-split categorical model.

Practice

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10_000, n_features=30, n_informative=15, random_state=42)
X_train, _, y_train, _ = train_test_split(X, y, test_size=0.2, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=5,
                          early_stopping_rounds=50, eval_metric="auc", random_state=42)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
auc = model.evals_result()["validation_0"]["auc"]
for r in [0, 50, 200, model.best_iteration, len(auc) - 1]:
    print(f"round {r:4d}  val AUC {auc[r]:.4f}")

$ python p1_eval_history.py
round    0  val AUC 0.8603
round   50  val AUC 0.9643
round  200  val AUC 0.9848
round  738  val AUC 0.9891
round  788  val AUC 0.9890

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 5000
region = rng.integers(0, 30, n)
risky = np.isin(region, rng.choice(30, 10, replace=False))
y = (rng.random(n) < np.where(risky, 0.6, 0.2)).astype(int)
X = pd.DataFrame({"region": pd.Categorical(region), "income": rng.normal(50, 10, n)})

model = xgb.XGBClassifier(n_estimators=50, max_depth=2, enable_categorical=True,
                          tree_method="hist", random_state=0)
print(f"XGBoost native categorical CV AUC "
      f"{cross_val_score(model, X, y, cv=5, scoring='roc_auc').mean():.4f}")

$ python p2_xgb_categorical.py
XGBoost native categorical CV AUC 0.6951

import lightgbm as lgb
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
model = lgb.LGBMClassifier(n_estimators=100, random_state=0, verbose=-1)
model.fit(data.data, data.target)

for kind in ["split", "gain"]:
    imp = model.booster_.feature_importance(importance_type=kind)
    top = imp.argsort()[::-1][:3]
    print(f"{kind:5s}:", [str(data.feature_names[i]) for i in top])

$ python p3_gain.py
split: ['worst texture', 'worst concave points', 'mean concave points']
gain : ['worst perimeter', 'worst area', 'worst concave points']