Handling Imbalanced Data

Real-World Scenario

A fraud detection model trained on 1 million transactions where 0.1% are fraud. Without handling imbalance, the model learns to always predict “not fraud” and scores 99.9% accuracy. The real goal: catch 90% of fraud (recall) while keeping false positive rate under 1% (precision). This requires resampling, cost-sensitive training, and threshold tuning.

Diagnosing the Problem

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_auc_score, average_precision_score
)

# Create imbalanced dataset: 98% class 0, 2% class 1
X, y = make_classification(
    n_samples=10_000,
    n_features=20,
    n_informative=10,
    weights=[0.98, 0.02],   # 98/2 split
    random_state=42,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                      stratify=y, random_state=42)

print(f"Training positives: {y_train.sum()} / {len(y_train)}  ({y_train.mean():.1%})")
print(f"Test positives:     {y_test.sum()} / {len(y_test)}  ({y_test.mean():.1%})")

# Naive model — ignores imbalance
naive = LogisticRegression(max_iter=1000)
naive.fit(X_train, y_train)

print("\nNaive model (no imbalance handling):")
y_pred = naive.predict(X_test)
print(classification_report(y_test, y_pred, target_names=["normal", "fraud"]))
print(f"AUC-ROC:  {roc_auc_score(y_test, naive.predict_proba(X_test)[:,1]):.4f}")
print(f"Avg Prec: {average_precision_score(y_test, naive.predict_proba(X_test)[:,1]):.4f}")

$ python imbalanced_data.py
Training positives: 203 / 8000  (2.5%)
Test positives:     51 / 2000  (2.5%)

Naive model (no imbalance handling):
              precision    recall  f1-score   support

      normal       0.98      1.00      0.99      1949
       fraud       1.00      0.18      0.30        51

    accuracy                           0.98      2000
   macro avg       0.99      0.59      0.64      2000
weighted avg       0.98      0.98      0.97      2000

AUC-ROC:  0.7502
Avg Prec: 0.4116

98% overall accuracy hides that the model only catches 18% of actual fraud (recall=0.18 on the fraud row) — it plays it safe, only flagging fraud when it’s extremely confident (precision=1.00), which misses most real cases. AUC-ROC (0.7502) looks moderate but Average Precision (0.4116) is the more honest number here: it’s sensitive to the class imbalance in a way AUC-ROC isn’t, and 0.41 on a 0-1 scale where 0.025 (the base fraud rate) is the random baseline shows the model has real but limited separating power.

Class Weights

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.utils.class_weight import compute_class_weight
from sklearn.metrics import classification_report, roc_auc_score, average_precision_score
import numpy as np

# class_weight="balanced" automatically sets weight inversely proportional to frequency
lr_balanced = LogisticRegression(class_weight="balanced", max_iter=1000)
lr_balanced.fit(X_train, y_train)
y_pred = lr_balanced.predict(X_test)

print("Balanced logistic regression:")
print(classification_report(y_test, y_pred, target_names=["normal", "fraud"]))
print(f"AUC-ROC:  {roc_auc_score(y_test, lr_balanced.predict_proba(X_test)[:,1]):.4f}")
print(f"Avg Prec: {average_precision_score(y_test, lr_balanced.predict_proba(X_test)[:,1]):.4f}")

# Manual weight computation (same result, but you can tune it)
weights = compute_class_weight("balanced", classes=np.array([0, 1]), y=y_train)
class_weight_dict = {0: weights[0], 1: weights[1]}
print(f"\nAuto-computed weights: class 0 = {weights[0]:.2f}, class 1 = {weights[1]:.2f}")

# For HistGradientBoosting, use sample_weight
from sklearn.utils import class_weight as cw_module
sample_weights = np.where(y_train == 1, weights[1], weights[0])

hgb = HistGradientBoostingClassifier(max_iter=200, random_state=42)
hgb.fit(X_train, y_train, sample_weight=sample_weights)
print(f"\nHGB with sample weights — AUC: {roc_auc_score(y_test, hgb.predict_proba(X_test)[:,1]):.4f}")

$ python imbalanced_data.py   # continued
Balanced logistic regression:
              precision    recall  f1-score   support

      normal       0.99      0.79      0.88      1949
       fraud       0.07      0.63      0.13        51

    accuracy                           0.78      2000
   macro avg       0.53      0.71      0.50      2000
weighted avg       0.96      0.78      0.86      2000

AUC-ROC:  0.7668
Avg Prec: 0.3301

Auto-computed weights: class 0 = 0.51, class 1 = 19.70

HGB with sample weights — AUC: 0.7649

class_weight="balanced" does exactly what it promises — fraud recall jumps from 0.18 to 0.63 — but at a steep cost most people don’t expect: fraud precision collapses from 1.00 to 0.07, so now 93% of the model’s fraud flags are false alarms, and Average Precision actually drops (0.4116 → 0.3301) even though AUC-ROC ticks up slightly. Class weights move where the model draws its decision boundary; they don’t improve the model’s underlying ability to rank fraud above normal transactions, which is what AP and AUC-ROC measure. Whether 0.07 precision is acceptable depends entirely on the cost of a false alarm versus a missed fraud — a decision only the threshold-tuning and business-cost sections below can actually make.

Resampling with imbalanced-learn

# pip install imbalanced-learn
from imblearn.over_sampling import SMOTE, ADASYN
from imblearn.under_sampling import RandomUnderSampler, TomekLinks
from imblearn.combine import SMOTETomek
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score

# SMOTE: Synthetic Minority Over-sampling Technique
# Creates synthetic minority samples by interpolating between existing ones
smote = SMOTE(sampling_strategy=0.1, random_state=42)  # minority:majority = 1:10
X_res, y_res = smote.fit_resample(X_train, y_train)
print(f"After SMOTE: {y_res.sum()} positives / {len(y_res)} total ({y_res.mean():.1%})")

# Train on resampled data
lr = LogisticRegression(max_iter=1000)
lr.fit(X_res, y_res)
print(f"SMOTE + LR — AUC: {roc_auc_score(y_test, lr.predict_proba(X_test)[:,1]):.4f}")

# Combine oversampling + Tomek Links cleaning (removes borderline samples)
smt = SMOTETomek(sampling_strategy=0.1, random_state=42)
X_comb, y_comb = smt.fit_resample(X_train, y_train)
lr.fit(X_comb, y_comb)
print(f"SMOTETomek + LR — AUC: {roc_auc_score(y_test, lr.predict_proba(X_test)[:,1]):.4f}")

# Best practice: resampling inside pipeline to prevent leakage
imb_pipeline = ImbPipeline([
    ("smote",  SMOTE(sampling_strategy=0.1, random_state=42)),
    ("scaler", StandardScaler()),
    ("model",  LogisticRegression(max_iter=1000)),
])
imb_pipeline.fit(X_train, y_train)
print(f"Imb-Pipeline AUC: {roc_auc_score(y_test, imb_pipeline.predict_proba(X_test)[:,1]):.4f}")

$ python imbalanced_data.py   # continued
After SMOTE: 779 positives / 8576 total (9.1%)
SMOTE + LR — AUC: 0.7589
SMOTETomek + LR — AUC: 0.7589
Imb-Pipeline AUC: 0.7591

sampling_strategy=0.1 means “synthesize minority samples until they’re 10% of the majority count,” which is why 779 positives (not exactly 10% of 8576, since the majority count also includes the synthesized-against original 7797 negatives) land at 9.1% overall. SMOTE alone and SMOTETomek (SMOTE plus removing ambiguous borderline pairs) score identically here to four decimal places — on this synthetic, well-behaved dataset there isn’t enough class overlap for Tomek Links’ cleanup step to change which points end up near the decision boundary. All three resampling variants land within 0.002 AUC of the class-weighted HGB from the previous section (0.7649) — resampling and class weighting are solving the same underlying problem here, and neither beats the other by much on this dataset.

Threshold Tuning

import numpy as np
from sklearn.metrics import (
    precision_recall_curve, f1_score, precision_score, recall_score
)
import matplotlib  # noqa — for documentation; use matplotlib.pyplot to plot

# Get predicted probabilities (don't use default 0.5 threshold)
y_probs = hgb.predict_proba(X_test)[:, 1]

# Find the threshold that maximizes F1 score
precisions, recalls, thresholds = precision_recall_curve(y_test, y_probs)

f1_scores = 2 * precisions[:-1] * recalls[:-1] / (precisions[:-1] + recalls[:-1] + 1e-9)
best_idx   = f1_scores.argmax()
best_threshold = thresholds[best_idx]

print(f"Default threshold (0.5):")
y_pred_default = (y_probs >= 0.5).astype(int)
print(f"  Precision: {precision_score(y_test, y_pred_default):.4f}")
print(f"  Recall:    {recall_score(y_test, y_pred_default):.4f}")
print(f"  F1:        {f1_score(y_test, y_pred_default):.4f}")

print(f"\nOptimal threshold ({best_threshold:.3f}):")
y_pred_optimal = (y_probs >= best_threshold).astype(int)
print(f"  Precision: {precision_score(y_test, y_pred_optimal):.4f}")
print(f"  Recall:    {recall_score(y_test, y_pred_optimal):.4f}")
print(f"  F1:        {f1_score(y_test, y_pred_optimal):.4f}")

# Business-driven threshold: require recall >= 0.90
recall_90_mask = recalls[:-1] >= 0.90
if recall_90_mask.any():
    # Among thresholds with recall >= 0.90, maximize precision
    best_prec_at_90recall = precisions[:-1][recall_90_mask].max()
    idx_90 = np.where(recall_90_mask & (precisions[:-1] == best_prec_at_90recall))[0][0]
    threshold_90 = thresholds[idx_90]
    y_pred_90 = (y_probs >= threshold_90).astype(int)
    print(f"\nThreshold for recall ≥ 90% ({threshold_90:.3f}):")
    print(f"  Precision: {precision_score(y_test, y_pred_90):.4f}")
    print(f"  Recall:    {recall_score(y_test, y_pred_90):.4f}")

$ python imbalanced_data.py   # continued
Default threshold (0.5):
  Precision: 0.8636
  Recall:    0.3725
  F1:        0.5205

Optimal threshold (0.201):
  Precision: 0.8214
  Recall:    0.4510
  F1:        0.5823

Threshold for recall ≥ 90% (0.000):
  Precision: 0.0365
  Recall:    0.9020

Lowering the threshold from 0.5 to the F1-optimal 0.201 trades a little precision (0.8636 → 0.8214) for a real recall gain (0.3725 → 0.4510) — a reasonable trade. But the “require recall ≥ 90%” business threshold is a cautionary result, not a success: it lands at threshold 0.000, meaning this HGB model has to flag almost every transaction as fraud to catch 90% of real fraud, crashing precision to 0.0365 (96% of flags are false alarms). That’s not a usable operating point — it means this particular model’s fraud scores don’t separate the classes well enough to support a 90% recall target at any reasonable precision, which is a genuinely useful thing to discover before shipping a threshold like that to production, not a bug in the code.

Evaluation Metrics for Imbalance

from sklearn.metrics import (
    roc_auc_score, average_precision_score,
    balanced_accuracy_score, matthews_corrcoef,
    confusion_matrix,
)
import numpy as np

y_probs = hgb.predict_proba(X_test)[:, 1]
y_pred  = (y_probs >= best_threshold).astype(int)

tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()

print("Evaluation metrics for imbalanced classification:")
print(f"  AUC-ROC:            {roc_auc_score(y_test, y_probs):.4f}  (threshold-independent)")
print(f"  Avg Precision (AP): {average_precision_score(y_test, y_probs):.4f}  (area under PR curve)")
print(f"  Balanced Accuracy:  {balanced_accuracy_score(y_test, y_pred):.4f}  (mean recall per class)")
print(f"  MCC:                {matthews_corrcoef(y_test, y_pred):.4f}  (robust to imbalance)")
print(f"\nConfusion matrix:")
print(f"  True Positives:  {tp:4d}  (fraud caught)")
print(f"  False Positives: {fp:4d}  (false alarms)")
print(f"  False Negatives: {fn:4d}  (missed fraud)")
print(f"  True Negatives:  {tn:4d}  (correct clear)")
print(f"\n  Fraud catch rate (recall): {tp/(tp+fn):.1%}")
print(f"  False alarm rate:          {fp/(fp+tn):.1%}")

# Which metrics matter depends on the business cost of each error type
COST_FN = 500   # cost of missing one fraud
COST_FP = 10    # cost of false alarm (manual review)
business_cost = fn * COST_FN + fp * COST_FP
print(f"\nBusiness cost: ${business_cost:,.0f}  (FN=${fn*COST_FN:,.0f}, FP=${fp*COST_FP:,.0f})")

$ python imbalanced_data.py   # continued
Evaluation metrics for imbalanced classification:
  AUC-ROC:            0.7649  (threshold-independent)
  Avg Precision (AP): 0.4597  (area under PR curve)
  Balanced Accuracy:  0.7242  (mean recall per class)
  MCC:                0.6016  (robust to imbalance)

Confusion matrix:
  True Positives:    23  (fraud caught)
  False Positives:    5  (false alarms)
  False Negatives:   28  (missed fraud)
  True Negatives:  1944  (correct clear)

  Fraud catch rate (recall): 45.1%
  False alarm rate:          0.3%

Business cost: $14,050  (FN=$14,000, FP=$50)

This block reuses the F1-optimal threshold (0.201) from the previous section, not the degenerate 90%-recall one — which is why precision and recall here (catching 45.1% of fraud with a 0.3% false-alarm rate) look far more usable than the earlier “recall ≥ 90%” attempt. The business-cost framing makes the real trade-off concrete: with a missed fraud costing 50x a false alarm ($500 vs $10), the 28 missed frauds ($14,000) dwarf the 5 false alarms ($50) in this run’s total cost — which argues for pushing the threshold lower than 0.201 despite the F1-optimal point sitting there, since F1 weights precision and recall equally while this business’s actual costs don’t.

Practice

1. Change weights=[0.98, 0.02] in the diagnosing block to weights=[0.999, 0.001] (0.1% fraud rate, matching the lesson’s opening scenario) and re-run. How much worse does the naive model’s fraud recall get?

   Solution

   With only ~8 fraud cases in a 2000-row test set, recall becomes far noisier and often drops toward 0 — the naive model has even less positive-class signal to learn from, and a handful of missed cases now swings recall by 10+ percentage points at a time.

2. In the class weights block, replace class_weight="balanced" with a manual dict like {0: 1, 1: 5} (a gentler weight than the auto-computed 19.70) and compare precision/ recall to the lesson’s balanced result.

   Solution

   A weight of 5 instead of ~20 should land precision and recall somewhere between the naive model’s (1.00 precision / 0.18 recall) and the fully-balanced model’s (0.07 precision / 0.63 recall) — class weight strength is a dial, not a binary switch.

3. In the resampling block, change SMOTE’s sampling_strategy from 0.1 to 0.5 (minority reaches half the majority count) and see how SMOTE + LR AUC changes.

   Solution

   More synthetic minority samples usually shifts the decision boundary further toward catching positives, similar in spirit to increasing the class weight — AUC (a ranking metric) typically moves less than precision/recall at a fixed threshold would.

4. In the threshold tuning block, find the threshold that maximizes precision subject to recall ≥ 0.5 (adapt the existing recall_90_mask pattern with a 0.5 cutoff instead of 0.90).

   Solution

   recall_50_mask = recalls[:-1] >= 0.50
   best_prec_at_50recall = precisions[:-1][recall_50_mask].max()
   idx_50 = np.where(recall_50_mask & (precisions[:-1] == best_prec_at_50recall))[0][0]
   threshold_50 = thresholds[idx_50]

   Since 0.5 recall is much closer to what this model can realistically hit (the F1-optimal point already reaches 0.451), expect a threshold near 0.201 with precision still well above the degenerate 0.0365 seen at the 90%-recall target.

5. In the business-cost block, swap the costs to COST_FN = 50 and COST_FP = 100 (a false alarm now costs more than a miss — e.g., an expensive manual review process) and recompute. Does the F1-optimal threshold still look like the right choice?

   Solution

   With false alarms now the expensive error, the 5 false positives cost $500 while the 28 missed frauds cost only $1,400 combined — still worse to under-flag here, but the gap is much smaller, and pushing the threshold higher than 0.201 (trading recall for fewer false alarms) would lower total cost, the opposite direction from the lesson’s original cost assumptions.