Feature Engineering Pipelines

Real-World Scenario

A credit scoring model uses customer financial data: income (numerical), employment type (categorical with nulls), account age (date), and transaction history (time series). Raw features fed directly to a model cause errors and underperform. A proper feature engineering pipeline handles missing values, encodes categoricals, creates interaction features, and runs safely inside cross-validation.

The Leakage Problem

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

# ── WRONG: fit scaler on all data BEFORE cross-validation ─────────────
scaler       = StandardScaler()
X_scaled_all = scaler.fit_transform(X)   # uses test fold info — LEAKAGE

bad_scores = cross_val_score(
    LogisticRegression(), X_scaled_all, y, cv=5, scoring="roc_auc"
)
print(f"Leaky CV AUC:  {bad_scores.mean():.4f}  (overly optimistic)")

# ── CORRECT: scaler inside the pipeline ────────────────────────────────
correct_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model",  LogisticRegression()),
])

good_scores = cross_val_score(correct_pipe, X, y, cv=5, scoring="roc_auc")
print(f"Correct CV AUC: {good_scores.mean():.4f}  (honest estimate)")

$ python fe_leakage.py
Leaky CV AUC:  0.9286  (overly optimistic)
Correct CV AUC: 0.9286  (honest estimate)

These two numbers come out identical here — not a mistake, but a property of this specific setup. make_classification draws every sample i.i.d. from the same distribution, so a StandardScaler fit on all 1000 rows learns almost exactly the same per-feature mean/std as one fit on any 800-row training fold, and roc_auc only cares about score ranking, which barely moves under a near-identical affine rescaling. The leakage is real either way — you’re still technically letting the test fold influence the transform — but it’s invisible here because nothing about the data distribution differs between train and test. Leakage shows up as a visible score gap when the preprocessing step is more sensitive (target encoding, feature selection by correlation with y) or when train and test genuinely differ (time-based splits, covariate shift). The lesson’s structural point stands regardless: put the scaler in the pipeline so cross_val_score can’t leak by construction, whether or not a given dataset happens to expose the mistake.

ColumnTransformer for Mixed Data Types

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split, cross_val_score

# Realistic mixed-type dataset
rng = np.random.default_rng(42)
n   = 2000

df = pd.DataFrame({
    "income":          rng.lognormal(10, 1, n),           # continuous, right-skewed
    "age":             rng.integers(18, 75, n),            # integer
    "credit_score":    rng.normal(680, 80, n).clip(300, 850),
    "debt_ratio":      rng.beta(2, 5, n),
    "employment_type": rng.choice(
        ["salaried", "self-employed", "unemployed", None],
        n, p=[0.6, 0.25, 0.1, 0.05]
    ),
    "loan_purpose":    rng.choice(
        ["home", "auto", "education", "personal"], n
    ),
    "credit_history":  rng.choice(
        ["excellent", "good", "fair", "poor"], n,
        p=[0.2, 0.4, 0.3, 0.1]
    ),
})
# Inject some nulls in numerical columns
df.loc[rng.choice(n, 100), "income"]       = np.nan
df.loc[rng.choice(n, 80),  "credit_score"] = np.nan

# Target: default (1) or not (0)
df["default"] = (
    (df["debt_ratio"] > 0.5) |
    (df["credit_history"].isin(["poor", "fair"])) |
    (df["employment_type"] == "unemployed")
).astype(int)

X = df.drop("default", axis=1)
y = df["default"]

# Column groups by transformation needed
num_features = ["income", "age", "credit_score", "debt_ratio"]
cat_ohe      = ["employment_type", "loan_purpose"]   # low cardinality → one-hot
cat_ord      = ["credit_history"]                     # ordinal → ordered integers

# Preprocessing pipelines per column type
num_pipe = Pipeline([
    ("imputer", KNNImputer(n_neighbors=5)),    # KNN imputation for numerical
    ("scaler",  StandardScaler()),
])

cat_ohe_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

cat_ord_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OrdinalEncoder(
        categories=[["poor", "fair", "good", "excellent"]],
        handle_unknown="use_encoded_value", unknown_value=-1,
    )),
])

preprocessor = ColumnTransformer([
    ("numerical",   num_pipe,       num_features),
    ("categorical", cat_ohe_pipe,   cat_ohe),
    ("ordinal",     cat_ord_pipe,   cat_ord),
], remainder="drop")

full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model",      HistGradientBoostingClassifier(max_iter=200, random_state=42)),
])

# Cross-validate the entire pipeline (no leakage)
scores = cross_val_score(full_pipeline, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
print(f"CV AUC: {scores.mean():.4f} ± {scores.std():.4f}")

$ python fe_columntransformer.py
CV AUC: 1.0000 ± 0.0000

A perfect 1.0000 AUC is a red flag, not a win — look at how default was built: it’s a deterministic rule over debt_ratio, credit_history, and employment_type, three columns that are handed straight to the model as features. HistGradientBoostingClassifier just learns the exact threshold rule instead of a real pattern. In a real dataset a score this clean almost always means the target is leaking through a feature (or, as here, the target literally is a function of the features) — the fix in production would be checking whether any input feature is a proxy for or component of the label itself, not celebrating the score.

Custom Feature Engineering Transformers

import pandas as pd
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline

class DateFeatureExtractor(BaseEstimator, TransformerMixin):
    """Extract temporal features from a date column."""

    def __init__(self, date_col: str, drop_original: bool = True):
        self.date_col      = date_col
        self.drop_original = drop_original

    def fit(self, X: pd.DataFrame, y=None):
        return self   # stateless transformer

    def transform(self, X: pd.DataFrame) -> pd.DataFrame:
        X = X.copy()
        dates = pd.to_datetime(X[self.date_col], errors="coerce")
        X[f"{self.date_col}_year"]         = dates.dt.year
        X[f"{self.date_col}_month"]        = dates.dt.month
        X[f"{self.date_col}_dayofweek"]    = dates.dt.dayofweek   # 0=Mon, 6=Sun
        X[f"{self.date_col}_is_weekend"]   = (dates.dt.dayofweek >= 5).astype(int)
        X[f"{self.date_col}_quarter"]      = dates.dt.quarter
        X[f"{self.date_col}_days_since"]   = (
            pd.Timestamp.now() - dates
        ).dt.days.fillna(-1).astype(int)

        if self.drop_original:
            X = X.drop(columns=[self.date_col])
        return X


class InteractionFeatures(BaseEstimator, TransformerMixin):
    """Create pairwise ratio and product features."""

    def __init__(self, pairs: list[tuple[str, str]], operations: list[str] = ("ratio", "product")):
        self.pairs      = pairs
        self.operations = operations

    def fit(self, X, y=None):
        return self

    def transform(self, X: pd.DataFrame) -> pd.DataFrame:
        X = X.copy()
        for col_a, col_b in self.pairs:
            if "ratio" in self.operations:
                X[f"{col_a}_div_{col_b}"] = X[col_a] / (X[col_b].replace(0, np.nan)).fillna(0)
            if "product" in self.operations:
                X[f"{col_a}_x_{col_b}"] = X[col_a] * X[col_b]
        return X


class TargetEncoder(BaseEstimator, TransformerMixin):
    """Replace categorical values with their target mean — fit-only on training data."""

    def __init__(self, cols: list[str], smoothing: float = 10.0):
        self.cols      = cols
        self.smoothing = smoothing
        self._maps     = {}
        self._global_mean = 0.0

    def fit(self, X: pd.DataFrame, y: pd.Series) -> "TargetEncoder":
        y = np.asarray(y)
        self._global_mean = float(y.mean())
        for col in self.cols:
            stats = pd.DataFrame({"y": y, "x": X[col].values})
            agg   = stats.groupby("x")["y"].agg(["mean", "count"])
            # Smoothed target encoding: blend category mean with global mean
            agg["smoothed"] = (
                agg["mean"] * agg["count"] + self._global_mean * self.smoothing
            ) / (agg["count"] + self.smoothing)
            self._maps[col] = agg["smoothed"].to_dict()
        return self

    def transform(self, X: pd.DataFrame) -> pd.DataFrame:
        X = X.copy()
        for col in self.cols:
            X[f"{col}_te"] = X[col].map(self._maps[col]).fillna(self._global_mean)
        return X.drop(columns=self.cols)


# Build a pipeline with custom transformers
rng = np.random.default_rng(42)
n   = 1000
df  = pd.DataFrame({
    "signup_date":  pd.date_range("2020-01-01", periods=n, freq="D").astype(str),
    "income":       rng.lognormal(10, 1, n),
    "loan_amount":  rng.lognormal(9, 1, n),
    "region":       rng.choice(["north", "south", "east", "west"], n),
})
y = (rng.uniform(0, 1, n) > 0.7).astype(int)

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# Note: custom transformers that need column names must work on DataFrames
pipeline = Pipeline([
    ("dates",        DateFeatureExtractor("signup_date")),
    ("interactions", InteractionFeatures([("income", "loan_amount")])),
    ("target_enc",   TargetEncoder(["region"])),
    ("num_prep",     ColumnTransformer([
        ("scale", StandardScaler(), slice(None))   # scale all remaining columns
    ], remainder="passthrough")),
    ("model", LogisticRegression(max_iter=1000)),
])

# Must use cross_val_score with a DataFrame to preserve column names
from sklearn.model_selection import StratifiedKFold
cv     = StratifiedKFold(5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, df, y, cv=cv, scoring="roc_auc")
print(f"CV AUC: {scores.mean():.4f} ± {scores.std():.4f}")

$ python fe_custom.py
CV AUC: 0.5173 ± 0.0377

This example needs two fixes to run at all against a real pipeline, both shown above. First, TargetEncoder.fit originally called y.values, which crashes with AttributeError: 'numpy.ndarray' object has no attribute 'values' — cross_val_score passes y through as whatever type you gave it (here, the plain ndarray from (rng.uniform(...) > 0.7).astype(int)), not always a pandas.Series; wrapping it in np.asarray(y) first makes the transformer work with either. Second, TargetEncoder originally left the raw region string column in the frame alongside its new region_te column, so the final StandardScaler step (slice(None), meaning “scale every remaining column”) tried to scale text and raised ValueError: could not convert string to float: 'north' — dropping the original column after encoding, matching the pattern DateFeatureExtractor already uses, fixes it. Once it runs, the AUC lands right at 0.52, barely above a coin flip — expected, since y here is pure rng.uniform(...) noise with no real relationship to any feature. This block exists to prove the pipeline plumbing works end to end, not to show a good score.

Practice

1. In the leakage block, replace LogisticRegression with KNeighborsClassifier (which is distance-based and more sensitive to feature scale) and re-check whether leaky vs. correct CV AUC still match exactly.

   Solution

   KNN’s predictions depend directly on distances between points, which a slightly different per-fold vs. global scaling can shift more than logistic regression’s linear decision boundary does — expect a small but potentially nonzero gap here, unlike the identical scores in the lesson’s example.

2. In the ColumnTransformer block, change default’s definition to something that isn’t a deterministic function of the input features — e.g. (rng.uniform(0, 1, n) < 0.3).astype(int) — and re-run. How does CV AUC change from 1.0000?

   Solution

   AUC should drop to roughly 0.5, since a random target has no real relationship to any feature the pipeline builds — this is a good sanity check to run whenever a model scores suspiciously well.

3. In the custom transformers block, remove the .drop(columns=self.cols) fix from TargetEncoder.transform and instead fix the StandardScaler step to only scale numeric columns by replacing slice(None) with an explicit column list. Which fix do you prefer, and why?

   Solution

   Explicitly listing numeric columns is more robust to future pipeline changes (adding a new categorical column won’t silently break scaling), while dropping the raw column in the transformer keeps each transformer self-contained — either fixes this specific bug, but explicit column lists in the ColumnTransformer scale better as pipelines grow.

4. Add a smoothing=0.0 TargetEncoder variant and compare its region weights to smoothing=10.0’s. Which categories’ encodings change the most?

   Solution

   Categories with fewer rows shift the most — with smoothing=0.0 a rare category’s encoding is just its raw (possibly noisy) mean, while smoothing=10.0 pulls it toward the global mean; a common category’s encoding barely changes either way since its count already dominates the smoothing term.

5. In InteractionFeatures, the ratio calculation replaces zero denominators with np.nan then fills with 0. What happens to income_div_loan_amount for a row where loan_amount is legitimately 0, versus what a domain expert might actually want?

   Solution

   That row gets a ratio of exactly 0, indistinguishable from “income is 0” — a domain expert reviewing loan data would likely want a sentinel value (e.g. -1 or a separate is_zero_denominator flag) instead, since silently mapping “undefined” to “0” can mislead a downstream model into treating it as a genuine low ratio.