Scikit-Learn Feature Engineering

Raw columns rarely suit a model as-is: features sit on wildly different scales, categories are strings, and values go missing. Every example here ran with scikit-learn 1.9.1 and pandas 3.0.

Why Scaling Matters

k-nearest neighbours measures distance, so a feature measured in large units drowns out the rest. The wine dataset mixes features in the thousands with features below 1.

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
print("largest column max:", X.max(axis=0).max(), " smallest column max:", X.max(axis=0).min())

raw = KNeighborsClassifier()
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier())
print(f"KNN raw features:    {cross_val_score(raw, X, y, cv=5).mean():.3f}")
print(f"KNN scaled features: {cross_val_score(scaled, X, y, cv=5).mean():.3f}")

$ python why_scale.py
largest column max: 1680.0  smallest column max: 0.66
KNN raw features:    0.691
KNN scaled features: 0.949

One feature (proline) reaches 1680 while another tops out below 1, so unscaled distances are effectively proline alone. Standardising gives each feature an equal say, and accuracy jumps. The scaler sits inside a pipeline so each cross-validation fold learns its own mean and standard deviation from training rows only.

Choosing a Scaler

Each scaler treats an outlier differently. This column has four ordinary values and one extreme one.

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

x = np.array([[10.0], [12.0], [11.0], [13.0], [500.0]])   # one outlier

for scaler in [StandardScaler(), MinMaxScaler(), RobustScaler()]:
    out = scaler.fit_transform(x).ravel().round(2)
    print(f"{type(scaler).__name__:15s} {out}")

$ python scalers.py
StandardScaler  [-0.51 -0.5  -0.5  -0.49  2.  ]
MinMaxScaler    [0.   0.   0.   0.01 1.  ]
RobustScaler    [ -1.    0.   -0.5   0.5 244. ]

MinMaxScaler squeezes the four normal values into 0.00-0.01 because the outlier defines the top of the range, and StandardScaler squeezes them almost as badly. RobustScaler centres on the median and divides by the interquartile range, so the ordinary values keep their spread (-1 to 0.5) and only the outlier lands far away.

Encoding Categories

OneHotEncoder makes one column per category. With handle_unknown="ignore", a category never seen during fit becomes all zeros instead of an error.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

train = pd.DataFrame({"color": ["red", "blue", "green", "red"]})
test = pd.DataFrame({"color": ["blue", "purple"]})        # purple is new

ohe = OneHotEncoder(sparse_output=False, handle_unknown="ignore").fit(train)
print(ohe.get_feature_names_out())
print(ohe.transform(test))

$ python onehot.py
['color_blue' 'color_green' 'color_red']
[[1. 0. 0.]
 [0. 0. 0.]]

Columns are sorted alphabetically, so blue comes first. The unseen purple row is all zeros: the model sees “none of the known colours” rather than crashing in production.

Ordinal encoding implies an order, and the default order is alphabetical.

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder

sizes = pd.DataFrame({"size": ["S", "M", "L", "XL", "M"]})

default = OrdinalEncoder().fit(sizes)
print("default categories:", default.categories_[0].tolist())
print("default codes:     ", default.transform(sizes).ravel())

ordered = OrdinalEncoder(categories=[["S", "M", "L", "XL"]]).fit(sizes)
print("explicit codes:    ", ordered.transform(sizes).ravel())

$ python ordinal_trap.py
default categories: ['L', 'M', 'S', 'XL']
default codes:      [2. 1. 0. 3. 1.]
explicit codes:     [0. 1. 2. 3. 1.]

Alphabetically, L < M < S < XL, so the default encoding tells the model that small is bigger than large. Nothing warns you; the codes are simply wrong. Passing categories= gives the intended S=0, M=1, L=2, XL=3.

Handling Missing Values

SimpleImputer fills gaps with a statistic learned from the training data. add_indicator=True also records where the gaps were.

import numpy as np
from sklearn.impute import SimpleImputer

X = np.array([[1.0, 20.0],
              [np.nan, 30.0],
              [3.0, np.nan],
              [5.0, 50.0]])

imp = SimpleImputer(strategy="median", add_indicator=True)
print(imp.fit_transform(X))
print(imp.get_feature_names_out(["age", "income"]))

$ python impute.py
[[ 1. 20.  0.  0.]
 [ 3. 30.  1.  0.]
 [ 3. 30.  0.  1.]
 [ 5. 50.  0.  0.]]
['age' 'income' 'missingindicator_age' 'missingindicator_income']

The median of 1, 3 and 5 is 3, and the median of 20, 30 and 50 is 30, so those fill the gaps. The two extra columns flag the imputed cells. Missingness is often informative, such as a customer who declined to give their income, and the indicator keeps that signal.

Creating Features

PolynomialFeatures adds squares and interaction terms so a linear model can fit curves.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures

X = np.array([[2.0, 3.0]])
poly = PolynomialFeatures(degree=2, include_bias=False)
print(poly.fit_transform(X))
print(poly.get_feature_names_out(["x1", "x2"]))

$ python poly.py
[[2. 3. 4. 6. 9.]]
['x1' 'x2' 'x1^2' 'x1 x2' 'x2^2']

From [2, 3] it builds x1, x2, x1², x1·x2, x2². The column count grows quickly: 10 input features at degree 2 become 65, so pair it with regularisation.

Dates hide several useful features. pandas extracts them in one line each.

import pandas as pd

df = pd.DataFrame({"order_date": pd.to_datetime(
    ["2024-03-01", "2024-03-02", "2024-06-15", "2024-12-25"])})

df["day_of_week"] = df["order_date"].dt.dayofweek      # Monday=0
df["is_weekend"] = (df["day_of_week"] >= 5).astype(int)
df["month"] = df["order_date"].dt.month
df["quarter"] = df["order_date"].dt.quarter
print(df)

$ python date_features.py
  order_date  day_of_week  is_weekend  month  quarter
0 2024-03-01            4           0      3        1
1 2024-03-02            5           1      3        1
2 2024-06-15            5           1      6        2
3 2024-12-25            2           0     12        4

A raw timestamp is one number a model can’t interpret, but weekday, weekend and season often drive behaviour such as order volume.

Selecting Features

SelectKBest scores each feature on its own; SelectFromModel keeps the features a model relies on.

from sklearn.datasets import load_diabetes
from sklearn.feature_selection import SelectKBest, f_regression, SelectFromModel
from sklearn.linear_model import Lasso

data = load_diabetes()
X, y, names = data.data, data.target, data.feature_names

kbest = SelectKBest(f_regression, k=4).fit(X, y)
print("SelectKBest keeps:    ", kbest.get_feature_names_out(names).tolist())

lasso = SelectFromModel(Lasso(alpha=1.0)).fit(X, y)
print("Lasso keeps:          ", lasso.get_feature_names_out(names).tolist())
print("Lasso coefficients:   ", lasso.estimator_.coef_.round(1).tolist())

$ python select_features.py
SelectKBest keeps:     ['bmi', 'bp', 's4', 's5']
Lasso keeps:           ['bmi', 'bp', 's5']
Lasso coefficients:    [0.0, -0.0, 367.7, 6.3, 0.0, 0.0, -0.0, 0.0, 307.6, 0.0]

The two methods agree on the strongest signals but not on everything. SelectKBest judges each feature in isolation, while Lasso fits all features together and zeroes out the ones that add nothing once the others are known.

Practice

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
raw = RandomForestClassifier(random_state=0)
scaled = make_pipeline(StandardScaler(), RandomForestClassifier(random_state=0))
print(f"raw    {cross_val_score(raw, X, y, cv=5).mean():.3f}")
print(f"scaled {cross_val_score(scaled, X, y, cv=5).mean():.3f}")

$ python p1_forest_scale.py
raw    0.983
scaled 0.983

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

train = pd.DataFrame({"color": ["red", "blue", "green"]})
test = pd.DataFrame({"color": ["purple"]})

ohe = OneHotEncoder(handle_unknown="error").fit(train)
try:
    ohe.transform(test)
except ValueError as e:
    print("ValueError:", e)

$ python p2_unknown_error.py
ValueError: Found unknown categories ['purple'] in column 0 during transform

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

df = pd.DataFrame({"city": ["Pune", "Delhi", np.nan, "Pune", np.nan]})
imp = SimpleImputer(strategy="most_frequent")
print(imp.fit_transform(df).ravel())

$ python p3_mode.py
['Pune' 'Delhi' 'Pune' 'Pune' 'Pune']

from sklearn.datasets import load_diabetes
from sklearn.preprocessing import PolynomialFeatures

X, _ = load_diabetes(return_X_y=True)
for degree in [2, 3]:
    n = PolynomialFeatures(degree=degree, include_bias=False).fit_transform(X).shape[1]
    print(f"degree {degree}: {X.shape[1]} -> {n} columns")

$ python p4_poly_count.py
degree 2: 10 -> 65 columns
degree 3: 10 -> 285 columns