What is the difference between supervised and unsupervised learning?

Supervised learning trains on labeled data — you have input features and known target labels, and the model learns to predict labels for new inputs. Unsupervised learning finds patterns in unlabeled data — clustering, dimensionality reduction, and anomaly detection don't require labels.

What is overfitting and how do I detect it?

Overfitting is when a model memorizes the training data instead of learning generalizable patterns. Detect it by comparing training accuracy vs validation accuracy — if training accuracy is much higher than validation accuracy, the model is overfitting. Fix it with regularization, more data, simpler models, or dropout.

Introduction to Machine Learning

What Is Machine Learning?

Machine learning is the practice of building systems that improve through experience. Instead of writing explicit rules — “if the email contains these 500 spam words, classify as spam” — you feed the system examples and it learns the rules itself.

This shifts the engineering problem from “write the rules” to “collect the right data and choose the right algorithm.” The data is doing the heavy lifting.

Three Types of Machine Learning

Supervised Learning

You have input features X and known labels y. The model learns a mapping f: X → y.

Classification — predict a discrete label: spam/not spam, cat/dog, fraud/legitimate
Regression — predict a continuous value: house price, stock return, patient readmission risk

Unsupervised Learning

You only have input features X — no labels. The model finds structure on its own.

Clustering — group similar data points: customer segments, document topics
Dimensionality Reduction — compress features while preserving information: PCA, t-SNE, UMAP
Anomaly Detection — identify unusual patterns: fraud, network intrusion

Reinforcement Learning

An agent takes actions in an environment and receives rewards. It learns to maximize cumulative reward: game playing (AlphaGo), robot control, recommendation systems.

The Bias-Variance Tradeoff

This is the most important concept in ML. Every model makes two types of errors:

Bias — systematic error from wrong assumptions. A linear model fitting non-linear data has high bias — it’s consistently wrong in the same way. High bias = underfitting.

Variance — sensitivity to noise in the training data. A deep decision tree memorizes training noise and performs poorly on new data. High variance = overfitting.

Error = Bias² + Variance + Irreducible Noise

Simple model (linear regression): High Bias, Low Variance
Complex model (deep tree):         Low Bias, High Variance
Goal: find the sweet spot          Moderate Bias, Moderate Variance

import numpy as np
import matplotlib
matplotlib.use('Agg')  # non-interactive backend
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(42)
n = 100
X = rng.uniform(0, 10, n).reshape(-1, 1)
y = np.sin(X.ravel()) + rng.normal(0, 0.3, n)   # true signal + noise

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

for degree in [1, 3, 10, 20]:
    model = Pipeline([
        ("poly",  PolynomialFeatures(degree=degree)),
        ("reg",   LinearRegression()),
    ])
    model.fit(X_train, y_train)

    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))

    print(f"Degree {degree:2d}: Train MSE={train_mse:.4f}  Test MSE={test_mse:.4f}  "
          f"{'⚠ OVERFIT' if test_mse > train_mse * 2 else '✓'}")
# Degree  1: Train MSE=0.1892  Test MSE=0.1910  ✓ (underfitting)
# Degree  3: Train MSE=0.0912  Test MSE=0.0945  ✓ (good fit)
# Degree 10: Train MSE=0.0814  Test MSE=0.1423  ✓ (slight overfit)
# Degree 20: Train MSE=0.0715  Test MSE=4.2301  ⚠ OVERFIT

The ML Workflow

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report

# 1. Load and understand the data
data = load_breast_cancer()
X, y = data.data, data.target

print(f"Dataset:    {X.shape[0]} samples, {X.shape[1]} features")
print(f"Target:     {data.target_names}")
print(f"Class dist: {dict(zip(*np.unique(y, return_counts=True)))}")

# 2. Split first — BEFORE any preprocessing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Preprocess — fit only on training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # fit + transform on train
X_test_scaled  = scaler.transform(X_test)       # transform only on test

# 4. Train
model = GradientBoostingClassifier(n_estimators=200, random_state=42)
model.fit(X_train_scaled, y_train)

# 5. Cross-validate on training data for unbiased performance estimate
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5, scoring="f1_macro")
print(f"\nCV F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

# 6. Final evaluation on the held-out test set — only do this ONCE
y_pred = model.predict(X_test_scaled)
print("\nTest Set Performance:")
print(classification_report(y_test, y_pred, target_names=data.target_names))

Choosing an Algorithm

Algorithm	When to use	Pros	Cons
Logistic Regression	Baseline, interpretability needed	Fast, interpretable	Linear boundaries only
Random Forest	General tabular classification/regression	Robust, few hyperparameters	Memory intensive
Gradient Boosting	Maximum accuracy on tabular data	State-of-the-art on tabular	Slow to train, more tuning
SVM	Small high-dimensional data (text)	Effective in high dims	Doesn’t scale to large n
K-Nearest Neighbors	Baselines, small data	Simple, no training	Slow inference
Neural Networks	Images, text, complex patterns	Flexible, scales	Needs lots of data, compute

Preventing Data Leakage

Data leakage is when information from the test set influences the model, giving artificially inflated metrics. It’s the most common mistake in ML.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = np.random.randn(1000, 20)
y = np.random.randint(0, 2, 1000)

# WRONG: Fit scaler on all data before splitting — test data leaks into training
scaler_wrong = StandardScaler()
X_scaled_all = scaler_wrong.fit_transform(X)  # uses test set statistics!
X_train_wrong, X_test_wrong = train_test_split(X_scaled_all, test_size=0.2)

# CORRECT: Split first, then fit only on train
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
scaler_correct = StandardScaler()
X_train_correct = scaler_correct.fit_transform(X_train)  # fit on train only
X_test_correct  = scaler_correct.transform(X_test)        # apply to test

Practice

The bias-variance loop above splits without a seed, so every run prints different numbers. Pin the split with random_state=42, add a ("scale", StandardScaler()) step before PolynomialFeatures, and sweep degrees 1, 3, 5, 7, 10 and 15. Which degree has the lowest test MSE?

Solution

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(42)
n = 100
X = rng.uniform(0, 10, n).reshape(-1, 1)
y = np.sin(X.ravel()) + rng.normal(0, 0.3, n)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for degree in [1, 3, 5, 7, 10, 15]:
    model = Pipeline([
        ("scale", StandardScaler()),
        ("poly",  PolynomialFeatures(degree=degree)),
        ("reg",   LinearRegression()),
    ])
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))
    print(f"Degree {degree:2d}: Train MSE={train_mse:.4f}  Test MSE={test_mse:.4f}")

$ python sweep.py
Degree  1: Train MSE=0.6415  Test MSE=0.4669
Degree  3: Train MSE=0.4453  Test MSE=0.3443
Degree  5: Train MSE=0.1085  Test MSE=0.0808
Degree  7: Train MSE=0.0845  Test MSE=0.0783
Degree 10: Train MSE=0.0777  Test MSE=0.0941
Degree 15: Train MSE=0.0729  Test MSE=0.1128

Train MSE falls at every step, but test MSE bottoms out at degree 7 and then climbs again. That U-shape is the tradeoff: degrees 1 and 3 underfit (high bias), and past 7 the extra flexibility fits noise (rising variance). Scaling matters here because raw x**15 for x near 10 is about 10^15, which makes the least-squares fit numerically unstable.

Overfitting gets much worse when data is scarce. Keep your task-1 script but change n = 100 to n = 30, which leaves 21 training points, and run degrees 1, 5, 7, 10 and 15. Bring back the lesson’s test_mse > train_mse * 2 flag.

Solution

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(42)
n = 30                                   # was 100: only 21 training points now
X = rng.uniform(0, 10, n).reshape(-1, 1)
y = np.sin(X.ravel()) + rng.normal(0, 0.3, n)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for degree in [1, 5, 7, 10, 15]:
    model = Pipeline([
        ("scale", StandardScaler()),
        ("poly",  PolynomialFeatures(degree=degree)),
        ("reg",   LinearRegression()),
    ])
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))
    print(f"Degree {degree:2d}: Train MSE={train_mse:.4f}  Test MSE={test_mse:.4f}  "
          f"{'OVERFIT' if test_mse > train_mse * 2 else 'ok'}")

$ python small_data.py
Degree  1: Train MSE=0.5745  Test MSE=0.5499  ok
Degree  5: Train MSE=0.0681  Test MSE=0.2042  OVERFIT
Degree  7: Train MSE=0.0519  Test MSE=0.7949  OVERFIT
Degree 10: Train MSE=0.0486  Test MSE=601.7502  OVERFIT
Degree 15: Train MSE=0.0433  Test MSE=1431406.6261  OVERFIT

Training error still looks excellent, but with 21 points a degree-10 polynomial has enough freedom to swing wildly between them, so test MSE explodes to 601. Degree 7 was the best model at n = 100 and is flagged here. Variance depends on how much data you have, not only on how complex the model is.

The algorithm table calls logistic regression the baseline. Add LogisticRegression(max_iter=1000) to the ML-workflow example and print CV macro-F1 and test macro-F1 for both models, one line each.

Solution

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled  = scaler.transform(X_test)

models = {
    "LogisticRegression": LogisticRegression(max_iter=1000),
    "GradientBoosting":   GradientBoostingClassifier(n_estimators=200, random_state=42),
}
for name, model in models.items():
    cv = cross_val_score(model, X_train_scaled, y_train, cv=5, scoring="f1_macro")
    model.fit(X_train_scaled, y_train)
    test_f1 = f1_score(y_test, model.predict(X_test_scaled), average="macro")
    print(f"{name:18s} CV F1={cv.mean():.4f} ± {cv.std():.4f}  Test F1={test_f1:.4f}")

$ python baseline.py
LogisticRegression CV F1=0.9787 ± 0.0139  Test F1=0.9812
GradientBoosting   CV F1=0.9623 ± 0.0141  Test F1=0.9521

The simple baseline wins on both CV and test. This dataset is close to linearly separable once it’s scaled, so 200 boosted trees add nothing but training time. That’s why you train the baseline first: it tells you whether the complex model is earning its cost.

The FAQ says to detect overfitting by comparing training and held-out accuracy. Print both for the gradient-boosting model at n_estimators 5, 50 and 200, using model.score.

Solution

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled  = scaler.transform(X_test)

for n_estimators in [5, 50, 200]:
    model = GradientBoostingClassifier(n_estimators=n_estimators, random_state=42)
    model.fit(X_train_scaled, y_train)
    train_acc = model.score(X_train_scaled, y_train)
    test_acc  = model.score(X_test_scaled, y_test)
    print(f"n_estimators={n_estimators:3d}  train acc={train_acc:.4f}  "
          f"test acc={test_acc:.4f}  gap={train_acc - test_acc:+.4f}")

$ python gap.py
n_estimators=  5  train acc=0.9758  test acc=0.9386  gap=+0.0372
n_estimators= 50  train acc=1.0000  test acc=0.9474  gap=+0.0526
n_estimators=200  train acc=1.0000  test acc=0.9561  gap=+0.0439

Training accuracy hits 100% by 50 trees, yet test accuracy keeps rising to 200. A perfect training score on its own doesn’t prove harmful overfitting. The held-out number is what decides, and here the gap stays around 4-5 points without growing.

The leakage example uses pure noise, so the two scalers end up almost identical. Make the test data different: seed the generator, add 3.0 to the last 200 rows, and split with shuffle=False so those rows become the test set. Then compare the two scalers.

Solution

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = rng.normal(0, 1, (1000, 20))
X[800:] += 3.0          # the last 200 rows come from a shifted distribution

# shuffle=False keeps row order, so the test set is exactly the shifted rows
X_train, X_test = train_test_split(X, test_size=0.2, shuffle=False)

scaler_wrong = StandardScaler().fit(X)          # sees the test rows
scaler_correct = StandardScaler().fit(X_train)  # train rows only

print(f"scaler mean, feature 0  wrong={scaler_wrong.mean_[0]:.3f}  correct={scaler_correct.mean_[0]:.3f}")
print(f"scaler std,  feature 0  wrong={scaler_wrong.scale_[0]:.3f}  correct={scaler_correct.scale_[0]:.3f}")
print(f"test mean after scaling wrong={scaler_wrong.transform(X_test).mean():.3f}  "
      f"correct={scaler_correct.transform(X_test).mean():.3f}")

$ python leak.py
scaler mean, feature 0  wrong=0.616  correct=0.014
scaler std,  feature 0  wrong=1.572  correct=1.011
test mean after scaling wrong=1.539  correct=3.011

The leaky scaler’s mean picked up a fifth of the shift (about 0.6), and its standard deviation widened from about 1.0 to 1.57. Together they make the shifted test rows look only 1.5 units off instead of 3. Leakage made new data look more like the training data than it really is, which is exactly the kind of difference your evaluation exists to catch.

What Is Machine Learning?

Three Types of Machine Learning

Supervised Learning

Unsupervised Learning

Reinforcement Learning

The Bias-Variance Tradeoff

The ML Workflow

Choosing an Algorithm

Preventing Data Leakage

Practice

Frequently Asked Questions