Unsupervised Learning

Real-World Scenario

An e-commerce company has 500,000 customer records with 50 behavioral features — no labels, no segments. They run K-Means to find 6 natural customer groups. PCA reduces the 50 features to 10 components (explaining 85% of variance) before clustering, dramatically improving cluster quality. UMAP provides a 2D visualization that confirms the clusters make business sense.

Principal Component Analysis (PCA)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits
import numpy as np

# Digits dataset: 1797 samples × 64 features (8×8 pixel images)
X, y = load_digits(return_X_y=True)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Full PCA — find how many components explain 95% of variance
pca_full = PCA()
pca_full.fit(X_scaled)

# Cumulative explained variance
cumvar = np.cumsum(pca_full.explained_variance_ratio_)
n_95 = np.argmax(cumvar >= 0.95) + 1
n_99 = np.argmax(cumvar >= 0.99) + 1
print(f"Components for 95% variance: {n_95}  (from 64 features)")
print(f"Components for 99% variance: {n_99}")

# Reduce dimensionality
pca = PCA(n_components=n_95, random_state=42)
X_reduced = pca.fit_transform(X_scaled)
print(f"Reduced shape: {X_reduced.shape}")
print(f"Variance explained: {pca.explained_variance_ratio_.sum():.3f}")

# Does PCA as preprocessing help or hurt this classifier?
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

clf_raw = RandomForestClassifier(100, random_state=42, n_jobs=-1)
clf_pca = RandomForestClassifier(100, random_state=42, n_jobs=-1)

score_raw = cross_val_score(clf_raw, X_scaled, y, cv=5, scoring="accuracy").mean()
score_pca = cross_val_score(clf_pca, X_reduced, y, cv=5, scoring="accuracy").mean()
print(f"RF on 64 features: {score_raw:.4f}")
print(f"RF on {n_95} PCA components: {score_pca:.4f}")

$ python ul_pca.py
Components for 95% variance: 40  (from 64 features)
Components for 99% variance: 54
Reduced shape: (1797, 40)
Variance explained: 0.951
RF on 64 features: 0.9394
RF on 40 PCA components: 0.9160

PCA actually costs about 2 points of accuracy here, not gains them — a Random Forest already handles 64 raw pixel features fine, since it only ever looks at one feature at a time per split and doesn’t suffer from the curse of dimensionality the way KNN or SVM do. PCA’s real payoff on this dataset is compression (40 components instead of 64, useful for storage or a distance-based downstream model), not accuracy for a tree ensemble — the “PCA improves classifiers” claim depends heavily on which classifier you plug in after it.

UMAP for Visualization

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)

# UMAP: install with: pip install umap-learn
import umap

reducer = umap.UMAP(
    n_components=2,
    n_neighbors=15,      # local vs global structure trade-off — higher = more global
    min_dist=0.1,        # how tightly points cluster — lower = tighter
    metric="euclidean",
    random_state=42,
)
X_2d = reducer.fit_transform(X_scaled)
print(f"UMAP output: {X_2d.shape}")  # (1797, 2)

# Check separation quality: same-class points should be close
from sklearn.metrics import silhouette_score
sil = silhouette_score(X_2d, y)
print(f"Silhouette score in 2D UMAP space: {sil:.3f}")

# Quick text-based check of separation
for digit in range(10):
    mask  = y == digit
    center = X_2d[mask].mean(axis=0)
    print(f"Digit {digit}: center=({center[0]:.1f}, {center[1]:.1f})")

$ python ul_umap.py
UMAP output: (1797, 2)
Silhouette score in 2D UMAP space: 0.589
Digit 0: center=(-6.5, 6.3)
Digit 1: center=(13.7, 7.0)
Digit 2: center=(14.1, 2.4)
Digit 3: center=(8.9, 3.3)
Digit 4: center=(11.9, 15.9)
Digit 5: center=(7.6, 8.8)
Digit 6: center=(6.2, -4.4)
Digit 7: center=(16.1, 14.4)
Digit 8: center=(11.6, 6.1)
Digit 9: center=(8.0, 6.4)

A silhouette score of 0.589 in just 2 dimensions is strong for 10 overlapping digit classes — most clustering algorithms consider anything above 0.5 well-separated. Digit 0 sits furthest from the rest (its center’s x-coordinate is negative while every other digit’s is positive), matching the fact that “0” is visually the most distinct digit shape. UMAP is stochastic even with random_state set across library versions, so exact coordinates can shift slightly on other machines — the cluster separation is what matters, not the specific numbers.

Anomaly Detection

from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler
import numpy as np

rng = np.random.default_rng(42)

# Simulated dataset: server metrics with anomalies
n_normal  = 1000
n_anomaly = 50

normal = rng.multivariate_normal([50, 70], [[100, 30], [30, 100]], n_normal)
anomaly = rng.uniform([0, 0], [200, 200], (n_anomaly, 2))

X = np.vstack([normal, anomaly])
y_true = np.array([1] * n_normal + [-1] * n_anomaly)  # 1=normal, -1=anomaly

scaler   = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Isolation Forest: randomly partitions data, anomalies need fewer splits
iso_forest = IsolationForest(
    contamination=0.05,  # expected fraction of anomalies
    n_estimators=100,
    random_state=42,
)
iso_pred = iso_forest.fit_predict(X_scaled)  # 1=normal, -1=anomaly

# Anomaly scores — more negative = more anomalous
iso_scores = iso_forest.score_samples(X_scaled)  # negative log-density

# Evaluate
from sklearn.metrics import classification_report, roc_auc_score
print("Isolation Forest:")
print(classification_report(y_true, iso_pred, target_names=["anomaly", "normal"]))

# LOF: density-based, compares point density to its neighbors
lof = LocalOutlierFactor(
    n_neighbors=20,
    contamination=0.05,
)
lof_pred = lof.fit_predict(X_scaled)

print("Local Outlier Factor:")
print(classification_report(y_true, lof_pred, target_names=["anomaly", "normal"]))

# Real-world usage: flag anomalies for human review
def flag_anomalies(X_new: np.ndarray, model, scaler, threshold: float = -0.5):
    X_s   = scaler.transform(X_new)
    preds = model.predict(X_s)
    return np.where(preds == -1)[0]

test_data = rng.uniform([0, 0], [200, 200], (5, 2))  # these should all be anomalies
flagged   = flag_anomalies(test_data, iso_forest, scaler)
print(f"Anomalies flagged: {flagged}")

$ python ul_anomaly.py
Isolation Forest:
              precision    recall  f1-score   support

     anomaly       0.83      0.88      0.85        50
      normal       0.99      0.99      0.99      1000

    accuracy                           0.99      1050
   macro avg       0.91      0.94      0.92      1050
weighted avg       0.99      0.99      0.99      1050

Local Outlier Factor:
              precision    recall  f1-score   support

     anomaly       0.83      0.88      0.85        50
      normal       0.99      0.99      0.99      1000

    accuracy                           0.99      1050
   macro avg       0.91      0.94      0.92      1050
weighted avg       0.99      0.99      0.99      1050

Anomalies flagged: [0 1 2 3 4]

Isolation Forest and LOF report identical precision/recall here, which is coincidence from how cleanly separated the two clusters are (a tight Gaussian blob vs. anomalies uniformly scattered across a much wider range) — on real, messier data the two methods typically disagree on which borderline points count as anomalies. Both flag all 5 of the held-out uniform-random points as anomalies, confirming flag_anomalies works end to end.

Topic Modeling with NMF

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF, LatentDirichletAllocation
import numpy as np

# Sample news headlines
documents = [
    "president signs climate change bill congress vote",
    "senate approves new immigration reform legislation",
    "election results show new party wins majority seats",
    "bitcoin ethereum cryptocurrency market price rally",
    "stock market dow jones record high earnings report",
    "interest rates inflation federal reserve monetary policy",
    "machine learning artificial intelligence deep learning model",
    "neural network transformer architecture language model training",
    "data science python pandas numpy scikit learn tutorial",
    "football championship game team score touchdown victory",
    "basketball playoffs game winner championship ring trophy",
    "soccer world cup final match penalty kick goal",
]

# TF-IDF vectorization
vectorizer = TfidfVectorizer(max_features=100, stop_words="english")
X_tfidf = vectorizer.fit_transform(documents)
feature_names = vectorizer.get_feature_names_out()

# NMF: decomposes into topics (non-negative matrix factorization)
N_TOPICS = 4
nmf = NMF(n_components=N_TOPICS, random_state=42, max_iter=200)
W = nmf.fit_transform(X_tfidf)   # document-topic matrix
H = nmf.components_               # topic-word matrix

def print_top_words(model, feature_names, n_top=8):
    for topic_idx, topic in enumerate(model.components_):
        top_words = [feature_names[i] for i in topic.argsort()[::-1][:n_top]]
        print(f"  Topic {topic_idx}: {', '.join(top_words)}")

print("NMF Topics:")
print_top_words(nmf, feature_names)

# Assign documents to dominant topic
dominant_topic = W.argmax(axis=1)
for doc, topic in zip(documents, dominant_topic):
    print(f"  [{topic}] {doc[:60]}")

$ python ul_nmf.py
NMF Topics:
  Topic 0: championship, game, team, victory, winner, ring, playoffs, basketball
  Topic 1: new, legislation, reform, senate, immigration, approves, party, results
  Topic 2: market, ethereum, price, rally, cryptocurrency, bitcoin, report, record
  Topic 3: learning, model, network, transformer, training, neural, language, architecture
  [3] president signs climate change bill congress vote
  [1] senate approves new immigration reform legislation
  [1] election results show new party wins majority seats
  [2] bitcoin ethereum cryptocurrency market price rally
  [2] stock market dow jones record high earnings report
  [3] interest rates inflation federal reserve monetary policy
  [3] machine learning artificial intelligence deep learning model
  [3] neural network transformer architecture language model train
  [3] data science python pandas numpy scikit learn tutorial
  [0] football championship game team score touchdown victory
  [0] basketball playoffs game winner championship ring trophy
  [3] soccer world cup final match penalty kick goal

The four topic word-lists read cleanly as sports / politics / finance / tech, but the document assignments expose the weak spot: “interest rates inflation federal reserve monetary policy” and the soccer headline both get dumped into topic 3 (tech) instead of finance and sports respectively. With only 12 short documents and 100 max TF-IDF features, there just isn’t enough shared vocabulary for NMF to place every document correctly — this is a toy-scale demonstration of the mechanism, not a production-quality topic model. Real topic modeling needs hundreds to thousands of documents before the topic-word and document-topic assignments both become reliable.

Practice

1. In the PCA block, print pca_full.explained_variance_ratio_[:5] — the variance share of just the first 5 components. What fraction of the 95%-total do they alone capture?

   Solution

   The first 5 components typically capture roughly 40-45% of the total variance on digits — most of the remaining ~50% needed to reach 95% comes from many components each contributing a small amount, which is why PCA needs 40 components here instead of just a handful.

2. Change n_neighbors in the UMAP block from 15 to 5, then to 50, and compare the resulting silhouette scores.

   Solution

   Smaller n_neighbors (5) tends to fragment classes into tighter local sub-clusters with a similar or slightly lower silhouette score; larger n_neighbors (50) smooths toward more global structure, which can also change (usually improve, up to a point) the between-class separation the silhouette score measures.

3. In the anomaly detection block, change contamination from 0.05 to the true value 50/1050 ≈ 0.0476 on both IsolationForest and LocalOutlierFactor. Does the classification report change much from the lesson’s 0.05?

   Solution

   Barely — 0.05 vs. 0.0476 asks each model to flag about the same number of points, so precision/recall for the anomaly class move by at most a percentage point or two on this dataset.

4. In the NMF block, add the two documents "tennis grand slam championship trophy final" and "treasury bonds yields federal reserve rate hike" to documents and re-run. Do they land in the topics you’d expect?

   Solution

   The tennis sentence should join topic 0 (sports) since it shares championship/trophy vocabulary with the existing football/basketball docs; the treasury sentence should pull toward topic 2 (finance) via federal reserve, which — unlike the lesson’s “interest rates” sentence — now has a federal reserve neighbor to anchor it, showing how more same-topic documents fix the earlier misclassification.

5. Swap NMF for LatentDirichletAllocation (already imported) with the same n_components=4 and compare its topic word-lists to NMF’s.

   Solution

   from sklearn.decomposition import LatentDirichletAllocation
   lda = LatentDirichletAllocation(n_components=4, random_state=42)
   W = lda.fit_transform(X_tfidf)
   print_top_words(lda, feature_names)

   LDA is a probabilistic model built for raw counts, not TF-IDF weights, so its topics on this same TF-IDF matrix tend to look less crisp than NMF’s — a good example of why the lesson picked NMF for this walkthrough.