Introduction to Scikit-Learn

Scikit-learn is the standard Python library for classical machine learning on tabular data: linear models, trees, ensembles, clustering, plus preprocessing and model selection. Every example here was run with scikit-learn 1.9.1 on Python 3.12.

The Estimator API

Every model follows one interface: fit(X, y) learns, predict(X) infers, and score(X, y) evaluates. This loop trains three unrelated algorithms with identical code.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

for model in [LogisticRegression(max_iter=1000),
              KNeighborsClassifier(),
              DecisionTreeClassifier(random_state=0)]:
    model.fit(X_train, y_train)
    print(f"{type(model).__name__:24s} test accuracy {model.score(X_test, y_test):.3f}")

$ python same_api.py
LogisticRegression       test accuracy 0.974
KNeighborsClassifier     test accuracy 0.974
DecisionTreeClassifier   test accuracy 0.974

Nothing about the loop knows which algorithm it holds. That shared contract is what makes pipelines, cross-validation and grid search work with any estimator. The three scores are identical here because each model misclassifies exactly one of the 38 test flowers; on a dataset this small and clean, very different algorithms often tie.

Your First Classifier

The iris dataset has 150 flowers, 4 measurements each, and 3 species to predict.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

iris = load_iris()
X, y = iris.data, iris.target
print("X shape:", X.shape, " classes:", iris.target_names.tolist())

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print("train:", X_train.shape, " test:", X_test.shape)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test),
                            target_names=iris.target_names))

$ python first_classifier.py
X shape: (150, 4)  classes: ['setosa', 'versicolor', 'virginica']
train: (120, 4)  test: (30, 4)
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        10
  versicolor       0.82      0.90      0.86        10
   virginica       0.89      0.80      0.84        10

    accuracy                           0.90        30
   macro avg       0.90      0.90      0.90        30
weighted avg       0.90      0.90      0.90        30

stratify=y keeps the 1:1:1 species ratio in both splits, which is why the test set has exactly 10 of each (the support column). Setosa is perfectly separable, so any errors fall between versicolor and virginica, whose measurements overlap.

Your First Regression Model

Regression predicts a number instead of a class. Here it is a California district’s median house value, in units of $100,000.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error, r2_score

data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42)

scaler = StandardScaler().fit(X_train)          # learn mean/std from train only
model = LinearRegression().fit(scaler.transform(X_train), y_train)
y_pred = model.predict(scaler.transform(X_test))

print(f"MAE: ${mean_absolute_error(y_test, y_pred) * 100_000:,.0f}")
print(f"R2:  {r2_score(y_test, y_pred):.4f}")
for name, coef in zip(data.feature_names, model.coef_):
    print(f"  {name:11s} {coef:+.3f}")

$ python first_regression.py
MAE: $53,320
R2:  0.5758
  MedInc      +0.854
  HouseAge    +0.123
  AveRooms    -0.294
  AveBedrms   +0.339
  Population  -0.002
  AveOccup    -0.041
  Latitude    -0.897
  Longitude   -0.870

An R² around 0.58 means a straight-line model explains a bit over half the variance, with an average miss of roughly $53,000. Because the features were standardised, the coefficients are comparable: MedInc (median income) dominates. The large opposite-signed Latitude and Longitude weights are location doing the work a linear model can’t express directly.

Never Score on Training Data

The most important rule in ML is that a model’s score on the rows it trained on proves nothing. Here the labels are pure coin flips, so no real pattern exists.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(0)
X = rng.standard_normal((1000, 10))
y = rng.integers(0, 2, 1000)          # random labels: nothing to learn

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)

print(f"score on training data: {tree.score(X_train, y_train):.3f}")
print(f"score on held-out data: {tree.score(X_test, y_test):.3f}")

$ python memorise.py
score on training data: 1.000
score on held-out data: 0.475

A perfect training score on random labels is memorisation: an unrestricted tree grows a leaf for every training row. The held-out score, near 0.5, is the honest one, since coin flips can’t be predicted.

Built-in Datasets

Scikit-learn ships small datasets inside the package and generates synthetic ones on demand.

from sklearn import datasets

for loader in [datasets.load_iris, datasets.load_digits,
               datasets.load_wine, datasets.load_breast_cancer]:
    X, y = loader(return_X_y=True)
    print(f"{loader.__name__:20s} X{X.shape}  classes={len(set(y))}")

X, y = datasets.make_classification(n_samples=1000, n_features=20,
                                    n_informative=10, n_classes=3, random_state=42)
print(f"{'make_classification':20s} X{X.shape}  classes={len(set(y))}")

X, y = datasets.make_regression(n_samples=500, n_features=5, noise=0.1, random_state=42)
print(f"{'make_regression':20s} X{X.shape}  y range {y.min():.1f}..{y.max():.1f}")

$ python datasets_tour.py
load_iris            X(150, 4)  classes=3
load_digits          X(1797, 64)  classes=10
load_wine            X(178, 13)  classes=3
load_breast_cancer   X(569, 30)  classes=2
make_classification  X(1000, 20)  classes=3
make_regression      X(500, 5)  y range -332.0..366.0

The load_* datasets need no network, which makes them ideal for experiments. The make_* generators let you control size, noise and difficulty, and random_state makes them repeat exactly.

Practice

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

for model in [GaussianNB(), SVC()]:
    model.fit(X_train, y_train)
    print(f"{type(model).__name__:10s} test accuracy {model.score(X_test, y_test):.3f}")

$ python p1_more_models.py
GaussianNB test accuracy 1.000
SVC        test accuracy 0.974

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(0)
X = rng.standard_normal((1000, 10))
y = rng.integers(0, 2, 1000)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print(f"train {tree.score(X_train, y_train):.3f}  test {tree.score(X_test, y_test):.3f}")

$ python p2_shallow.py
train 0.575  test 0.515

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
model = RandomForestClassifier(random_state=42).fit(iris.data, iris.target)

flower = np.array([5.9, 3.0, 5.1, 1.8]).reshape(1, -1)   # one sample, 4 features
pred = model.predict(flower)[0]
print(iris.target_names[pred], model.predict_proba(flower).round(2))

$ python p3_one_flower.py
virginica [[0.   0.02 0.98]]

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score

data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42)

model = RandomForestRegressor(n_estimators=50, random_state=42, n_jobs=-1)
y_pred = model.fit(X_train, y_train).predict(X_test)
print(f"MAE: ${mean_absolute_error(y_test, y_pred) * 100_000:,.0f}")
print(f"R2:  {r2_score(y_test, y_pred):.4f}")

$ python p4_forest_reg.py
MAE: $32,980
R2:  0.8040