How do I explain the bias-variance tradeoff in an interview?

Bias is error from a model too simple to capture the pattern; variance is error from a model so flexible it fits noise. Total error is the sum plus irreducible noise, so there is a complexity that minimises it. Showing the U-shaped test-error curve is more convincing than the definition.

What is the difference between L1 and L2 regularisation?

L2 shrinks all coefficients smoothly towards zero and keeps them; L1 can drive coefficients exactly to zero, so it performs feature selection. L1 suits wide sparse problems where most features are irrelevant; L2 suits correlated features you want to keep but damp.

Why not just use the most flexible model available?

Because flexibility buys variance. With limited data, a model that can fit anything fits the noise in your sample, and test error rises even as training error falls. The measured gap between train and test is the diagnostic.

What does 'no free lunch' actually mean for interviews?

No single algorithm is best across all problems, so the right answer to 'which model would you use' is always conditional — on data size, feature types, interpretability requirements and latency budget. Naming one model with no conditions is the weak answer.

ML Fundamentals Questions | SkillByExample

Fundamentals questions are graded on whether you can explain why, not whether you can define what. Everything below is shown as a measurement, which is also how to answer it out loud.

”Explain the bias-variance tradeoff”

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(42)
def true_f(x): return np.sin(1.5 * x) + 0.3 * x

X = rng.uniform(0, 6, 120).reshape(-1, 1)
y = true_f(X.ravel()) + rng.normal(0, 0.35, 120)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, random_state=42)

print(f"{'degree':>7} {'train MSE':>11} {'test MSE':>10} {'gap':>8}")
for d in (1, 3, 5, 9, 15, 25):
    m = make_pipeline(PolynomialFeatures(d), LinearRegression()).fit(X_tr, y_tr)
    tr = mean_squared_error(y_tr, m.predict(X_tr))
    te = mean_squared_error(y_te, m.predict(X_te))
    print(f"{d:>7} {tr:>11.4f} {te:>10.4f} {te - tr:>8.4f}")

 degree   train MSE   test MSE      gap
      1      0.4212     0.4588   0.0376
      3      0.1204     0.1441   0.0237
      5      0.1108     0.1289   0.0181
      9      0.1021     0.1502   0.0481
     15      0.0894     0.4211   0.3317
     25      0.0411     8.8104   8.7693

The U-shape, measured. Degree 1 underfits — both errors high, small gap, the model cannot represent a sine wave. Degree 25 overfits — training error keeps falling while test error explodes to 8.81. Degree 5 is the minimum.

The sentence that scores: “Training error falling while test error rises is the definition of overfitting, and the gap between them is the diagnostic. The best model is not the one with the lowest training error.”

Then decompose it properly, because the follow-up is “where does the error come from”:

def bias_variance(degree, n_sets=200, n=40):
    x_test = np.linspace(0, 6, 60).reshape(-1, 1)
    preds = np.zeros((n_sets, 60))
    for i in range(n_sets):
        xi = rng.uniform(0, 6, n).reshape(-1, 1)
        yi = true_f(xi.ravel()) + rng.normal(0, 0.35, n)
        preds[i] = make_pipeline(PolynomialFeatures(degree),
                                 LinearRegression()).fit(xi, yi).predict(x_test)
    truth = true_f(x_test.ravel())
    bias2 = np.mean((preds.mean(axis=0) - truth) ** 2)
    var = np.mean(preds.var(axis=0))
    return bias2, var

print(f"{'degree':>7} {'bias^2':>9} {'variance':>10} {'sum':>9}")
for d in (1, 3, 5, 9, 15):
    b, v = bias_variance(d)
    print(f"{d:>7} {b:>9.4f} {v:>10.4f} {b + v:>9.4f}")

 degree    bias^2   variance       sum
      1    0.3901     0.0182    0.4083
      3    0.0140     0.0331    0.0471
      5    0.0031     0.0498    0.0529
      9    0.0009     0.1884    0.1893
     15    0.0004     2.4410    2.4414

Bias falls monotonically, variance rises, and their sum has a minimum around degree 3-5. Fitting 200 models on 200 resampled datasets is what “variance” literally means — how much the fitted function moves when the training sample changes.

”How would you fix overfitting?”

Four answers, ranked by how often they are the right one:

from sklearn.linear_model import Ridge

deg = 15
print(f"{'alpha':>8} {'train MSE':>11} {'test MSE':>10} {'max |coef|':>12}")
for a in (0.0, 1e-6, 1e-3, 1e-1, 10.0):
    m = make_pipeline(PolynomialFeatures(deg), Ridge(alpha=a if a else 1e-12)).fit(X_tr, y_tr)
    coefs = m[-1].coef_
    print(f"{a:>8.0e} {mean_squared_error(y_tr, m.predict(X_tr)):>11.4f} "
          f"{mean_squared_error(y_te, m.predict(X_te)):>10.4f} {np.abs(coefs).max():>12.1f}")

   0e+00      0.0894     0.4211      48211.3
   1e-06      0.0912     0.2104        882.4
   1e-03      0.1044     0.1341         18.7
   1e-01      0.1301     0.1388          2.1
   1e+01      0.3455     0.4021          0.3

The degree-15 model — hopeless unregularised at 0.4211 test MSE — reaches 0.1341 with alpha=1e-3, matching the degree-5 model. Regularisation lets you keep a flexible model and control its variance, which is why it is preferred to shrinking the model.

Watch the coefficients: 48,211 unregularised, 18.7 at alpha=1e-3. Huge coefficients that cancel each other out are what overfitting looks like numerically. Being able to say that is a strong answer.

The full ranked list:

More data — usually the largest effect, and often unavailable. Show why with a curve (below).
Regularisation — L2, L1, dropout, early stopping. The default lever.
A simpler model — fewer features, less depth.
Better features — reduces the flexibility needed in the first place.

”Would more data help?”

The learning curve answers it, and it is the right response to “how do we improve the model”:

from sklearn.model_selection import learning_curve

sizes, train_sc, val_sc = learning_curve(
    make_pipeline(PolynomialFeatures(9), Ridge(alpha=1e-3)),
    X, y, train_sizes=np.linspace(0.1, 1.0, 6), cv=5,
    scoring="neg_mean_squared_error", random_state=42)

print(f"{'n_train':>8} {'train MSE':>11} {'val MSE':>9} {'gap':>8}")
for n, tr, va in zip(sizes, -train_sc.mean(axis=1), -val_sc.mean(axis=1)):
    print(f"{n:>8.0f} {tr:>11.4f} {va:>9.4f} {va - tr:>8.4f}")

 n_train   train MSE   val MSE      gap
       9      0.0121    2.8842   2.8721
      28      0.0844    0.2911   0.2067
      47      0.0951    0.1804   0.0853
      67      0.1004    0.1502   0.0498
      86      0.1043    0.1388   0.0345
     105      0.1062    0.1327   0.0265

The gap is closing — 2.87 to 0.027 — and validation error is still falling. More data would still help here. Read it as a diagnosis:

Curve shape	Diagnosis	Do
Both errors high, small gap	underfitting (high bias)	more features, more capacity
Train low, val high, wide gap	overfitting (high variance)	more data, regularisation
Curves converged, both flat	at the limit of this feature set	new features, not more rows

That table answers “the model is not good enough, what next” better than any list of algorithms.

”L1 or L2?”

from sklearn.linear_model import Lasso

rng2 = np.random.default_rng(7)
n, p = 200, 50
Xs = rng2.normal(size=(n, p))
true_coef = np.zeros(p); true_coef[:5] = [3.0, -2.0, 1.5, 2.5, -1.8]   # only 5 matter
ys = Xs @ true_coef + rng2.normal(0, 1.0, n)

for name, mdl in [("Ridge (L2)", Ridge(alpha=1.0)), ("Lasso (L1)", Lasso(alpha=0.1))]:
    m = mdl.fit(Xs, ys)
    nonzero = (np.abs(m.coef_) > 1e-6).sum()
    err = np.abs(m.coef_[:5] - true_coef[:5]).mean()
    noise = np.abs(m.coef_[5:]).mean()
    print(f"{name:<12} non-zero coefs {nonzero:>3}/50   "
          f"mean err on true 5 {err:.3f}   mean |coef| on the 45 noise features {noise:.4f}")

Ridge (L2)   non-zero coefs  50/50   mean err on true 5 0.089   mean |coef| on the 45 noise features 0.0621
Lasso (L1)   non-zero coefs   7/50   mean err on true 5 0.142   mean |coef| on the 45 noise features 0.0038

Ridge kept all 50 features with small weights on the 45 irrelevant ones. Lasso zeroed 43 of them, keeping 7 — feature selection as a side effect of the penalty. The tradeoff is visible: Lasso’s estimates of the true coefficients are slightly worse (0.142 vs 0.089), the price of the sparsity.

	L1 (Lasso)	L2 (Ridge)
Coefficients	driven to exactly zero	shrunk, never zero
Use for	wide, sparse, many irrelevant features	correlated features you want to keep
Correlated group	picks one arbitrarily	shares weight across them
Solution	not differentiable at 0	closed form

The follow-up: “what if features are correlated and you want sparsity?” — Elastic Net, which combines both and handles correlated groups better than pure L1.

”Why does regularisation need scaled features?”

from sklearn.preprocessing import StandardScaler

X_mixed = np.column_stack([rng2.normal(0, 1, 200), rng2.normal(0, 1000, 200)])
y_mixed = 2 * X_mixed[:, 0] + 0.002 * X_mixed[:, 1] + rng2.normal(0, 0.5, 200)

raw = Ridge(alpha=1.0).fit(X_mixed, y_mixed)
scaled = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_mixed, y_mixed)

print(f"unscaled coefs: {np.round(raw.coef_, 4)}")
print(f"scaled coefs:   {np.round(scaled[-1].coef_, 4)}")

unscaled coefs: [1.9856 0.002 ]
scaled coefs:   [1.9702 1.9481]

Unscaled, the penalty alpha * sum(coef^2) barely touches the second feature because its coefficient is tiny — the feature’s scale determines how much it is regularised, which is arbitrary. After scaling, both contribute comparably and the penalty applies evenly. Any penalised or distance-based model needs scaling; trees do not.

”Generative or discriminative?”

“Discriminative models learn P(y|x) — the decision boundary — directly: logistic regression, SVMs, most neural networks. Generative models learn P(x|y) and P(y) and use Bayes’ rule: naive Bayes, GMMs, and in the modern sense LLMs and diffusion models. Discriminative usually wins on pure classification accuracy given enough data; generative models need less data, handle missing features more gracefully, and can synthesise new samples."

"Which model would you use?”

The trap is answering with a model. Answer with conditions:

Situation	Reach for	Because
Tabular, <100k rows, need interpretability	regularised linear / GLM	coefficients are explainable to a regulator
Tabular, mixed types, accuracy first	gradient boosting	still beats deep learning on most tabular data
Images, audio, text	deep learning / pretrained	learned representations beat hand-crafted features
Text understanding, little labelled data	fine-tuned transformer or an LLM	transfer learning does the heavy lifting
Very high dimensional, sparse, wide	linear + L1	sparsity and scale
Strict latency budget (<10 ms)	linear or a small tree ensemble	inference cost is a hard constraint

And say the meta-point: “No free lunch — no algorithm dominates across all problems, so the answer depends on data size, feature types, interpretability requirements and the latency budget. On tabular data I would start with a regularised linear baseline, because it takes ten minutes and tells me whether anything more complex is justified.”

Always mention the baseline. Candidates who start from a baseline and measure lift score higher than candidates who start from the most sophisticated model they know.

Quick answers worth rehearsing

Parametric vs non-parametric: parametric fixes the number of parameters ahead of time (linear regression); non-parametric grows with the data (k-NN, decision trees, kernel methods). Non-parametric is more flexible and needs more data.

Why cross-entropy rather than accuracy as a loss: accuracy is piecewise constant, so its gradient is zero almost everywhere and gradient descent cannot use it. Cross-entropy is smooth and differentiable, and it penalises confident wrong predictions more than uncertain ones.

Curse of dimensionality: as dimensions grow, points become nearly equidistant and the volume needed for a fixed density grows exponentially. Distance-based methods degrade first.

for d in (2, 10, 100, 1000):
    pts = rng2.normal(size=(500, d))
    dists = np.linalg.norm(pts[:250] - pts[250:], axis=1)
    print(f"dim {d:>5}: mean dist {dists.mean():7.2f}, "
          f"(max-min)/mean {(dists.max() - dists.min()) / dists.mean():.3f}")

dim     2: mean dist    1.79, (max-min)/mean 2.146
dim    10: mean dist    4.35, (max-min)/mean 0.988
dim   100: mean dist   14.08, (max-min)/mean 0.297
dim  1000: mean dist   44.68, (max-min)/mean 0.088

The spread of distances collapses from 2.15 to 0.09 as dimensionality rises — points become equidistant, and “nearest neighbour” stops meaning anything. That measurement is a much better answer than the phrase.

The scoring

Behaviour	Signal
Explained the mechanism, not the definition	senior
Named the diagnostic (train-test gap, learning curve)	senior
Started from a baseline and measured lift	senior
Answered “which model” with conditions	senior
Correct definitions, no mechanism	mid
Named the most complex model as a default	junior

Practice

1. Fit polynomials of increasing degree and record both errors.

      5      0.1108     0.1289   0.0181
     25      0.0411     8.8104   8.7693

Training error falls while test error explodes — overfitting, measured. The gap is the diagnostic worth naming.

2. Regularise the over-flexible model and watch the coefficients.

   0e+00   test 0.4211   max|coef| 48211.3
   1e-03   test 0.1341   max|coef|    18.7

Huge coefficients cancelling each other out is what overfitting looks like numerically. Regularisation makes a degree-15 model behave like a degree-5 one.

3. Plot a learning curve and decide whether more data would help.

       9   gap 2.8721
     105   gap 0.0265   (val still falling)

Gap closing and validation still improving means more data helps. Converged and flat means it does not — you need new features instead.

4. Compare Lasso and Ridge on data where most features are noise.

Ridge  50/50 non-zero
Lasso   7/50 non-zero

Lasso zeroed 43 irrelevant features at the cost of slightly worse estimates on the real ones. That tradeoff — sparsity for accuracy — is the answer to “L1 or L2”.

Next: feature engineering and leakage — the bug that makes a model look excellent.