The AI and ML Interview

“ML interview” covers four different jobs with four different loops. Working out which one you are in — ideally before the first round — decides what to prepare.

The four roles

The job title is unreliable — “Machine Learning Engineer” is used for all four. Ask the recruiter directly: “Is this role closer to building models or to serving them in production?” The answer changes what you revise.

Common to all four:

Round 1  Screen           fundamentals, one metric question, a project deep dive
Round 2  Coding           numpy/pandas manipulation, sometimes an algorithm from scratch
Round 3  ML depth         evaluation, leakage, overfitting, a model-choice tradeoff
Round 4  ML system design  end-to-end: data → training → serving → monitoring
Round 5  Behavioural       a project you owned, and a decision you got wrong

The warm-up that filters

“You have built a model to predict which customers will churn. It gets 94% accuracy. Ship it?”

Almost everyone has an answer to this. The answers separate sharply.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, confusion_matrix, classification_report,
                             roc_auc_score, average_precision_score)

rng = np.random.default_rng(42)
n = 10_000

tenure   = rng.gamma(2.0, 12.0, n)
tickets  = rng.poisson(0.6, n)
monthly  = rng.normal(45, 15, n).clip(5, 200)

logit = -3.4 - 0.020 * tenure + 0.45 * tickets + 0.008 * monthly
churn = rng.binomial(1, 1 / (1 + np.exp(-logit)))

X = np.column_stack([tenure, tickets, monthly])
X_tr, X_te, y_tr, y_te = train_test_split(X, churn, test_size=0.25,
                                          random_state=42, stratify=churn)

print(f"rows: {n:,}   churn rate: {churn.mean():.1%}")

rows: 10,000   churn rate: 6.2%

There it is, before any modelling: 6.2% of customers churn. Say that number first.

model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
pred = model.predict(X_te)
proba = model.predict_proba(X_te)[:, 1]

print(f"accuracy: {accuracy_score(y_te, pred):.3f}")
print(f"a model that always predicts 'no churn': {1 - y_te.mean():.3f}")

accuracy: 0.941
a model that always predicts 'no churn': 0.938

94.1% against a 93.8% baseline. The model is worth 0.3 percentage points of accuracy over predicting “nobody churns” — a rule you could write on a napkin.

What it is actually doing

cm = confusion_matrix(y_te, pred)
print("            pred_no  pred_yes")
print(f"actual_no   {cm[0,0]:7d}  {cm[0,1]:8d}")
print(f"actual_yes  {cm[1,0]:7d}  {cm[1,1]:8d}")
print()
print(classification_report(y_te, pred, target_names=["retained", "churned"], digits=3))

            pred_no  pred_yes
actual_no      2344         2
actual_yes      145         9

              precision    recall  f1-score   support

    retained      0.942     0.999     0.970      2346
     churned      0.818     0.058     0.109       154

    accuracy                          0.941      2500
   macro avg      0.880     0.529     0.539      2500
weighted avg      0.934     0.941     0.917      2500

Recall on the class you care about is 0.058. The model finds 9 of 154 churners. As a retention tool it is useless, and accuracy hid that completely.

Use metrics that ignore the majority class:

print(f"ROC AUC:            {roc_auc_score(y_te, proba):.3f}")
print(f"PR AUC (avg prec):  {average_precision_score(y_te, proba):.3f}")
print(f"random PR baseline: {y_te.mean():.3f}")

ROC AUC:            0.771
PR AUC (avg prec):  0.191
random PR baseline: 0.062

There is signal — PR AUC of 0.19 against a 0.06 baseline is 3× better than random. The ranking works; the decision threshold is what is broken.

The threshold is a business decision

print(f"{'threshold':>10} {'precision':>10} {'recall':>8} {'flagged':>8} {'caught':>7}")
for t in (0.50, 0.20, 0.10, 0.062, 0.04):
    p = (proba >= t).astype(int)
    tp, fp = ((p == 1) & (y_te == 1)).sum(), ((p == 1) & (y_te == 0)).sum()
    prec = tp / max(tp + fp, 1)
    rec = tp / y_te.sum()
    print(f"{t:>10.3f} {prec:>10.3f} {rec:>8.3f} {tp+fp:>8d} {tp:>7d}")

 threshold  precision   recall  flagged  caught
     0.500      0.818    0.058       11       9
     0.200      0.404    0.247       94      38
     0.100      0.221    0.539      376      83
     0.062      0.152    0.708      717     109
     0.040      0.107    0.870     1252     134

0.5 is a library default, not a decision. Then do the arithmetic that makes it a decision:

VALUE_OF_SAVE, COST_OF_CONTACT, SAVE_RATE = 400, 15, 0.30

print(f"{'threshold':>10} {'contacted':>10} {'saves':>7} {'cost':>9} {'benefit':>9} {'net':>9}")
best = None
for t in (0.50, 0.20, 0.10, 0.062, 0.04, 0.02):
    p = (proba >= t)
    contacted = p.sum()
    saves = ((p) & (y_te == 1)).sum() * SAVE_RATE
    cost, benefit = contacted * COST_OF_CONTACT, saves * VALUE_OF_SAVE
    net = benefit - cost
    best = max(best or (net, t), (net, t))
    print(f"{t:>10.3f} {contacted:>10d} {saves:>7.0f} {cost:>9,.0f} {benefit:>9,.0f} {net:>9,.0f}")
print(f"\noptimal threshold: {best[1]} (net £{best[0]:,.0f})")

 threshold  contacted   saves      cost   benefit       net
     0.500         11       3       165     1,080       915
     0.200         94      11     1,410     4,560     3,150
     0.100        376      25     5,640     9,960     4,320
     0.062        717      33    10,755    13,080     2,325
     0.040       1,252      40    18,780    16,080    -2,700
     0.020       1,914      44    28,710    17,640   -11,070

optimal threshold: 0.1 (net £4,320)

The best threshold is 0.10, worth about £4,320 per quarter on this sample — and going further turns profitable at 0.10 into a £2,700 loss at 0.04, because contacting everyone costs more than the saves are worth. That table is the answer to the question.

Two answers, same model

Marked down:

“94% accuracy is good, so I’d ship it. I could try a random forest to push it higher.”

Marked up:

“94% is close to the 93.8% base rate — predicting ‘nobody churns’ scores nearly the same, so accuracy is the wrong metric here. Recall on churners is 5.8%, so it catches 9 of 154. There is real signal — PR AUC 0.19 against a 0.06 baseline — but the 0.5 threshold is wrong for a 6% positive rate. What is the cost of contacting a customer and the value of a save? At £15 and £400 with a 30% save rate, the optimum is around 0.10: contact 15% of the base, catch half the churn, net about £4,300. And before shipping I would check the features for leakage — anything recorded after a cancellation decision would explain a suspiciously good model.”

Same model, same numbers. The second answer questioned the metric, quantified the tradeoff, and raised leakage unprompted.

What each round rewards

The pattern across all five: candidates who interrogate the problem beat candidates who optimise the answer.

Preparing

The highest-return item is not another algorithm. It is being able to talk for thirty minutes about one model you shipped: what you tried, what the baseline was, what broke in production, and what you would do differently. That covers the deep dive and most of the behavioural round.

Practice

accuracy: 0.941
always-predict-no baseline: 0.938

actual_yes      145         9

     0.500      0.818    0.058
     0.100      0.221    0.539

optimal threshold: 0.1 (net £4,320)
     0.040 ... net -2,700

Next: fundamentals — bias-variance, regularisation, and the questions behind the questions.