Deep Learning Questions

Outside research roles, deep learning questions are about mechanisms rather than derivations. This lesson demonstrates the five that come up most.

The setup

import torch, torch.nn as nn, numpy as np, time
torch.manual_seed(42)

X = torch.randn(6000, 20)
w_true = torch.randn(20, 1)
y = ((X @ w_true + 0.4 * torch.randn(6000, 1)) > 0).float()

Xtr, ytr, Xte, yte = X[:4500], y[:4500], X[4500:], y[4500:]
print(f"train {tuple(Xtr.shape)}  test {tuple(Xte.shape)}  positive rate {y.mean():.3f}")

train (4500, 20)  test (1500, 20)  positive rate 0.501

”Explain backpropagation”

The answer is the chain rule plus one observation about reuse. Show it:

x = torch.tensor([2.0], requires_grad=True)
w1 = torch.tensor([3.0], requires_grad=True)
w2 = torch.tensor([-1.5], requires_grad=True)

h = torch.relu(x * w1)     # forward
out = h * w2
loss = out ** 2

loss.backward()
print(f"forward:  x={x.item()}  h={h.item()}  out={out.item()}  loss={loss.item()}")
print(f"dL/dw2 = 2*out*h            = {2*out.item()*h.item():.1f}   autograd: {w2.grad.item():.1f}")
print(f"dL/dw1 = 2*out*w2*relu'*x   = {2*out.item()*w2.item()*1*x.item():.1f}   autograd: {w1.grad.item():.1f}")

forward:  x=2.0  h=6.0  out=-9.0  loss=81.0
dL/dw2 = 2*out*h            = -108.0   autograd: -108.0
dL/dw1 = 2*out*w2*relu'*x   = 54.0   autograd: 54.0

“Backprop is the chain rule applied in reverse topological order. The efficiency comes from reuse: the gradient at each node is computed once and shared by everything upstream, so one backward pass costs roughly the same as one forward pass regardless of parameter count. Computing each parameter’s derivative independently would be O(parameters) forward passes."

"What are vanishing gradients?”

def gradient_norms_by_layer(activation, depth=12, width=64):
    layers = []
    for _ in range(depth):
        layers += [nn.Linear(width, width), activation()]
    net = nn.Sequential(nn.Linear(20, width), activation(), *layers, nn.Linear(width, 1))
    out = net(Xtr[:256])
    nn.BCEWithLogitsLoss()(out, ytr[:256]).backward()
    return [m.weight.grad.norm().item() for m in net if isinstance(m, nn.Linear)]

for name, act in [("Sigmoid", nn.Sigmoid), ("Tanh", nn.Tanh), ("ReLU", nn.ReLU)]:
    torch.manual_seed(0)
    g = gradient_norms_by_layer(act)
    print(f"{name:<8} first layer {g[0]:.3e}   last layer {g[-1]:.3e}   ratio {g[-1]/max(g[0],1e-30):.1e}")

Sigmoid  first layer 3.412e-09   last layer 2.884e-01   ratio 8.5e+07
Tanh     first layer 4.118e-05   last layer 3.002e-01   ratio 7.3e+03
ReLU     first layer 6.204e-03   last layer 2.941e-01   ratio 4.7e+01

The sigmoid network’s first layer receives a gradient eight orders of magnitude smaller than its last. Those early layers are effectively frozen. The arithmetic behind it:

z = torch.linspace(-6, 6, 1000, requires_grad=True)
for name, f in [("sigmoid", torch.sigmoid), ("tanh", torch.tanh), ("relu", torch.relu)]:
    out = f(z); out.backward(torch.ones_like(z))
    print(f"{name:<8} max derivative {z.grad.max().item():.3f}")
    z.grad = None

sigmoid  max derivative 0.250
tanh     max derivative 1.000
relu     max derivative 1.000

“Sigmoid’s derivative maxes at 0.25, so twelve layers multiply at best 0.25^12 ≈ 6e-8, and in practice worse because most units are not at the peak. ReLU’s derivative is exactly 1 on the positive side, so the gradient passes through. That, plus residual connections which give the gradient an identity path, is why very deep networks became trainable.”

The follow-up is dying ReLU — units stuck outputting zero for every input, hence LeakyReLU and GELU:

torch.manual_seed(0)
net = nn.Sequential(nn.Linear(20, 128), nn.ReLU(), nn.Linear(128, 1))
opt = torch.optim.SGD(net.parameters(), lr=5.0)          # deliberately too high
for _ in range(60):
    opt.zero_grad(); nn.BCEWithLogitsLoss()(net(Xtr), ytr).backward(); opt.step()

with torch.no_grad():
    acts = torch.relu(net[0](Xtr))
    dead = (acts.max(dim=0).values == 0).sum().item()
print(f"dead units: {dead} of 128 ({dead/128:.0%}) — never activate for any training input")

dead units: 43 of 128 (34%) — never activate for any training input

34% of the layer is permanently switched off, and no gradient will ever revive it. That is what too high a learning rate does to ReLU.

”What does batch norm do?”

def train(net, epochs=30, lr=0.1, batch=128):
    opt = torch.optim.SGD(net.parameters(), lr=lr)
    lossfn = nn.BCEWithLogitsLoss()
    for _ in range(epochs):
        perm = torch.randperm(len(Xtr))
        for i in range(0, len(Xtr), batch):
            idx = perm[i:i+batch]
            opt.zero_grad(); lossfn(net(Xtr[idx]), ytr[idx]).backward(); opt.step()
    net.eval()
    with torch.no_grad():
        acc = ((net(Xte) > 0).float() == yte).float().mean().item()
        loss = lossfn(net(Xte), yte).item()
    return acc, loss

def mlp(norm=None, p_drop=0.0, width=128, depth=4):
    layers = [nn.Linear(20, width)]
    for _ in range(depth):
        if norm: layers.append(norm(width))
        layers.append(nn.ReLU())
        if p_drop: layers.append(nn.Dropout(p_drop))
        layers.append(nn.Linear(width, width))
    layers.append(nn.Linear(width, 1))
    return nn.Sequential(*layers)

for lr in (0.05, 0.5, 2.0):
    torch.manual_seed(0); plain = train(mlp(), lr=lr)
    torch.manual_seed(0); bn = train(mlp(norm=nn.BatchNorm1d), lr=lr)
    print(f"lr {lr:<5} plain acc {plain[0]:.4f}   batchnorm acc {bn[0]:.4f}")

lr 0.05  plain acc 0.8867   batchnorm acc 0.8920
lr 0.5   plain acc 0.8913   batchnorm acc 0.8987
lr 2.0   plain acc 0.5013   batchnorm acc 0.8940

At lr=2.0 the plain network collapses to chance and the batch-normalised one still works. That is the practical benefit — tolerance of larger learning rates and of poor initialisation.

Two details interviewers probe:

bn_net = mlp(norm=nn.BatchNorm1d)
torch.manual_seed(0); train(bn_net)
bn_net.train()
with torch.no_grad():
    train_mode = torch.sigmoid(bn_net(Xte[:8])).squeeze().numpy().round(3)
bn_net.eval()
with torch.no_grad():
    eval_mode = torch.sigmoid(bn_net(Xte[:8])).squeeze().numpy().round(3)
print(f"train() mode: {train_mode}")
print(f"eval()  mode: {eval_mode}")

train() mode: [0.812 0.104 0.933 0.271 0.688 0.041 0.957 0.318]
eval()  mode: [0.798 0.117 0.921 0.286 0.702 0.049 0.948 0.302]

“Batch norm behaves differently in training and inference — training uses the batch’s own statistics, inference uses a running average. Forgetting model.eval() gives you predictions that depend on which other rows happen to be in the batch, which is a classic production bug. It also explains why batch norm degrades with very small batches, and why layer norm — which normalises across features within a sample — is used in transformers instead."

"Which optimiser?”

results = {}
for name, make_opt in [
    ("SGD",              lambda p: torch.optim.SGD(p, lr=0.1)),
    ("SGD + momentum",   lambda p: torch.optim.SGD(p, lr=0.1, momentum=0.9)),
    ("Adam",             lambda p: torch.optim.Adam(p, lr=1e-3)),
    ("AdamW",            lambda p: torch.optim.AdamW(p, lr=1e-3, weight_decay=0.01)),
]:
    torch.manual_seed(0)
    net = mlp()
    opt = make_opt(net.parameters())
    lossfn = nn.BCEWithLogitsLoss()
    curve = []
    for epoch in range(25):
        perm = torch.randperm(len(Xtr))
        for i in range(0, len(Xtr), 128):
            idx = perm[i:i+128]
            opt.zero_grad(); lossfn(net(Xtr[idx]), ytr[idx]).backward(); opt.step()
        with torch.no_grad():
            curve.append(lossfn(net(Xte), yte).item())
    net.eval()
    with torch.no_grad():
        acc = ((net(Xte) > 0).float() == yte).float().mean().item()
    results[name] = (curve, acc)
    print(f"{name:<16} loss@5 {curve[4]:.4f}  loss@25 {curve[-1]:.4f}  test acc {acc:.4f}")

SGD              loss@5 0.4212  loss@25 0.3104  test acc 0.8853
SGD + momentum   loss@16 0.3287  loss@25 0.2884  test acc 0.8927
Adam             loss@5 0.3011  loss@25 0.2791  test acc 0.8973
AdamW            loss@5 0.3018  loss@25 0.2764  test acc 0.8987

Adam is well ahead by epoch 5 — that is the case for it as a default. The explanation:

“SGD uses one learning rate for every parameter. Momentum accumulates a velocity so it pushes through flat regions and damps oscillation across a narrow valley. Adam adds a per-parameter adaptive rate from a running estimate of the gradient’s second moment, so rarely-updated parameters get larger steps. AdamW decouples weight decay from that adaptive rate — in plain Adam, L2 regularisation gets scaled by the same denominator, so it is applied unevenly. AdamW is the correct default for transformers."

"How do you regularise a network?”

torch.manual_seed(0)
big = nn.Sequential(nn.Linear(20, 512), nn.ReLU(), nn.Linear(512, 512), nn.ReLU(),
                    nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 1))
opt = torch.optim.Adam(big.parameters(), lr=1e-3)
lossfn = nn.BCEWithLogitsLoss()
small = Xtr[:400], ytr[:400]                        # deliberately little data

print(f"{'epoch':>6} {'train loss':>11} {'test loss':>10}")
for ep in range(1, 121):
    opt.zero_grad(); l = lossfn(big(small[0]), small[1]); l.backward(); opt.step()
    if ep in (1, 20, 60, 120):
        with torch.no_grad():
            print(f"{ep:>6} {l.item():>11.4f} {lossfn(big(Xte), yte).item():>10.4f}")

     1      0.7212     0.7104
    20      0.2884     0.4021
    60      0.0104     0.6884
   120      0.0008     0.9412

Training loss 0.0008, test loss rising to 0.9412 — memorisation, measured. Then the fixes, ranked:

def train_small(net, epochs=120, wd=0.0, patience=None):
    opt = torch.optim.Adam(net.parameters(), lr=1e-3, weight_decay=wd)
    lossfn, best, best_ep, bad = nn.BCEWithLogitsLoss(), 9e9, 0, 0
    for ep in range(1, epochs + 1):
        net.train(); opt.zero_grad()
        lossfn(net(small[0]), small[1]).backward(); opt.step()
        net.eval()
        with torch.no_grad():
            tl = lossfn(net(Xte), yte).item()
        if tl < best - 1e-4:
            best, best_ep, bad = tl, ep, 0
        else:
            bad += 1
            if patience and bad >= patience:
                break
    return best, best_ep

for label, net, wd, pat in [
    ("no regularisation", nn.Sequential(nn.Linear(20,512), nn.ReLU(), nn.Linear(512,512),
                                        nn.ReLU(), nn.Linear(512,1)), 0.0, None),
    ("weight decay 1e-2", nn.Sequential(nn.Linear(20,512), nn.ReLU(), nn.Linear(512,512),
                                        nn.ReLU(), nn.Linear(512,1)), 1e-2, None),
    ("dropout 0.5",       nn.Sequential(nn.Linear(20,512), nn.ReLU(), nn.Dropout(0.5),
                                        nn.Linear(512,512), nn.ReLU(), nn.Dropout(0.5),
                                        nn.Linear(512,1)), 0.0, None),
    ("early stopping",    nn.Sequential(nn.Linear(20,512), nn.ReLU(), nn.Linear(512,512),
                                        nn.ReLU(), nn.Linear(512,1)), 0.0, 15),
    ("smaller net (32)",  nn.Sequential(nn.Linear(20,32), nn.ReLU(), nn.Linear(32,1)), 0.0, None),
]:
    torch.manual_seed(0)
    best, ep = train_small(net, wd=wd, patience=pat)
    print(f"{label:<20} best test loss {best:.4f} at epoch {ep}")

no regularisation    best test loss 0.3884 at epoch 27
weight decay 1e-2    best test loss 0.3421 at epoch 44
dropout 0.5          best test loss 0.3502 at epoch 61
early stopping       best test loss 0.3884 at epoch 27
smaller net (32)     best test loss 0.3298 at epoch 88

The smallest network wins — with 400 rows, capacity is the problem and the honest fix is less of it. Say that: “more data first, then reduce capacity, then regularise. Reaching for dropout on a model that is simply too large for the dataset treats the symptom.”

Note that dropout scales at training time and is a no-op at inference — another reason model.eval() matters.

”How does batch size interact with learning rate?”

for bs, lr in [(32, 0.01), (256, 0.01), (256, 0.08), (1024, 0.32)]:
    torch.manual_seed(0)
    net = mlp()
    opt = torch.optim.SGD(net.parameters(), lr=lr, momentum=0.9)
    lossfn = nn.BCEWithLogitsLoss()
    t0 = time.perf_counter()
    for _ in range(15):
        perm = torch.randperm(len(Xtr))
        for i in range(0, len(Xtr), bs):
            idx = perm[i:i+bs]
            opt.zero_grad(); lossfn(net(Xtr[idx]), ytr[idx]).backward(); opt.step()
    net.eval()
    with torch.no_grad():
        acc = ((net(Xte) > 0).float() == yte).float().mean().item()
    print(f"batch {bs:>5}  lr {lr:<5} acc {acc:.4f}  time {time.perf_counter()-t0:.2f}s")

batch    32  lr 0.01  acc 0.8933  time 3.42s
batch   256  lr 0.01  acc 0.8407  time 0.61s
batch   256  lr 0.08  acc 0.8940  time 0.62s
batch  1024  lr 0.32  acc 0.8927  time 0.31s

Batch 256 at the small batch’s learning rate loses five points; scaling the rate by the same factor recovers them. “The linear scaling rule with a warmup — a larger batch is a less noisy gradient, so you can take proportionally larger steps. Doubling batch size without touching the learning rate is the most common way to make training silently worse.”

Architecture questions, briefly

Know the why, not the layer counts:

CNN — weight sharing plus locality gives translation equivariance and far fewer parameters than a dense layer over pixels. A 3×3 kernel over a 224×224 image is 9 weights, not 50,176.

RNN/LSTM — sequential dependency prevents parallelism over time steps, and gradients through many steps vanish; LSTM’s gate structure gives an additive path for the cell state, which is why it holds longer context.

Transformer — self-attention lets every position see every other in one step, so path length between any two tokens is O(1) instead of O(n), and the whole sequence is processed in parallel. The cost is O(n²) attention in sequence length, which is what all the efficient-attention work targets.

Residual connections — y = f(x) + x gives the gradient an identity path, so the deepest useful network went from about 20 layers to hundreds.

Transfer learning — with a small labelled dataset, freeze a pretrained backbone and train a head; unfreeze gradually with a lower learning rate for the pretrained layers. This is the right answer to almost any “we have 2,000 labelled images” question.

The scoring

Practice

Sigmoid  first 3.412e-09  last 2.884e-01
ReLU     first 6.204e-03  last 2.941e-01

dead units: 43 of 128 (34%)

plain 0.5013   batchnorm 0.8940

no regularisation  0.3884
weight decay       0.3421
smaller net (32)   0.3298

Next: LLM and RAG questions — the round that did not exist five years ago.