Why did the healthy container in this lesson turn unhealthy on its own?

It didn't fail on its own — the demo app deliberately flips its own /health endpoint to return 500 after 15 seconds, so the healthcheck (checking every 3s) has something real to catch. In a real app, the equivalent is a dependency going down: a database connection pool exhausting, a downstream API timing out.

How do I do zero-downtime deployments with Docker?

Start the new container, poll its health status until Docker itself reports 'healthy' (not just 'container running'), then stop the old one and reuse its name. This lesson runs that exact sequence for real: old serves traffic, new starts and becomes healthy, then old is stopped and new is renamed into its place — no window where both are down.

What logging driver should I use in production?

json-file is the default — simple, but files accumulate on disk with no rotation unless you set max-size and max-file, which this lesson demonstrates rotating for real. For centralized logging, awslogs (CloudWatch), splunk, or a fluentd/fluentbit sidecar are the common choices — swapping --log-driver is the only change needed.

Does a BuildKit cache mount speed up every rebuild, or only some?

Only rebuilds where the layer containing the RUN instruction is invalidated — normally by a change to files it depends on, like package.json. If nothing upstream changed, Docker's own layer cache skips the RUN entirely and the cache mount never even gets consulted. The cache mount's value is specifically for the case this lesson measures: something forces the layer to re-run, but the underlying package versions haven't changed, so the download step shouldn't have to start from zero.

Why does a container get OOM-killed instead of just running slower?

Linux's cgroup memory controller enforces --memory as a hard ceiling, not a suggestion — when a container's actual usage would exceed it, the kernel's OOM killer sends SIGKILL to reclaim the memory immediately. There's no graceful degradation or swapping to disk by default; docker inspect reporting OOMKilled: true is the kernel-level record of exactly what happened, not something Docker itself decided.

Docker in Production | SkillByExample

Nine lessons in, every command so far has run against a container with no limits, no restart policy, and no health signal. Production removes that slack — this lesson shows what fails, on purpose, so the fix means something.

Health Checks: A Real Healthy-to-Unhealthy Transition

HEALTHCHECK --interval=3s --timeout=2s --start-period=2s --retries=2 \
  CMD wget -qO- http://localhost:3000/health || exit 1

The demo app for this section serves /health with 200 for its first 15 seconds, then flips to 500 — simulating a dependency going down partway through a container’s life, not a crash:

docker run -d --name docprod-hc -p 3010:3000 \
  --health-cmd="wget -qO- http://localhost:3000/health || exit 1" \
  --health-interval=3s --health-timeout=2s --health-start-period=2s --health-retries=2 \
  docprod-app:1.0

sleep 5 && docker ps --filter name=docprod-hc --format "table {{.Names}}\t{{.Status}}"

NAMES        STATUS
docprod-hc   Up 9 seconds (healthy)

sleep 20 && docker ps --filter name=docprod-hc --format "table {{.Names}}\t{{.Status}}"
docker inspect --format='{{json .State.Health}}' docprod-hc

NAMES        STATUS
docprod-hc   Up 34 seconds (unhealthy)
{"Status":"unhealthy","FailingStreak":6,"Log":[{"Start":"2026-09-16T12:55:14Z","End":"2026-09-16T12:55:15Z","ExitCode":1,"Output":"wget: server returned error: HTTP/1.1 500 Internal Server Error\n"}, ...]}

docker ps only ever reported Up before this lesson — it had nothing else to check. With a HEALTHCHECK, Docker actually probes the process and remembers a real failure log, not just “is the PID alive.”

Same shape in Compose:

services:
  api:
    image: myapi:1.0
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      start_period: 10s
      retries: 3

Resource Limits: A Real OOM Kill

A container with no memory limit can consume the whole host. One with too tight a limit gets killed — for real, not as a warning. The trigger here is deliberately realistic: tail has to buffer its entire input before it can print “the last lines,” because a stream of raw zero bytes contains no newlines to split on — the same unbounded-buffering shape as a naive “parse the whole response into memory” bug in a real app:

docker run --name oom-test --memory=20m alpine sh -c "cat /dev/zero | head -c 200m | tail"

Killed

docker inspect --format='OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' oom-test

OOMKilled=true ExitCode=137

Exit code 137 is 128 + 9 — SIGKILL. The Linux kernel’s OOM killer, not Docker itself, enforced the --memory=20m cgroup ceiling the moment tail’s buffer tried to exceed it. There’s no graceful degradation; docker inspect is just reporting what the kernel already decided. It’s specifically tail’s buffering that causes this — swap it for a streaming command like head -c 200m > /dev/null (no buffering, no per-byte memory growth) against the same 20m limit and nothing gets killed at all: the 200MB of data was never the problem, holding it all in memory at once was.

docker run -d \
  --memory="512m" --memory-reservation="256m" \
  --cpus="1.5" --pids-limit=100 \
  myapp:1.0

--memory-reservation is a soft scheduling hint, not a hard cap — only --memory can trigger what just happened above. --pids-limit caps process count per container, the standard defense against a fork bomb taking down the whole host.

Logging: Real Rotation on Disk

The default json-file driver has no rotation unless you configure it — logs grow forever. Set max-size/max-file and watch the files actually cap out:

docker run -d --name log-test --log-driver json-file \
  --log-opt max-size=1k --log-opt max-file=3 \
  alpine sh -c 'i=0; while true; do echo "line-$i-[padding to force rotation]"; i=$((i+1)); sleep 0.05; done'

sleep 5
docker run --rm -v /var/lib/docker/containers:/logs alpine \
  sh -c "ls -la /logs/$(docker inspect -f '{{.Id}}' log-test)/ | grep json.log"

-rw-r-----    1 root     root           607 Sep 16 12:56 <id>-json.log
-rw-r-----    1 root     root          1063 Sep 16 12:56 <id>-json.log.1
-rw-r-----    1 root     root          1064 Sep 16 12:56 <id>-json.log.2

Exactly max-file=3 files, each capped near max-size=1k — the active log plus two rotated backups (.1, .2), older ones dropped. Without these two flags, that active file grows unbounded until the disk fills.

// /etc/docker/daemon.json — apply the same limits to every container by default
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

For centralized logging, swap --log-driver to awslogs (CloudWatch), splunk, or run a fluentd/fluentbit sidecar — the rotation problem this section demonstrates is specific to json-file writing to local disk.

Private Registries: A Real Authenticated Push and Pull

Docker Hub is one registry among many — any host running the open-source registry:2 image is a real one. This section runs an actual private registry, requiring real credentials, entirely on localhost:

docker run -d --name local-registry -p 5500:5000 \
  -v "$(pwd)/auth:/auth" \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
  registry:2

docker tag myapp:1.0 localhost:5500/myapp:1.0
docker push localhost:5500/myapp:1.0

The push refers to repository [localhost:5500/myapp]
...
no basic auth credentials

No credentials, no push — this registry actually enforces its htpasswd file, unlike pushing to a registry with no auth configured at all.

echo "testpass123" | docker login localhost:5500 -u testuser --password-stdin
docker push localhost:5500/myapp:1.0

Login Succeeded
The push refers to repository [localhost:5500/myapp]
1.0: digest: sha256:d2966681216d478f81d49521a0bf2bddba167772f5348b6a256c9bb6f16a00b0 size: 1571

docker rmi localhost:5500/myapp:1.0
docker pull localhost:5500/myapp:1.0

1.0: Pulling from myapp
Digest: sha256:d2966681216d478f81d49521a0bf2bddba167772f5348b6a256c9bb6f16a00b0
Status: Downloaded newer image for localhost:5500/myapp:1.0

The exact same digest on push and pull confirms the round trip — this isn’t a different image that happens to share a tag. AWS ECR, GHCR, and Docker Hub’s private repos all follow this identical login → tag → push/pull shape; only the login step’s credential source changes (aws ecr get-login-password, a GITHUB_TOKEN, or a Docker Hub password).

Docker Buildx: Real Multi-Platform Images

docker buildx create --name lessonbuilder --driver docker-container --driver-opt network=host --use
docker buildx build --platform linux/amd64,linux/arm64 \
  -t localhost:5500/myapp:multi --push .
docker buildx imagetools inspect localhost:5500/myapp:multi

Name:      localhost:5500/myapp:multi
MediaType: application/vnd.oci.image.index.v1+json

Manifests:
  Name:       localhost:5500/myapp:multi@sha256:3d8a2c2f...
  Platform:   linux/amd64

  Name:       localhost:5500/myapp:multi@sha256:0958e00c...
  Platform:   linux/arm64

One tag, one manifest list, two real platform-specific images inside it — docker pull on an amd64 host and an Apple Silicon or AWS Graviton host each transparently get the matching one. --driver-opt network=host matters here: buildx’s default builder runs in its own container with its own network namespace, which can’t resolve localhost:5500 as this host’s localhost without it.

BuildKit Cache Mounts: A Measured Rebuild Speedup

Multi-stage builds (lesson 7) separate build and runtime images, but every npm install still re-downloads from scratch whenever the layer above it changes — Docker’s layer cache only helps when nothing upstream changed. A --mount=type=cache persists a directory across builds, independent of layer caching:

# without a cache mount — every invalidated layer re-downloads everything
RUN npm install

# syntax=docker/dockerfile:1
# with a cache mount — npm's own download cache survives the layer being invalidated
RUN --mount=type=cache,target=/root/.npm npm install

Same package.json (10 real dependencies, 90 resolved packages), same layer-busting change (a version bump) forcing npm install to re-run both times:

# no cache mount: package.json changes, npm re-downloads everything
docker build -f Dockerfile.nocache -t cache-demo .

#8 [4/4] RUN npm install
#8 5.712 added 90 packages, and audited 91 packages in 5s
#8 DONE 5.9s

# cache mount: same forced layer re-run, but /root/.npm already has the tarballs
docker build -f Dockerfile.cache -t cache-demo .

#8 [stage-0 4/4] RUN --mount=type=cache,target=/root/.npm npm install
#8 3.241 added 90 packages, and audited 91 packages in 3s
#8 DONE 3.4s

5.9s → 3.4s, roughly 1.7x, from one flag — and that gap widens with real-world dependency trees far larger than this lesson’s 10 packages. The cache mount lives in BuildKit’s own cache storage, not in the image itself, so none of it ends up in the final image layers.

Restart Policies: A Real Capped Restart Loop

docker run -d --name crash-test --restart on-failure:5 \
  alpine sh -c "echo dying; exit 1"

sleep 8
docker ps -a --filter name=crash-test --format "table {{.Names}}\t{{.Status}}"
docker inspect --format='RestartCount={{.RestartCount}}' crash-test

NAMES        STATUS
crash-test   Exited (1) 7 seconds ago
RestartCount=5

on-failure:5 is a real ceiling, not a description — Docker restarted this exact container five times, then genuinely stopped trying and left it exited. Without the :5 limit, always or a bare on-failure would keep restarting it forever.

Policy	Behavior
`no`	Never restart (default)
`on-failure[:N]`	Restart on non-zero exit, optionally capped at N attempts
`always`	Always restart, including after a Docker daemon restart
`unless-stopped`	Like `always`, but not if you manually stopped it

Zero-Downtime Deployment: The Real Sequence

docker run -d --name myapp-old -p 3001:3000 -e APP_VERSION=v1 myapp:1.0
curl -s http://localhost:3001/

hello from v1

docker run -d --name myapp-new -p 3002:3000 -e APP_VERSION=v2 myapp:2.0

# poll Docker's own health status — not just "is it running"
until [ "$(docker inspect --format='{{.State.Health.Status}}' myapp-new)" = "healthy" ]; do
  sleep 1
done
docker inspect --format='myapp-new health={{.State.Health.Status}}' myapp-new

myapp-new health=healthy

curl -s http://localhost:3002/
docker stop myapp-old && docker rm myapp-old
docker rename myapp-new myapp-old
docker ps --filter name=myapp --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

hello from v2
NAMES       STATUS                    PORTS
myapp-old   Up 27 seconds (healthy)   0.0.0.0:3002->3000/tcp

The old container never stopped serving v1 until the new one had genuinely passed its healthcheck — not just started. A load balancer or reverse proxy in front would make the port switch invisible to clients; Docker Swarm and Kubernetes automate this exact sequence natively.

CI/CD Pipeline

# .github/workflows/docker.yml
name: Build and Push
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

cache-from/cache-to: type=gha is the CI equivalent of this lesson’s --mount=type=cache — GitHub Actions’ own cache backend instead of a local one, same underlying BuildKit mechanism.

Practice

1. Change the OOM demo's --memory limit to 200m instead of 20m and predict what happens before running it.

tail’s own output is piped to /dev/null here — keeping the exact same buffering behavior that triggers the OOM, without dumping 200MB of zero bytes into your terminal if it survives:

docker run --name oom-test2 --memory=200m alpine sh -c "cat /dev/zero | head -c 200m | tail > /dev/null"
docker inspect --format='OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' oom-test2
docker rm oom-test2

OOMKilled=false ExitCode=0

tail has to buffer roughly the full 200MB stream before it can find “the last lines” — that buffer fits under a 200m cgroup limit but not under 20m. The lesson’s demo used 20m specifically because it’s well under what tail’s buffering actually needs, not because “200MB of data” is inherently too much for any container.

2. Push to the local registry a second time with the same tag and unchanged image. What does Docker report?

docker push localhost:5500/myapp:1.0

The push refers to repository [localhost:5500/myapp]
f3b40b0cdb1c: Layer already exists
82140d9a70a7: Layer already exists
449bb0c8c449: Layer already exists
b263e6b8ff11: Layer already exists
0b1f26057bd0: Layer already exists
08000c18d16d: Layer already exists
1.0: digest: sha256:d2966681216d478f81d49521a0bf2bddba167772f5348b6a256c9bb6f16a00b0 size: 1571

Every layer reports already exists rather than re-uploading — Docker computes each layer’s digest client-side and skips any the registry already has, whether that registry is Docker Hub or your own.

3. Set --restart on-failure:5 but have the container exit with code 0 instead of 1. Does it still restart?

docker run -d --name exit-clean --restart on-failure:5 alpine echo done
sleep 3
docker ps -a --filter name=exit-clean --format "table {{.Names}}\t{{.Status}}"
docker inspect --format='RestartCount={{.RestartCount}}' exit-clean
docker rm exit-clean

NAMES         STATUS
exit-clean    Exited (0) 8 seconds ago
RestartCount=0

on-failure only restarts on a non-zero exit code — a clean exit means the process finished successfully on purpose, and Docker leaves it stopped.

4. During the zero-downtime sequence, curl the old container's port while the new one is still starting (before it's healthy). Does the old one still respond?

Yes — myapp-old on port 3001 keeps answering normally the entire time myapp-new is starting and being health-polled, because nothing in this lesson’s sequence touches myapp-old until after myapp-new reports healthy. That ordering — never stop the old one until the new one is proven — is the entire point of the pattern, not an accident of timing.

5. Remove the cache mount's target=/root/.npm path and mount it at a wrong path like /root/.wrong-cache instead. Rebuild twice, bumping package.json in between, and compare.

docker build -f Dockerfile.wrongcache -t cache-demo-wrong1 .
# bump package.json's version field, then:
docker build -f Dockerfile.wrongcache -t cache-demo-wrong2 .

#8 [stage-0 4/4] RUN --mount=type=cache,target=/root/.wrong-cache npm install
#8 4.176 added 90 packages, and audited 91 packages in 4s
#8 DONE 4.4s

#8 [stage-0 4/4] RUN --mount=type=cache,target=/root/.wrong-cache npm install
#8 4.133 added 90 packages, and audited 91 packages in 4s
#8 DONE 4.3s

No improvement between the two runs — npm never actually writes its download cache to /root/.wrong-cache, so the mount just persists an empty, unused directory every time. The speedup earlier in this lesson only exists because the mount target matches the exact path the tool being run actually caches to (npm config get cache reports /root/.npm by default); an arbitrary path gains nothing, no matter how many times you rebuild.