Graceful Shutdown and Health Checks

// server-graceful.js (added route)
import express from "express";

const app = express();
let shuttingDown = false;

app.get("/healthz", (req, res) => res.json({ status: "ok" }));

app.get("/readyz", (req, res) => {
  if (shuttingDown) {
    res.status(503).json({ status: "shutting down" });
    return;
  }
  res.json({ status: "ready" });
});

app.get("/slow", async (req, res) => {
  console.log("slow request started");
  await new Promise((resolve) => setTimeout(resolve, 3000));
  console.log("slow request finished");
  res.json({ done: true });
});

app.get("/very-slow", async (req, res) => { // new
  console.log("very-slow request started"); // new
  await new Promise((resolve) => setTimeout(resolve, 5000)); // new
  console.log("very-slow request finished"); // new
  res.json({ done: true }); // new
}); // new

const server = app.listen(3000, () => console.log("listening on http://localhost:3000"));

process.on("SIGTERM", () => {
  console.log("SIGTERM received: failing readiness, then draining");
  shuttingDown = true;
  setTimeout(() => {
    server.close(() => {
      console.log("all connections drained, exiting");
      process.exit(0);
    });
  }, 5000);
});

graceful-demo already exited at the end of the main example, so this route needs a fresh container running the edited file before it can be tested:

$ docker rm -f graceful-demo
graceful-demo
$ docker run -d --name graceful-demo --network node-lab -v "$(pwd)":/app -w /app node:26 node server-graceful.js
123297c1cce5361a55ccbca4f81bc4979e014c249d6987719b4c57e1870eec1f
$ until docker logs graceful-demo 2>&1 | grep -q listening; do sleep 0.2; done
$ docker exec curl-side curl -s -S -i --max-time 10 http://graceful-demo:3000/very-slow &
$ sleep 0.5 && docker stop -t 3 graceful-demo
curl: (52) Empty reply from server
graceful-demo
$ wait

No — the graceful handler still only has however long the grace period gives it. A 3-second docker stop -t 3 grace period doesn’t even cover the 5-second pre-close delay on its own, let alone the in-flight request behind it; it gets SIGKILLed before server.close() is ever called, the exact same way the abrupt server failed earlier. “Graceful” means attempting to drain, not an unlimited amount of time to do it — the grace period has to cover the pre-close delay and the slowest request the app expects to actually finish in.

// server-graceful.js (no pre-close delay)
import express from "express";

const app = express();
let shuttingDown = false;

app.get("/healthz", (req, res) => res.json({ status: "ok" }));

app.get("/readyz", (req, res) => {
  if (shuttingDown) {
    res.status(503).json({ status: "shutting down" });
    return;
  }
  res.json({ status: "ready" });
});

app.get("/slow", async (req, res) => {
  console.log("slow request started");
  await new Promise((resolve) => setTimeout(resolve, 3000));
  console.log("slow request finished");
  res.json({ done: true });
});

const server = app.listen(3000, () => console.log("listening on http://localhost:3000"));

process.on("SIGTERM", () => {
  console.log("SIGTERM received, draining connections");
  shuttingDown = true;
  server.close(() => {
    console.log("all connections drained, exiting");
    process.exit(0);
  });
});

health-demo from the main example already exited, so this edited handler needs its own fresh container too:

$ docker rm -f health-demo
health-demo
$ docker run -d --name health-demo --network node-lab -v "$(pwd)":/app -w /app node:26 node server-graceful.js
02a48f64782e9b82e3aacf91003f6e512c2e1367d1e2193f7118010ed8f6032f
$ until docker logs health-demo 2>&1 | grep -q listening; do sleep 0.2; done
$ docker kill --signal=TERM health-demo
health-demo
$ docker exec curl-side curl -s -S -i -m 1 http://health-demo:3000/readyz
curl: (28) Resolving timed out after 1001 milliseconds

$ docker logs health-demo
listening on http://localhost:3000
SIGTERM received, draining connections
all connections drained, exiting

Not a connection timeout — a DNS resolution timeout. With nothing in flight to drain, server.close()’s callback fires on the very next tick, so process.exit(0) runs within a millisecond or two of the SIGTERM handler starting. By the time the curl in the other container even starts resolving health-demo, the container has already exited and Docker’s embedded DNS has nothing left to answer with — the lookup itself hangs until -m 1’s deadline, rather than a GET ever reaching a socket. The entire point of the original 5-second setTimeout before server.close() was to hold that window open long enough for a readiness probe to actually see {"status":"shutting down"} before the container disappears; removing it doesn’t just change how a new connection gets refused, it removes the window entirely.

This needs the regular handler again (Practice 2’s edit removed the delay), running in yet another fresh container, since health-demo is still the exited container from Practice 2 — the file below is identical to the main example’s server-graceful.js, nothing changed:

// server-graceful.js (regular handler again)
import express from "express";

const app = express();
let shuttingDown = false;

app.get("/healthz", (req, res) => res.json({ status: "ok" }));

app.get("/readyz", (req, res) => {
  if (shuttingDown) {
    res.status(503).json({ status: "shutting down" });
    return;
  }
  res.json({ status: "ready" });
});

app.get("/slow", async (req, res) => {
  console.log("slow request started");
  await new Promise((resolve) => setTimeout(resolve, 3000));
  console.log("slow request finished");
  res.json({ done: true });
});

const server = app.listen(3000, () => console.log("listening on http://localhost:3000"));

process.on("SIGTERM", () => {
  console.log("SIGTERM received: failing readiness, then draining");
  shuttingDown = true;
  setTimeout(() => {
    server.close(() => {
      console.log("all connections drained, exiting");
      process.exit(0);
    });
  }, 5000);
});

$ docker rm -f health-demo
health-demo
$ docker run -d --name health-demo --network node-lab -v "$(pwd)":/app -w /app node:26 node server-graceful.js
77faff3c08e7e80d567cce6d3af91e6ce5bb64c1fe784f45d425b1ada149400b
$ until docker logs health-demo 2>&1 | grep -q listening; do sleep 0.2; done
$ date +%s.%N
1793339023.555437600
$ docker kill --signal=TERM health-demo
health-demo
$ docker wait health-demo
0
$ date +%s.%N
1793339030.197058600

The two timestamps above are a little over six seconds apart — the setTimeout(..., 5000) delay before server.close() runs regardless of whether anything is actually in flight to drain, plus some real overhead in docker kill itself actually delivering the signal and in docker wait reporting back. An idle server still pays the full pre-close delay; it just has nothing to wait for once close() is called, so the exit code 0 comes back as soon as that delay and the now-trivial drain are done. The exact figure drifts a little run to run, and never drops anywhere close to the 5-second delay alone, because docker kill and docker wait each add real overhead of their own on top of it.

// server-graceful.js (added a stuck route and a force-exit timeout)
import express from "express";

const app = express();
let shuttingDown = false;

app.get("/healthz", (req, res) => res.json({ status: "ok" }));

app.get("/readyz", (req, res) => {
  if (shuttingDown) {
    res.status(503).json({ status: "shutting down" });
    return;
  }
  res.json({ status: "ready" });
});

app.get("/slow", async (req, res) => {
  console.log("slow request started");
  await new Promise((resolve) => setTimeout(resolve, 3000));
  console.log("slow request finished");
  res.json({ done: true });
});

app.get("/stuck", async (req, res) => { // new
  console.log("stuck request started"); // new
  await new Promise(() => {}); // new — never resolves, simulates a dead upstream call
}); // new

const server = app.listen(3000, () => console.log("listening on http://localhost:3000"));

process.on("SIGTERM", () => {
  console.log("SIGTERM received: failing readiness, then draining");
  shuttingDown = true;
  const forceExit = setTimeout(() => { // new
    console.log("drain timeout exceeded, forcing exit"); // new
    process.exit(1); // new
  }, 10000); // new
  setTimeout(() => {
    server.close(() => {
      clearTimeout(forceExit); // new
      console.log("all connections drained, exiting");
      process.exit(0);
    });
  }, 5000);
});

graceful-demo already exited at the end of Practice 1, so this needs a fresh container running the edited file again:

$ docker rm -f graceful-demo
graceful-demo
$ docker run -d --name graceful-demo --network node-lab -v "$(pwd)":/app -w /app node:26 node server-graceful.js
93293e0ffc27b09d0eacd61e8a8d33be9fe68a32ee22ecf30c4f9bb736687203
$ until docker logs graceful-demo 2>&1 | grep -q listening; do sleep 0.2; done
$ docker exec curl-side curl -s -S -i --max-time 15 http://graceful-demo:3000/stuck &
$ sleep 0.5 && docker stop -t 15 graceful-demo
curl: (52) Empty reply from server
graceful-demo
$ wait

$ docker logs graceful-demo
listening on http://localhost:3000
stuck request started
SIGTERM received: failing readiness, then draining
drain timeout exceeded, forcing exit
$ docker inspect graceful-demo --format '{{.State.ExitCode}}'
1

A connection that somehow never finishes (a stuck upstream call, a client that opened a connection and went silent) would otherwise leave server.close()’s callback waiting forever, since close() only calls back once every connection has ended on its own — "all connections drained, exiting" never prints here. Ten seconds after SIGTERM, the independent forceExit timer fires instead, logs "drain timeout exceeded, forcing exit", and calls process.exit(1) — a different exit code from the clean path’s 0, so an orchestrator (or docker inspect) can tell “drained in time” apart from “force-killed by its own code” after the fact. clearTimeout(forceExit) on the normal path only matters when the clean close actually wins that race.