MTTR Isn't a Guess — Compute It From Alertmanager's Own Timestamps

Alertmanager already sends every piece of data needed to compute MTTR. Nobody has to manually time an incident — the notification itself carries startsAt, and a resolved notification adds endsAt. The only thing missing is somewhere to keep them.

Persisting what was already being thrown away

The webhook receiver from the alerting lesson printed each notification and discarded it. A five-line change keeps it instead:

# webhook_receiver.py (excerpt)
for alert in payload.get("alerts", []):
    record = {
        "received_at": time.time(),
        "status": alert.get("status"),
        "alertname": alert.get("labels", {}).get("alertname"),
        "startsAt": alert.get("startsAt"),
        "endsAt": alert.get("endsAt"),
        "fingerprint": alert.get("fingerprint"),
    }
    with open("/data/incident_log.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Two real incidents

Triggering the same HighErrorRate alert from the alerting lesson twice, a few minutes apart, produces a genuine four-line history — one firing and one resolved record per incident:

$ cat data/incident_log.jsonl
{"received_at": 1789376566.6387277, "status": "firing", "alertname": "HighErrorRate", "startsAt": "2026-09-14T09:02:41.566Z", "endsAt": "0001-01-01T00:00:00Z", "fingerprint": "2939db0640e00b01"}
{"received_at": 1789376606.6224978, "status": "resolved", "alertname": "HighErrorRate", "startsAt": "2026-09-14T09:02:41.566Z", "endsAt": "2026-09-14T09:03:21.566Z", "fingerprint": "2939db0640e00b01"}
{"received_at": 1789376666.5289724, "status": "firing", "alertname": "HighErrorRate", "startsAt": "2026-09-14T09:04:16.566Z", "endsAt": "0001-01-01T00:00:00Z", "fingerprint": "2939db0640e00b01"}
{"received_at": 1789376706.5243535, "status": "resolved", "alertname": "HighErrorRate", "startsAt": "2026-09-14T09:04:16.566Z", "endsAt": "2026-09-14T09:05:01.566Z", "fingerprint": "2939db0640e00b01"}

The unresolved firing records carry Alertmanager’s placeholder zero-time (0001-01-01T00:00:00Z) in endsAt — Go’s zero value for a timestamp, and the signal that this incident hadn’t ended yet at the moment this notification was sent.

Computing MTTR from that log

for line in open("data/incident_log.jsonl"):
    record = json.loads(line)
    starts = datetime.fromisoformat(record["startsAt"].replace("Z", "+00:00"))
    if record["status"] == "firing":
        lag = record["received_at"] - starts.timestamp()
        print(f"{record['fingerprint']} notified {lag:.1f}s after startsAt")
    elif record["status"] == "resolved":
        ends = datetime.fromisoformat(record["endsAt"].replace("Z", "+00:00"))
        duration_s = (ends - starts).total_seconds()
        durations.append(duration_s)
        print(f"{record['fingerprint']} resolved: {duration_s:.0f}s duration")

$ python compute_mttr.py
2939db0640e00b01 notified 5.1s after startsAt
2939db0640e00b01 resolved: 40s duration
2939db0640e00b01 notified 10.0s after startsAt
2939db0640e00b01 resolved: 45s duration

incidents: 2
mean notification lag: 7.5s
MTTR: 42.5s

A real, computed 42.5-second MTTR, from two real incidents, entirely derived from data Alertmanager was sending anyway.

The gotcha: 42.5 seconds is not how long anything was actually broken

Both incidents were triggered by roughly two seconds of real /work traffic — 150 requests fired in a tight loop. The measured 40s and 45s durations are ten to twenty times longer than that. Nothing is wrong with the math; the inflation comes from two places this lesson’s own earlier config already explains:

• for: 20s delays when the alert is allowed to fire at all, but startsAt is set to the moment the condition first crossed the threshold — not when for: finished — so that part doesn’t inflate endsAt - startsAt. It inflates the notification lag instead (the 5.1s and 10.0s above are each roughly one for: cycle’s worth of evaluation granularity plus group_wait).
• rate(app_errors_total[1m])’s one-minute window is the real cause of the long duration. The burst’s error count stays inside that window, keeping the computed rate above the 0.05 threshold, for most of a minute after the actual requests finished — the same behavior demonstrated directly with docker restart in the rate() lesson. endsAt isn’t set until the rate genuinely drops back under threshold, which is a function of the window size, not of how long the real problem lasted.

A real MTTR pipeline built this way is measuring “how long the alerting rule considered this a problem,” which correlates with, but overstates, the true incident length by however wide the rule’s own evaluation window is.

Practice

1. Add a third real incident (repeat the burst-and-wait cycle once more) and recompute MTTR with all three durations averaged. Confirm the log now has six lines.

   Solution

   $ wc -l data/incident_log.jsonl
   6 data/incident_log.jsonl
   $ python compute_mttr.py
   2939db0640e00b01 notified 5.1s after startsAt
   2939db0640e00b01 resolved: 40s duration
   2939db0640e00b01 notified 10.0s after startsAt
   2939db0640e00b01 resolved: 45s duration
   2939db0640e00b01 notified 9.9s after startsAt
   2939db0640e00b01 resolved: 40s duration
   
   incidents: 3
   mean notification lag: 8.3s
   MTTR: 41.7s

   A third real incident (also a 40s duration) pulled the mean from 42.5s down to 41.7s — with only two or three samples, one unusually long or short incident swings MTTR a lot, which is exactly why a production MTTR needs enough real incidents accumulated before the number means anything.

2. Compute the max duration instead of the mean, using the same durations list already built by compute_mttr.py.

   Solution

   print(f"max duration: {max(durations):.0f}s")

   max duration: 45s

   A max is far more sensitive to a single bad incident than a mean — useful for spotting “did anything take unusually long” separately from the average.

3. Change the alert rule’s expression from rate(app_errors_total[1m]) to rate(app_errors_total[30s]), restart Prometheus, and trigger one more real incident. Compare its measured duration against the 40s/45s durations from the [1m] window.

   Solution

   Running both a [1m]-window rule and a [30s]-window rule side by side against the exact same burst produces two very different durations for the same real incident:

   {"alertname": "HighErrorRateNarrowWindow", "startsAt": "...09:10:01.566Z", "endsAt": "...09:10:16.566Z"}
   {"alertname": "HighErrorRate",             "startsAt": "...09:10:01.566Z", "endsAt": "...09:10:46.566Z"}

   15 seconds for the [30s] window versus 45 seconds for the [1m] window — identical startsAt, identical underlying burst, and a 3x difference in measured duration purely from the window size in the PromQL expression.

4. Add resolved_count and firing_count tallies to compute_mttr.py by counting record["status"] values directly, and print both alongside the existing MTTR output — a basic sanity check that every firing eventually got a matching resolved.

   Solution

   from collections import Counter
   statuses = Counter()
   for line in open("data/incident_log.jsonl"):
       statuses[json.loads(line)["status"]] += 1
   print(statuses)

   Counter({'firing': 2, 'resolved': 2})

   Equal counts confirm every incident that started also ended in this log — a mismatch (more firing than resolved) would mean an incident is still open, or that the receiver’s log was read mid-incident.