A Firing Alert Doesn't Mean Anyone Was Notified

Prometheus decides an alert should fire. Alertmanager is the thing that actually tells a human. Those are two different programs, connected by one line of config — and that line can be wrong without either program complaining.

The rule: rate() crossing a threshold, sustained

.scratch/alert_rules.yml defines one alerting rule against the same metrics-app used in the last two lessons:

groups:
  - name: demo-app
    rules:
      - alert: HighErrorRate
        expr: rate(app_errors_total[1m]) > 0.05
        for: 20s
        labels:
          severity: warning
        annotations:
          summary: "High error rate on {{ $labels.job }}"

for: 20s means the expression has to stay true for 20 full seconds before this alert is allowed to fire — a single noisy scrape isn’t enough. Prometheus loads it straight from prometheus.yml’s rule_files: list, next to a rule_manager startup log line:

$ docker logs prometheus | grep "rule manager"
time=2026-09-14T07:14:13.426Z level=INFO source=manager.go:211 msg="Starting rule manager..." component="rule manager"

Watching pending become firing

Hitting /work 150 times produces a real burst of errors (the endpoint has an intentional ~10% failure rate):

$ for i in $(seq 1 150); do curl -s http://localhost:8000/work > /dev/null; done
$ curl -s http://localhost:8000/metrics | grep _total
app_requests_total 150
app_errors_total 20

Seconds later, Prometheus’s own rules API shows the alert has noticed — but it hasn’t fired yet:

$ curl -s http://localhost:9090/api/v1/rules
{"status":"success","data":{"groups":[{"name":"demo-app","rules":[{
  "state":"pending",
  "name":"HighErrorRate",
  "duration":20,
  "alerts":[{"state":"pending","activeAt":"2026-09-14T07:15:36.566932265Z","value":"3.6366281184086124e-01"}]
}]}]}}

value is the live rate() result — 0.36 errors/sec, well past the 0.05 threshold — and activeAt is the instant it first crossed. Twenty real seconds later, with no other action taken, the same endpoint shows the transition:

$ curl -s http://localhost:9090/api/v1/rules
{"status":"success","data":{"groups":[{"name":"demo-app","rules":[{
  "state":"firing",
  "alerts":[{"state":"firing","activeAt":"2026-09-14T07:15:36.566932265Z","value":"3.467292212200563e-01"}]
}]}]}}

activeAt didn’t change — Prometheus is still counting from the original crossing — only state did. This whole progression happened inside Prometheus, with zero network calls to anywhere else.

Delivery is a second, separate step

For anyone to actually see this alert, Prometheus has to push it to Alertmanager, which then routes it to a receiver. .scratch/alertmanager.yml routes everything to one webhook:

route:
  receiver: "webhook-receiver"
  group_wait: 5s
  group_interval: 10s
  repeat_interval: 1h

receivers:
  - name: "webhook-receiver"
    webhook_configs:
      - url: "http://webhook-receiver:9095/"
        send_resolved: true

The receiver on the other end is a 20-line Python HTTP server (.scratch/webhook_receiver.py) that just logs whatever POST body it gets. With the correct address wired into prometheus.yml (alerting.alertmanagers pointing at alertmanager:9093), the same firing alert from above shows up on the receiver, for real:

$ docker logs webhook-receiver
RECEIVED alerts=['HighErrorRate'] statuses=['firing']

The bug: one wrong port, and delivery vanishes

Change nothing except the port Prometheus sends alerts to:

# prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9199"]   # should be 9093

Restart Prometheus with that config and run the exact same burst again. Prometheus’s rule engine doesn’t care that the address is wrong — it fires right on schedule:

$ curl -s http://localhost:9090/api/v1/rules
{"status":"success","data":{"groups":[{"name":"demo-app","rules":[{
  "state":"firing",
  "alerts":[{"state":"firing","activeAt":"2026-09-14T08:45:56.566932265Z","value":"2.1016838200530966e-01"}]
}]}]}}

But check where Prometheus actually thinks Alertmanager lives, and what Alertmanager itself received:

$ curl -s http://localhost:9090/api/v1/alertmanagers
{"status":"success","data":{"activeAlertmanagers":[{"url":"http://alertmanager:9199/api/v2/alerts"}],"droppedAlertmanagers":[]}}

$ curl -s http://localhost:9093/api/v2/alerts
[]

$ docker logs webhook-receiver
RECEIVED alerts=['HighErrorRate'] statuses=['firing']

One line in the webhook log — the same one from before the bug was introduced. Nothing new arrived. /api/v1/alertmanagers confirms Prometheus is dutifully sending to :9199, a port nothing is listening on. Prometheus does log the failure — but only to its own container logs, which nobody watching Alertmanager or the webhook receiver would ever see:

$ docker logs prometheus | grep sendloop
time=2026-09-14T08:52:26.565Z level=ERROR source=sendloop.go:239 msg="Error sending alerts" component=notifier alertmanager=http://alertmanager:9199/api/v2/alerts count=1 err="Post \"http://alertmanager:9199/api/v2/alerts\": dial tcp 172.19.0.5:9199: connect: connection refused"

The alert fired. The failure to deliver it is real and logged — just in a place a paged engineer, staring at a silent phone, has no reason to be looking.

The fix

Put the port back and repeat the burst:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

$ curl -s http://localhost:9090/api/v1/alertmanagers
{"status":"success","data":{"activeAlertmanagers":[{"url":"http://alertmanager:9093/api/v2/alerts"}],"droppedAlertmanagers":[]}}

$ curl -s http://localhost:9093/api/v2/alerts
[{"labels":{"alertname":"HighErrorRate","instance":"metrics-app:8000","job":"demo-app","severity":"warning"},
  "annotations":{"summary":"High error rate on demo-app"},
  "status":{"state":"active"},
  "startsAt":"2026-09-14T08:47:51.566Z",
  "fingerprint":"2939db0640e00b01"}]

$ docker logs webhook-receiver
RECEIVED alerts=['HighErrorRate'] statuses=['firing']
RECEIVED alerts=['HighErrorRate'] statuses=['firing']

A second line, and a real alert object — fingerprint, startsAt, the exact annotations from the rule — sitting in Alertmanager’s own API. Same rule, same burst, same for: duration; the only thing that changed was whether the pipe between the two programs was correctly addressed.

Runbooks travel with the alert

Alertmanager forwards every annotation on the rule to the receiver verbatim, which is why a runbook link belongs there instead of in a separate wiki a paged engineer has to go find:

annotations:
  summary: "High error rate on {{ $labels.job }}"
  runbook_url: "https://runbooks.internal/high-error-rate"

Whatever shows up in Slack, PagerDuty, or this lesson’s webhook log carries that URL alongside the alert name — the fix-it steps are one field away, not a search.

Practice

1. Change for: from 20s to 45s, restart Prometheus, and repeat the 150-request burst. Poll /api/v1/rules every few seconds and confirm the alert now spends longer in pending before reaching firing — measure the real gap between the activeAt timestamp and the moment state flips.

   Solution

   $ curl -s "http://localhost:9090/api/v1/rules" | grep -o '"state":"[a-z]*"'
   "state":"pending"
   # ... wait, poll again ...
   $ curl -s "http://localhost:9090/api/v1/rules" | grep -o '"state":"[a-z]*"'
   "state":"pending"
   # still pending past 20s now — for: 45s means it needs 45 continuous seconds

   The activeAt timestamp stays fixed at the first crossing; only the amount of real wall time before state becomes firing changes, directly matching the new for: value.

2. Point alertmanagers.static_configs.targets at a hostname that doesn’t resolve at all (e.g. ["nonexistent-host:9093"]) instead of a wrong port on a real host. Restart Prometheus and check docker logs prometheus for how this failure mode differs from the wrong-port case.

   Solution

   $ docker logs prometheus | grep -i alertmanager
   time=2026-09-14T08:51:04.141Z level=ERROR source=sendloop.go:239 msg="Error sending alerts" component=notifier alertmanager=http://nonexistent-host:9093/api/v2/alerts count=1 err="Post \"http://nonexistent-host:9093/api/v2/alerts\": dial tcp: lookup nonexistent-host on 127.0.0.11:53: no such host"

   Both failure modes log the same level=ERROR source=sendloop.go line — the difference is only in the wrapped err text: a bad hostname fails at DNS resolution (no such host), while the wrong-port case earlier in this lesson fails one step later, at the TCP handshake against a real, reachable host (connect: connection refused). Either way the failure lands only in Prometheus’s own container logs, never in Alertmanager or the receiver.

3. Add a second rule, alert: ErrorRateCritical, with expr: rate(app_errors_total[1m]) > 0.2 and for: 10s, alongside the existing HighErrorRate rule. Run a burst large enough to cross both thresholds and check /api/v1/rules for both alert states at once.

   Solution

   groups:
     - name: demo-app
       rules:
         - alert: HighErrorRate
           expr: rate(app_errors_total[1m]) > 0.05
           for: 20s
         - alert: ErrorRateCritical
           expr: rate(app_errors_total[1m]) > 0.2
           for: 10s

   Both rules evaluate independently against the same underlying metric — a large enough burst puts both into pending at the same time, but ErrorRateCritical’s shorter for: means it reaches firing first.

4. Set send_resolved: false in the webhook receiver config, restart Alertmanager, let the current alert resolve (error rate drops back below threshold and the window in rate() ages out), and confirm the webhook receiver never logs a resolved status — compare against the default (send_resolved: true) behavior.

   Solution

   With send_resolved: false, docker logs webhook-receiver shows only the original firing line, permanently — Alertmanager suppresses the follow-up POST it would otherwise send once the alert’s endsAt passes, so a receiver configured this way can only ever tell you an alert started, never that it ended.