What is an error budget, concretely, in PromQL terms?

It's the complement of the SLO target (1 minus the target), expressed as an allowed error fraction — a 99.5% availability SLO allows 0.5% of requests to fail. A recording rule turns that into a live number by comparing the actual error fraction over a window to that 0.5% allowance; this lesson's orders:error_budget_remaining:ratio12h rule does exactly that over a 12-hour window, and goes negative the moment the real error fraction exceeds 0.5%.

Why does a burn-rate alert need two time windows (long and short) instead of one?

The long window (1h in the real SRE workbook pattern, scaled to 60s in this lesson) confirms the problem is sustained rather than a single noisy scrape; the short window (5m scaled to 5s) makes the alert clear quickly once the problem actually stops, instead of waiting for the slower long window to fully flush a resolved incident out of its average. This lesson's alert used an AND of both conditions, and resolved the moment the short window recovered even while the 60-second (1h-scaled, long) window was still showing an elevated rate.

What does a burn rate of 14.4 actually mean?

14.4 is a multiplier on the sustainable error rate, not a percentage by itself. It comes from deciding how much of a 30-day error budget a window is allowed to consume if the condition holds for the window's full duration — for a 1-hour window, allowing 2% of the 30-day budget to burn in that one hour gives 0.02 / (1h / 720h) = 0.02 x 720 = 14.4. A burn rate reading of 14.4 means the current error rate is 14.4 times the one steady rate that would exactly exhaust the 30-day budget over 30 days.

Why did the fast-burn alert resolve while its long-window burn rate was still over 100x?

Because the alert's expression is an AND of both windows, and the short window recovers much faster than the long one. In this lesson's real run, both windows were read in a single query at the same instant right after the fix: the 5-second window had already dropped to 0, while the 60-second (1h-scaled, long) window was still at 126.3x — a 60-second average needs most of 60 seconds of clean traffic before it stops reflecting the spike. The AND meant the alert cleared the instant the short window recovered, without waiting on the slow one.

How do this lesson's scaled windows map to the real SRE workbook windows?

Every window is compressed by the same factor of 60: the real fast-burn pair (1 hour long window, 5 minute short window) becomes 60 seconds and 5 seconds here, the real slow-burn pair (6 hours, 30 minutes) becomes 360 seconds and 30 seconds, and the real 30-day SLO compliance period becomes 12 hours. The burn-rate thresholds themselves (14.4 for fast, 6 for slow) are unchanged, because they're dimensionless ratios derived from what fraction of the 30-day budget each window represents, not from the window's absolute length.

Error Budgets and Multi-Window Burn-Rate Alerts

An SLO target by itself is just a number on a slide. An error budget is that same number turned into something a Prometheus rule can watch, burn down, and alert on in real time.

An app with a success/error counter

app.py is a stdlib-only HTTP server exposing Prometheus text-format metrics, with one addition over earlier monitoring lessons’ apps: a /control endpoint that lets this lesson script a real error spike on demand, instead of waiting for one.

# app.py
import random
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs

lock = threading.Lock()
success_count = 0
error_count = 0
error_rate = 0.002  # healthy baseline: well inside a 0.5% error budget


class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        pass  # keep container logs quiet; state is served over HTTP instead

    def do_GET(self):
        global success_count, error_count
        parsed = urlparse(self.path)
        if parsed.path == "/work":
            with lock:
                failed = random.random() < error_rate
                if failed:
                    error_count += 1
                else:
                    success_count += 1
            self.send_response(500 if failed else 200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(b"error\n" if failed else b"ok\n")
        elif parsed.path == "/metrics":
            with lock:
                s, e = success_count, error_count
            body = (
                "# HELP http_requests_total Total HTTP requests handled, by status\n"
                "# TYPE http_requests_total counter\n"
                f'http_requests_total{{status="success"}} {s}\n'
                f'http_requests_total{{status="error"}} {e}\n'
            ).encode()
            self.send_response(200)
            self.send_header("Content-Type", "text/plain; version=0.0.4")
            self.end_headers()
            self.wfile.write(body)
        else:
            self.send_response(404)
            self.end_headers()

    def do_POST(self):
        global error_rate
        parsed = urlparse(self.path)
        if parsed.path == "/control":
            qs = parse_qs(parsed.query)
            error_rate = float(qs.get("error_rate", [str(error_rate)])[0])
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(f"error_rate={error_rate}\n".encode())
        else:
            self.send_response(404)
            self.end_headers()


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), Handler)
    server.serve_forever()

$ curl -s -X POST "http://localhost:8000/control?error_rate=0.9"
error_rate=0.9

One query parameter is the entire mechanism this lesson uses to turn a healthy service into a failing one and back, on command.

Turning 99.5% into an error budget, and real windows into scaled ones

This lesson’s target: 99.5% of requests succeed, measured over a 30-day compliance window — a 0.5% (0.005) error budget. The Google SRE workbook’s multi-window burn-rate pattern pairs a long window with a short window so an alert both confirms a sustained problem and clears quickly once it’s over. Every real window below is scaled down by the same factor of 60 so this lesson runs in minutes instead of hours or days:

Purpose	Real window(s)	Burn-rate threshold	Scaled window(s)
SLO compliance period	30d	—	12h
Fast burn (page)	1h / 5m	14.4	60s / 5s
Slow burn (ticket)	6h / 30m	6	360s / 30s

14.4 and 6 don’t change when the windows are scaled — they’re multipliers on the sustainable error rate, derived from what fraction of the 30-day budget each window is allowed to consume before paging, which is a dimensionless ratio, not a function of how long a second lasts in this demo. 14.4 specifically comes from allowing 2% of the 30-day budget to burn within a 1-hour window: 0.02 / (1h / 720h) = 14.4.

Compose: the app, Alertmanager, Prometheus, and a webhook receiver

services:
  metrics-app:
    image: python:3.14-slim
    volumes:
      - ./app.py:/app/app.py
    working_dir: /app
    command: python app.py
    ports:
      - "8000:8000"

  webhook-receiver:
    image: python:3.14-slim
    volumes:
      - ./webhook_receiver.py:/app/webhook_receiver.py
    working_dir: /app
    command: python webhook_receiver.py
    ports:
      - "9095:9095"

  alertmanager:
    image: prom/alertmanager:v0.34.1
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
    depends_on:
      - webhook-receiver

  prometheus:
    image: prom/prometheus:v3.15.1
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules.yml:/etc/prometheus/rules.yml
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    depends_on:
      - metrics-app
      - alertmanager

webhook_receiver.py just logs whatever POST body Alertmanager sends it:

# webhook_receiver.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length) or b"{}")
        alerts = [a["labels"]["alertname"] for a in body.get("alerts", [])]
        statuses = [a["status"] for a in body.get("alerts", [])]
        print(f"RECEIVED alerts={alerts} statuses={statuses}", flush=True)
        self.send_response(200)
        self.end_headers()


if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 9095), Handler).serve_forever()

alertmanager.yml routes everything to that receiver, with short timers so this demo doesn’t sit around waiting:

route:
  receiver: "webhook-receiver"
  group_wait: 1s
  group_interval: 5s
  repeat_interval: 1h

receivers:
  - name: "webhook-receiver"
    webhook_configs:
      - url: "http://webhook-receiver:9095/"
        send_resolved: true

group_wait: 1s is how long Alertmanager waits after the first alert in a new group before sending anything, in case more alerts land in the same window to batch together. group_interval: 5s is how long it waits after that before sending an update to an already-notified group — including the resolved update. Both are real seconds here, not scaled: this lesson’s resolved webhook arrives within about 5 seconds of the alert actually clearing, because that’s genuinely how fast this config is set to notify, not because anything else sped it up.

prometheus.yml scrapes every second — far tighter than production, needed here so a 5-second window still has enough samples to compute a rate from:

global:
  scrape_interval: 1s
  evaluation_interval: 5s
rule_files:
  - /etc/prometheus/rules.yml
scrape_configs:
  - job_name: "orders-app"
    static_configs:
      - targets: ["metrics-app:8000"]
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

$ docker compose down -v
$ docker compose up -d 2>&1 | grep -E "Started|Running"
 Container slo-clean-metrics-app-1 Started
 Container slo-clean-webhook-receiver-1 Started
 Container slo-clean-alertmanager-1 Started
 Container slo-clean-prometheus-1 Started
$ docker exec slo-clean-prometheus-1 prometheus --version
prometheus, version 3.15.1 (branch: HEAD, revision: bae3eced0a5369e050e83ea73bc381d7c4b1a719)
  build user:       root@03bc6b0f3ef7
  build date:       20261021-13:31:17
  go version:       go1.27.3
  platform:         linux/amd64
  tags:             netgo,builtinassets
$ docker exec slo-clean-alertmanager-1 alertmanager --version
alertmanager, version 0.34.1 (branch: HEAD, revision: 73c6bfe7393929211294c1954f30d8ed78e4d0ad)
  build user:       root@68a5f2da6323
  build date:       20260917-07:53:02
  go version:       go1.26.8
  platform:         linux/amd64
  tags:             netgo
$ docker exec slo-clean-metrics-app-1 python --version
Python 3.14.10
$ curl -s http://localhost:9090/-/ready
Prometheus Server is Ready.

Every image tag above is pinned; the version strings printed are what this lesson’s output actually came from.

A steady load, from loadgen.py

Everything below needs real, continuous traffic hitting /work — not just the one-off /control calls. loadgen.py is a tight loop with no output of its own; it just keeps a steady request rate running for a fixed number of seconds.

# loadgen.py
import sys
import time
import urllib.request

seconds = int(sys.argv[1]) if len(sys.argv) > 1 else 30
rps = float(sys.argv[2]) if len(sys.argv) > 2 else 20.0
interval = 1.0 / rps
end = time.time() + seconds
while time.time() < end:
    try:
        urllib.request.urlopen("http://localhost:8000/work", timeout=1)
    except Exception:
        pass
    time.sleep(interval)

$ python loadgen.py 1200 20

1200 seconds (20 minutes) at 20 requests/second — long enough to cover every example below and every Practice task, including the slow-burn pair, which needs several real minutes of sustained traffic before its 360-second window moves. Run from the host against the container’s published 8000 port and left running in the background for the rest of this lesson: every http_requests_total count and every recording-rule value from here on is real traffic this one command generated, not something injected directly into Prometheus.

Recording rules: error ratio, burn rate, and a real error budget

rules.yml computes the fast-burn pair’s error ratio and burn rate at both the 5-second (short) and 60-second (1h-scaled, long) windows, a 60-second “headroom” number, and — over the real, scaled 12-hour compliance window — the actual error budget:

groups:
  - name: orders-slo
    interval: 5s
    rules:
      # --- short window (demo 5s stands in for a real 5m) ---
      - record: orders:requests:rate5s
        expr: sum(rate(http_requests_total[5s]))
      - record: orders:errors:rate5s
        expr: sum(rate(http_requests_total{status="error"}[5s]))
      - record: orders:error_ratio:rate5s
        expr: orders:errors:rate5s / orders:requests:rate5s
      - record: orders:burn_rate:ratio5s
        expr: orders:error_ratio:rate5s / 0.005

      # --- long window (demo 60s stands in for a real 1h) ---
      - record: orders:requests:rate60s
        expr: sum(rate(http_requests_total[60s]))
      - record: orders:errors:rate60s
        expr: sum(rate(http_requests_total{status="error"}[60s]))
      - record: orders:error_ratio:rate60s
        expr: orders:errors:rate60s / orders:requests:rate60s
      - record: orders:burn_rate:ratio60s
        expr: orders:error_ratio:rate60s / 0.005

      # headroom at the 60s window: how far the CURRENT short-term rate is from the
      # sustainable rate, right now -- not the actual accumulated 30-day budget
      - record: orders:burn_rate_headroom:ratio60s
        expr: 1 - orders:burn_rate:ratio60s

      # the real error budget, over the real compliance window scaled by the same x60
      # factor as everything else in this lesson (30d -> 12h)
      - record: orders:error_budget_remaining:ratio12h
        expr: >-
          1 - (sum(increase(http_requests_total{status="error"}[12h]))
          / (sum(increase(http_requests_total[12h])) * 0.005))

      - alert: ErrorBudgetBurnFast
        expr: orders:burn_rate:ratio60s > 14.4 and orders:burn_rate:ratio5s > 14.4
        for: 10s
        labels:
          severity: page
        annotations:
          summary: "Error budget burning {{ $value | printf \"%.1f\" }}x the sustainable rate (1h/5m window)"

orders:burn_rate_headroom:ratio60s and orders:error_budget_remaining:ratio12h are two different questions that happen to look similar. The first is 1 - burn_rate over the short, reactive 60-second window — it answers “at this exact moment, how much louder could errors get before the sustainable rate is exceeded,” and it swings fast. The second integrates increase() over the real, scaled 12-hour compliance window — it answers “of the actual budget for this period, how much is left,” and in a real deployment it moves far more slowly, because it’s averaging over 12 real hours of traffic, not 60 seconds. With the steady 20 requests/second load running at the healthy 0.002 baseline error rate, after about 95 seconds of warm-up:

$ curl -s http://localhost:8000/metrics
# HELP http_requests_total Total HTTP requests handled, by status
# TYPE http_requests_total counter
http_requests_total{status="success"} 1635
http_requests_total{status="error"} 4
$ curl -s --data-urlencode 'query={__name__=~"orders:burn_rate:ratio5s|orders:burn_rate:ratio60s"}' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:burn_rate:ratio5s"}, "value": [1793092052.6, "0"]}, {"metric": {"__name__": "orders:burn_rate:ratio60s"}, "value": [1793092052.6, "0.6615214994487322"]}]
$ curl -s --data-urlencode 'query=orders:burn_rate_headroom:ratio60s' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:burn_rate_headroom:ratio60s"}, "value": [1793092052.755, "0.3384785005512678"]}]
$ curl -s --data-urlencode 'query=orders:error_budget_remaining:ratio12h' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:error_budget_remaining:ratio12h"}, "value": [1793092052.907, "0.4677312042581504"]}]

The 60-second headroom (0.3385) and the 12-hour budget (0.4677) are both healthy and positive, but they are not the same number — a real, measurable gap even this early. That’s expected: the headroom reacts to whatever the last 60 seconds looked like (which happened to include a slightly larger share of the handful of early errors), while the 12-hour budget rule is already averaging over everything the process has seen so far. Neither is “the” number — they answer different questions, and in a real deployment with actual history behind the 12-hour rule, the gap between them would be the normal case, not an exception.

Scripting a spike and watching the burn rate climb

$ curl -s -X POST "http://localhost:8000/control?error_rate=0.9"
error_rate=0.9

With the same 20 req/s load still running, both windows climb — read together, from a single query, so both numbers come from the exact same instant:

$ curl -s --data-urlencode 'query={__name__=~"orders:burn_rate:ratio5s|orders:burn_rate:ratio60s"}' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:burn_rate:ratio5s"}, "value": [1793092094.324, "186.44067796610167"]}, {"metric": {"__name__": "orders:burn_rate:ratio60s"}, "value": [1793092094.324, "62.56983240223464"]}]
$ curl -s --data-urlencode 'query=orders:error_budget_remaining:ratio12h' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:error_budget_remaining:ratio12h"}, "value": [1793092094.475, "-31.249198350893266"]}]

186x and 63x the sustainable rate on the two short-term windows, and even the 12-hour budget rule — immature as it still is this early in the run — has swung negative, because the burst is now a large fraction of this process’s entire short lifetime.

The alert fires — both windows, not one

$ curl -s "http://localhost:9090/api/v1/rules" | python -c "
import json,sys
d=json.load(sys.stdin)
for g in d['data']['groups']:
    for r in g['rules']:
        if r.get('name')=='ErrorBudgetBurnFast':
            print(json.dumps(r))
"
{"state": "firing", "name": "ErrorBudgetBurnFast", "query": "orders:burn_rate:ratio60s > 14.4 and orders:burn_rate:ratio5s > 14.4", "duration": 10, "keepFiringFor": 0, "labels": {"severity": "page"}, "annotations": {"summary": "Error budget burning {{ $value | printf \"%.1f\" }}x the sustainable rate (1h/5m window)"}, "alerts": [{"labels": {"alertname": "ErrorBudgetBurnFast", "severity": "page"}, "annotations": {"summary": "Error budget burning 78.4x the sustainable rate (1h/5m window)"}, "state": "firing", "activeAt": "2026-10-27T09:07:49.444501875Z", "value": "7.838745800671892e+01"}], "health": "ok", "evaluationTime": 0.00011471, "lastEvaluation": "2026-10-27T09:08:14.433609996Z", "type": "alerting"}

Both halves of the and were true for a full 10s, so it fired — delivered all the way through to the webhook receiver:

$ curl -s http://localhost:9093/api/v2/alerts | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)))"
[{"annotations": {"summary": "Error budget burning 46.8x the sustainable rate (1h/5m window)"}, "endsAt": "2026-10-27T09:11:59.444Z", "fingerprint": "b82cd17bc01d43ea", "receivers": [{"name": "webhook-receiver"}], "startsAt": "2026-10-27T09:07:59.444Z", "status": {"inhibitedBy": [], "mutedBy": [], "silencedBy": [], "state": "active"}, "updatedAt": "2026-10-27T09:08:04.436Z", "generatorURL": "http://6ba281d1e898:9090/graph?g0.expr=orders%3Aburn_rate%3Aratio60s+%3E+14.4+and+orders%3Aburn_rate%3Aratio5s+%3E+14.4&g0.tab=1", "labels": {"alertname": "ErrorBudgetBurnFast", "severity": "page"}}]
$ docker compose logs webhook-receiver | grep RECEIVED
webhook-receiver-1  | RECEIVED alerts=['ErrorBudgetBurnFast'] statuses=['firing']

Resolving: the short window clears first

$ curl -s -X POST "http://localhost:8000/control?error_rate=0.002"
error_rate=0.002

A few seconds later, both windows read together again, from one query at one instant:

$ curl -s --data-urlencode 'query={__name__=~"orders:burn_rate:ratio5s|orders:burn_rate:ratio60s"}' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:burn_rate:ratio5s"}, "value": [1793092124.932, "0"]}, {"metric": {"__name__": "orders:burn_rate:ratio60s"}, "value": [1793092124.932, "126.28062360801782"]}]
$ curl -s "http://localhost:9090/api/v1/rules" | python -c "
import json,sys
d=json.load(sys.stdin)
for g in d['data']['groups']:
    for r in g['rules']:
        if r.get('name')=='ErrorBudgetBurnFast':
            print(r['state'])
"
inactive

The 5-second window had already dropped all the way to 0 — well under the 14.4 threshold — while the 60-second (1h-scaled, long) window was still at 126.3x, because a minute-long average needs most of a minute of clean traffic before the spike stops dominating it. The alert’s and means either window failing the threshold resolves it, so the fast-recovering short window cleared the alert (confirmed above: inactive) without waiting on the slow one:

$ curl -s http://localhost:9093/api/v2/alerts
[]
$ docker compose logs webhook-receiver | grep RECEIVED
webhook-receiver-1  | RECEIVED alerts=['ErrorBudgetBurnFast'] statuses=['firing']
webhook-receiver-1  | RECEIVED alerts=['ErrorBudgetBurnFast'] statuses=['resolved']

That resolved line arrived within alertmanager.yml’s group_interval: 5s of the alert actually clearing — a real few seconds, not an instant one, because Alertmanager’s own timer is what paces it, not something this lesson rushed past.

Practice

1. Add the slow-burn pair (6h/30m scaled to 360s/30s, threshold 6) alongside the existing fast-burn rules, and confirm it fires on the same kind of spike. This task's rules must be loaded before Practice 2 can use them.

# rules.yml (slow-burn pair, added alongside the fast-burn rules)
      - record: orders:requests:rate360s
        expr: sum(rate(http_requests_total[360s]))
      - record: orders:errors:rate360s
        expr: sum(rate(http_requests_total{status="error"}[360s]))
      - record: orders:error_ratio:rate360s
        expr: orders:errors:rate360s / orders:requests:rate360s
      - record: orders:burn_rate:ratio360s
        expr: orders:error_ratio:rate360s / 0.005

      - record: orders:requests:rate30s
        expr: sum(rate(http_requests_total[30s]))
      - record: orders:errors:rate30s
        expr: sum(rate(http_requests_total{status="error"}[30s]))
      - record: orders:error_ratio:rate30s
        expr: orders:errors:rate30s / orders:requests:rate30s
      - record: orders:burn_rate:ratio30s
        expr: orders:error_ratio:rate30s / 0.005

      - alert: ErrorBudgetBurnSlow
        expr: orders:burn_rate:ratio360s > 6 and orders:burn_rate:ratio30s > 6
        for: 30s
        labels:
          severity: ticket
        annotations:
          summary: "Error budget burning {{ $value | printf \"%.1f\" }}x the sustainable rate (6h/30m window)"

$ docker compose up -d --force-recreate prometheus 2>&1 | grep -E "Recreated|Started"
 Container slo-clean-prometheus-1 Recreated
 Container slo-clean-prometheus-1 Started
$ curl -s -X POST "http://localhost:8000/control?error_rate=0.9"
error_rate=0.9
$ curl -s "http://localhost:9090/api/v1/rules" | python -c "
import json,sys
d=json.load(sys.stdin)
for g in d['data']['groups']:
    for r in g['rules']:
        if r.get('name')=='ErrorBudgetBurnSlow':
            print(json.dumps(r))
"
{"state": "firing", "name": "ErrorBudgetBurnSlow", "query": "orders:burn_rate:ratio360s > 6 and orders:burn_rate:ratio30s > 6", "duration": 30, "keepFiringFor": 0, "labels": {"severity": "ticket"}, "annotations": {"summary": "Error budget burning {{ $value | printf \"%.1f\" }}x the sustainable rate (6h/30m window)"}, "alerts": [{"labels": {"alertname": "ErrorBudgetBurnSlow", "severity": "ticket"}, "annotations": {"summary": "Error budget burning 59.4x the sustainable rate (6h/30m window)"}, "state": "firing", "activeAt": "2026-10-27T09:09:19.444501875Z", "value": "5.938375350140056e+01"}], "health": "ok", "evaluationTime": 0.000166969, "lastEvaluation": "2026-10-27T09:10:04.457199147Z", "type": "alerting"}

59.4x — a lower threshold (6 instead of 14.4) crossed by the same spike, confirmed firing with its own ticket severity, completely independently of the fast-burn alert above it in the same file. Recreating the Prometheus container only reloads the rules: the image declares /prometheus as a volume, and Compose reuses it, so the earlier history survives (Practice 3’s budget still shows the fast-burn spike). The spike is scripted fresh so this alert fires on traffic you can point to. It doesn’t take long: at a 90% error rate the 360-second window only needs about 3% of its samples bad (6 × 0.005) to cross 6, so a few seconds of spike are enough and the 30-second for is most of the wait. The loadgen.py 1200 20 started earlier keeps supplying the healthy traffic around it.

2. Right after Practice 1's spike, confirm the slow-burn alert's long window can stay elevated from that spike while its short window is already back to normal — and that the alert stays inactive anyway. Needs Practice 1's rules and spike already run; wait about 45 seconds after the spike ends before querying.

Resetting error_rate back to 0.002, then querying both windows together, from one query at one instant, once the short window has had a little time to recover:

$ curl -s -X POST "http://localhost:8000/control?error_rate=0.002"
error_rate=0.002
$ curl -s --data-urlencode 'query={__name__=~"orders:burn_rate:ratio30s|orders:burn_rate:ratio360s"}' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:burn_rate:ratio360s"}, "value": [1793092250.024, "57.19819238218206"]}, {"metric": {"__name__": "orders:burn_rate:ratio30s"}, "value": [1793092250.024, "0"]}]
$ curl -s "http://localhost:9090/api/v1/rules" | python -c "
import json,sys
d=json.load(sys.stdin)
for g in d['data']['groups']:
    for r in g['rules']:
        if r.get('name')=='ErrorBudgetBurnSlow':
            print(r['state'])
"
inactive

The 360-second window was still showing 57.2x — a real echo of the spike that just fired this alert, since 6 minutes hadn’t yet passed — while the 30-second window had already dropped to 0. The and kept the alert inactive once the short window recovered, even with that stale long window still elevated: a lagging long window alone, with a currently-healthy short window, is exactly the case multi-window alerting exists to not page on.

3. Query orders:error_budget_remaining:ratio12h again a little later, after more healthy traffic has run, and confirm it's climbing back toward 1 rather than staying stuck at the negative value the spike left it at.

$ curl -s --data-urlencode 'query=orders:error_budget_remaining:ratio12h' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:error_budget_remaining:ratio12h"}, "value": [1793092256.021, "-55.23016712502645"]}]

A minute later, with the service back at its healthy baseline the whole time:

$ curl -s --data-urlencode 'query=orders:error_budget_remaining:ratio12h' http://localhost:9090/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"__name__": "orders:error_budget_remaining:ratio12h"}, "value": [1793092342.95, "-42.79009705543675"]}]

Still negative (-42.8, up from -55.2 sixty seconds earlier) — climbing, but slowly, because increase() over a 12-hour window only moves as fast as real traffic accumulates within it. In this short demo the window is still dominated by two back-to-back spikes (the fast-burn one and Practice 1’s slow-burn one); in a real 30-day-scaled-to-12-hour deployment, a brief spike would recover fully once enough clean traffic had piled up behind it, the same direction this number is already moving in.

A real production SLO rule set tracks the actual 30-day window directly (via increase() over 30d, exactly like orders:error_budget_remaining:ratio12h does at the scaled 12h here, just with real history behind it), and layers the slow-burn pair from Practice 1 alongside the fast pair rather than running either alone — the fast pair catches what needs a human now, the slow pair catches what’s draining the budget without an acute spike to point at.

Programming

Web Development

Generative AI

Machine Learning

Cybersecurity

Data Engineering

Cloud

DevOps

System Design

GPU & Parallel

Databases

Testing

Interviews