Alerting & Runbooks

Alerts are only useful if they lead to action. Without runbooks, alerts turn into “tribal knowledge” emergencies.

Learning outcomes

You’ll be able to:

• write alert rules with clear impact
• attach/maintain runbooks
• create a minimal incident response loop

1) Alert design principles

Alert on user impact

Instead of:

• CPU > 95% Prefer:
• error rate increased
• p95 latency above SLO
• checkout failures > threshold

Include a response hint

Every alert should answer:

• what does this mean
• what should you check first
• who owns it

2) Create a runbook template

Use this structure:

Title: <service/alert name>
Impact: <what broke for users>
When: <time window or trigger conditions>
Where to look:
  - Dashboard: <link>
  - Logs query: <query>
  - Trace examples: <link>
Likely causes:
  - <cause1>
  - <cause2>
Step-by-step actions:
  1) ...
  2) ...
Rollback/Mitigation:
  - ...
Owners:
  - ...

3) Example alert scenarios (and what to check)

Scenario A: Latency spikes

First checks:

• recent deploys
• DB latency / saturation
• thread pool / queue depth Mitigation:
• rollback
• scale out
• reduce expensive requests

Scenario B: Error rate increases

First checks:

• error type distribution
• downstream service health
• logs with error signature Mitigation:
• switch feature flag off
• increase retries/backoff carefully

4) Practical alert tuning workflow

1. Collect baseline data (last 2–4 weeks)
2. Choose thresholds aligned to SLO impact
3. Test alerts in staging or using shadow mode
4. Review after incidents (what fired vs what should have fired)
5. Iterate (alert hygiene is continuous)

5) Next steps

Next tutorials:

• instrumenting services for better signals
• OpenTelemetry mindset
• integrating CI/CD quality gates (fail fast before production)