Monitoring & Observability Tutorial with Examples.
Monitoring and observability tutorial: metrics, logs, traces, Prometheus, Grafana, OpenTelemetry and alerting strategies.
Beginner
Start here — no prior experience needed
Intermediate
For developers with core concepts down
PromQL's rate(): Why a Raw Counter Is Never the Answer
A counter's raw value can't tell you a request rate on its own — and after a real app restart resets it to zero, a naive before/after subtraction goes negative. rate() gets it right; the arithmetic you'd do by hand doesn't.
A Firing Alert Doesn't Mean Anyone Was Notified
Prometheus evaluates alert rules and Alertmanager delivers notifications — two separate systems. A misconfigured link between them lets an alert fire silently forever, with real Prometheus and Alertmanager API output proving it.
Advanced
Production-grade patterns for experienced engineers
OpenTelemetry Buffers Spans — a SIGKILL Can Lose Them
OpenTelemetry's BatchSpanProcessor holds finished spans in memory and exports them on a timer. A process killed before that timer fires loses every span it created — proven with a real trace, a real SIGKILL, and force_flush().
MTTR Isn't a Guess — Compute It From Alertmanager's Own Timestamps
Every alert notification Alertmanager sends already carries the raw material for MTTR: startsAt and endsAt. Log them for real, compute a real mean-time-to-resolve, and see why the number it produces is longer than the actual outage.
Also part of Monitoring & Observability
Taught in other topics. Each link opens the lesson in its home topic.
Observability Synthesis: Logs, Metrics, Traces, and Alerts as One Signal
Generate structured logs, aggregate them into metrics, fire an alert off a real threshold breach, and use a trace ID to walk straight back from the alert to the log lines that explain it.
Pipeline Observability: Tracing, Anomaly Detection, and Alerting
Trace a pipeline run stage-by-stage, then flag a run 3x longer than usual and an error rate crossing its threshold - operational health, not data quality.
MLOps Model Monitoring
Detect data drift with KS tests and PSI, avoid the large-sample p-value trap, see concept drift that inputs can't show, and turn checks into alerts.