Prometheus & Grafana: Metrics and Dashboards

Prometheus collects metrics; Grafana visualises them. Together they form the most widely used open-source monitoring stack in DevOps.

Learning outcomes

By the end you can:

• run Prometheus and Grafana locally with Docker Compose
• understand the Prometheus data model and labels
• write basic PromQL queries
• build a Grafana dashboard
• define alert rules in Prometheus

1) Core concepts

Metric types:

• Counter: ever-increasing value (total requests, errors)—always use rate() or increase() to query it
• Gauge: current value that can go up or down (memory, active connections)
• Histogram: distribution of values with buckets (request latency)—enables percentile queries
• Summary: pre-calculated quantiles on the client side

Labels give context to metrics:

http_requests_total{method="GET", status="200", path="/api/users"}

2) Run the stack with Docker Compose

# docker-compose.yml
version: "3.9"

services:
  prometheus:
    image: prom/prometheus:v2.52.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=15d"

  grafana:
    image: grafana/grafana:10.4.2
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin123

  node-exporter:
    image: prom/node-exporter:v1.8.0
    ports:
      - "9100:9100"
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"

volumes:
  prometheus_data:
  grafana_data:

3) Prometheus configuration

# prometheus.yml
global:
  scrape_interval: 15s       # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]

  - job_name: node-exporter
    static_configs:
      - targets: [node-exporter:9100]

  - job_name: myapp
    static_configs:
      - targets: [myapp:8080]
    metrics_path: /metrics

Start everything:

docker compose up -d
# Prometheus: http://localhost:9090
# Grafana:    http://localhost:3000  (admin / admin123)

4) PromQL basics

Rate of a counter (requests per second)

rate(http_requests_total[5m])

Error rate percentage

rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m]) * 100

p95 latency from a histogram

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Memory usage in MB

node_memory_MemAvailable_bytes / 1024 / 1024

CPU utilisation percentage

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Active connections gauge

myapp_active_connections

5) Recording rules (pre-compute expensive queries)

Recording rules run on a schedule and store the result as a new metric. They make dashboards faster and avoid re-computing complex expressions.

# prometheus.yml — add a rule_files entry
rule_files:
  - "rules/*.yml"

# rules/http.yml
groups:
  - name: http
    interval: 1m
    rules:
      - record: job:http_requests_total:rate5m
        expr: rate(http_requests_total[5m])

      - record: job:http_error_rate:rate5m
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
            / rate(http_requests_total[5m])

Now use job:http_error_rate:rate5m in dashboards—much cheaper to query.

6) Alert rules in Prometheus

# rules/alerts.yml
groups:
  - name: application
    rules:
      - alert: HighErrorRate
        expr: job:http_error_rate:rate5m > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
          runbook_url: "https://runbooks.example.com/high-error-rate"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency above 500ms on {{ $labels.job }}"
          description: "Current p95: {{ $value | humanizeDuration }}"

7) Grafana: connect to Prometheus

1. Open http://localhost:3000
2. Go to Connections → Data sources → Add data source
3. Select Prometheus
4. URL: http://prometheus:9090
5. Click Save & Test

Build a dashboard panel

1. Click Dashboards → New → New dashboard → Add visualization
2. Select the Prometheus data source
3. Enter a PromQL query: rate(http_requests_total[5m])
4. Choose a visualization type (Time series, Stat, Gauge)
5. Add a meaningful title and units

Import community dashboards

Many pre-built dashboards exist on Grafana Dashboards:

• Node Exporter Full: ID 1860
• Kubernetes cluster monitoring: ID 315

Import via Dashboards → Import → Enter dashboard ID.

Next steps

• OpenTelemetry: unified observability with traces, metrics, and logs
• Alertmanager: routing alerts to Slack, PagerDuty, and email
• Incident management and post-mortem practices