Why can't I just read the raw counter value as "how many requests am I getting"?

A counter's raw value is a running total since the process started — it tells you the cumulative count, not a rate. Ten requests over the last hour and ten requests over the last second both look identical as a raw number; you need to know how much it changed over a specific time window to get anything resembling "requests per second", which is exactly what rate() computes.

What does rate(app_requests_total[1m]) actually calculate?

The per-second average rate of increase over the trailing 1-minute window, computed from the actual data points Prometheus has for that series in that range. It's not simply (now - 60s ago) / 60 — Prometheus extrapolates slightly at the edges of the range to account for scrape timing, but the core idea is exactly that: change over time, not a raw count.

Why didn't rate() go negative when the counter reset from 13 to 5 after a restart?

rate() (and increase()) specifically detect counter resets — whenever a sample's value is lower than the previous one, Prometheus assumes the counter restarted at zero and adjusts the calculation to add the pre-reset value back in, rather than computing a nonsensical negative delta. This reset-handling is precisely why counters should always be queried through rate()/increase() and never read or diffed raw.

What would happen if I tried to compute the rate myself by just subtracting two raw values?

You'd get exactly the naive, wrong answer this lesson demonstrates: 5 (after restart) minus 13 (before restart) is -8, which is nonsense for a value that's supposed to only increase. This is the concrete, mechanical reason the standing advice "always wrap a counter in rate()" exists — it's not a style preference, it's the only way to get a real number across a restart.

Is increase() different from rate()?

increase(metric[5m]) is exactly rate(metric[5m]) times the number of seconds in the range — one gives a per-second rate, the other gives a total count over that window. Both apply the identical counter-reset handling; pick whichever unit makes more sense for what you're displaying (a rate for a graph's y-axis, an increase for "how many errors happened in the last 5 minutes").

PromQL's rate(): Why a Raw Counter Is Never the Answer

A counter's raw value can't tell you a request rate on its own — and after a real app restart resets it to zero, a naive before/after subtraction goes negative. rate() gets it right; the arithmetic you'd do by hand doesn't.

By SkillByExample · Updated Sep 14, 2026

monitoring prometheus promql grafana

The most common Prometheus mistake isn’t a config error — it’s reading a counter’s raw value and treating it as if it already answered the question.

The raw counter alone answers nothing useful

for i in 1 2 3 4 5 6 7 8 9 10; do curl -s http://localhost:8000/work > /dev/null; done
curl -s "http://localhost:9090/api/v1/query?query=app_requests_total"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"app_requests_total","instance":"metrics-app:8000","job":"demo-app"},"value":[1789369427.899,"13"]}]}}

13 — but 13 over what period? Ten seconds, ten minutes, since the process started an hour ago? The raw number alone can’t distinguish “quiet all day, then a burst” from “steady low traffic.” A counter’s raw value is a running total; it was never meant to be read directly.

rate() turns a running total into an actual rate

curl -s "http://localhost:9090/api/v1/query?query=rate(app_requests_total%5B1m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369437.853,"0.18183140592043057"]}]}}

0.18 requests per second, averaged over the trailing minute — a real, usable rate, computed from how much the counter actually moved across that window, not from its raw value at one instant.

The bug: a real app restart resets the counter, and naive math goes negative

docker restart metrics-app
curl -s http://localhost:8000/metrics

# HELP app_requests_total Total HTTP requests handled
# TYPE app_requests_total counter
app_requests_total 0

A real process restart — the counter is genuinely back to 0, not simulated. After a small amount of new traffic:

for i in 1 2 3 4 5; do curl -s http://localhost:8000/work > /dev/null; done
curl -s "http://localhost:9090/api/v1/query?query=app_requests_total"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"app_requests_total","instance":"metrics-app:8000","job":"demo-app"},"value":[1789369479.265,"5"]}]}}

The counter now reads 5 — down from 13 before the restart. If you tried to compute a rate by hand the obvious way, subtracting the earlier reading from this one, you’d get 5 - 13 = -8: a negative count of requests, which is nonsense for a value that can only ever increase.

The fix: rate() already knows about restarts

curl -s "http://localhost:9090/api/v1/query?query=rate(app_requests_total%5B2m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369556.723,"0.04348658004139922"]}]}}

A small positive number, computed correctly across the exact same restart. rate() detects that the counter’s value dropped between samples, recognizes that as a reset rather than an error, and adds the pre-reset value back into the total instead of computing a raw, negative difference. This reset-handling is the entire reason the standing advice “never read a counter raw, always wrap it in rate() or increase()” exists — it isn’t a style preference, it’s the only way the number stays meaningful across a restart.

The same query through Grafana looks different — because it’s a different query

Wiring a real Grafana into the stack, on the same Docker network as Prometheus:

docker run -d --name grafana --network monitoring-net -p 3001:3000 grafana/grafana:latest

Then adding Prometheus as a datasource through Grafana’s own API (default login is admin/admin):

curl -s -u admin:admin -X POST http://localhost:3001/api/datasources \
  -H "Content-Type: application/json" \
  -d '{"name":"Prometheus","type":"prometheus","url":"http://prometheus:9090","access":"proxy","isDefault":true}'

{"datasource":{"id":1,"uid":"cfy7slbzea7swf","name":"Prometheus","type":"prometheus","url":"http://prometheus:9090", ...},"message":"Datasource added"}

Running the exact same rate(app_requests_total[2m]) through Grafana’s datasource proxy as an instant query gives back the identical value curl got directly from Prometheus:

curl -s -u admin:admin "http://localhost:3001/api/datasources/proxy/uid/cfy7slbzea7swf/api/v1/query?query=rate(app_requests_total%5B2m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789377301.141,"0.2608945203454243"]}]}}

But a Grafana panel never issues that instant query — it issues a range query, because a panel draws a line, not a single number. The same expression through query_range:

curl -s -u admin:admin "http://localhost:3001/api/datasources/proxy/uid/cfy7slbzea7swf/api/v1/query_range?query=rate(app_requests_total%5B2m%5D)&start=1789377197&end=1789377317&step=15"

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"values":[[1789377197,"0"],[1789377212,"0"],[1789377227,"0"],[1789377242,"0"],[1789377257,"0"],[1789377272,"0"],[1789377287,"0"],[1789377302,"0.2608945203454244"],[1789377317,"0.2608945203454243"]]}]}}

Nine points, one per 15-second step — flat at 0 for the first seven, because the burst that produced 0.26 happened right at the end of this two-minute range. The number itself never changed; what changed is that a dashboard panel evaluates the query repeatedly across a whole time window instead of once “now,” which is exactly why a graph can show a line building up to a value instead of a single steady number — the underlying PromQL is unchanged, but a panel and a one-off API call are never asking Prometheus the same actual question.

Practice

1. Query increase() instead of rate() over the same 2-minute window and compare the numbers.

curl -s "http://localhost:9090/api/v1/query?query=increase(app_requests_total%5B2m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369634.120,"5.217799653885956"]}]}}

increase() reports the total count over the window instead of a per-second rate — rate(x[2m]) multiplied by 120 seconds lands close to this same value, since they’re computing the same underlying reset-aware delta, just scaled differently.

2. Generate a burst of traffic and query rate() with a short window (15s) versus a longer one (5m).

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do curl -s http://localhost:8000/work > /dev/null; done
curl -s "http://localhost:9090/api/v1/query?query=rate(app_requests_total%5B15s%5D)"
curl -s "http://localhost:9090/api/v1/query?query=rate(app_requests_total%5B5m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369681.800,"2.0006001800540165"]}]}}
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369690.980,"0.20340568925712849"]}]}}

2.00 requests per second in the short window versus 0.20 in the 5-minute one — the same 20-request burst, but the short window reacts fast and reports something close to the burst’s actual intensity, while the 5-minute window smooths it into a small fraction of a much longer averaging period. Neither is “more correct” — window size is a real tradeoff between responsiveness and noise, not a fixed default to leave alone.

3. Query app_errors_total with rate() to see the real error rate, not just the raw error count.

curl -s "http://localhost:9090/api/v1/query?query=rate(app_errors_total%5B5m%5D)"

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"metrics-app:8000","job":"demo-app"},"value":[1789369904.470,"0.013335066892029299"]}]}}

Same principle applied to a different counter — app_errors_total’s raw value is just as uninterpretable alone as app_requests_total’s was; rate() is the only way to turn it into “how often are errors actually happening” rather than “how many errors ever.”

Next: alerting — a threshold on a rate() query, and the for: duration that decides how long a breach has to last before anyone gets paged.