Skip to main content
Monitoring & Observability · lesson 1 of 5

Beginner

1

Intermediate

2

Advanced

2

Also part of Monitoring & Observability

3
Open the Monitoring & Observability page →
Monitoring & Observability beginner Lesson 1 of 5

Also part of: DevOps

Prometheus Scrapes You — It's Pull, Not Push

Prometheus doesn't wait for your app to send it metrics. It calls your app's /metrics endpoint on a timer — and a wrong port doesn't error on your side, it just shows up as a real 'down' target in Prometheus's own API.

By SkillByExample · Updated

monitoring prometheus metrics observability

Prometheus never asks your application to send it anything. It shows up, on its own schedule, and asks your application for its current numbers.

A metrics endpoint, in plain text

# metrics_app.py (excerpt)
def do_GET(self):
    if self.path == "/metrics":
        body = (
            "# HELP app_requests_total Total HTTP requests handled\n"
            "# TYPE app_requests_total counter\n"
            f"app_requests_total {request_count}\n"
            "# HELP app_uptime_seconds Seconds since process start\n"
            "# TYPE app_uptime_seconds gauge\n"
            f"app_uptime_seconds {uptime:.2f}\n"
        ).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.end_headers()
        self.wfile.write(body)
curl -s http://localhost:8000/metrics
# HELP app_requests_total Total HTTP requests handled
# TYPE app_requests_total counter
app_requests_total 0
# HELP app_errors_total Total HTTP 500 responses
# TYPE app_errors_total counter
app_errors_total 0
# HELP app_uptime_seconds Seconds since process start
# TYPE app_uptime_seconds gauge
app_uptime_seconds 10.35

No Prometheus client library, no special protocol — plain text over HTTP, from Python’s standard library alone. # TYPE ... counter versus ... gauge isn’t decoration: it’s how Prometheus’s own tooling later knows a counter only ever increases and a gauge can move both ways.

Prometheus does the asking

# prometheus.yml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "demo-app"
    static_configs:
      - targets: ["metrics-app:8000"]
curl -s http://localhost:9090/api/v1/targets
{"status":"success","data":{"activeTargets":[{"discoveredLabels":{"__address__":"metrics-app:8000", ...},"labels":{"instance":"metrics-app:8000","job":"demo-app"},"scrapePool":"demo-app","scrapeUrl":"http://metrics-app:8000/metrics","lastError":"","lastScrape":"2026-09-14T06:55:34.557188228Z","lastScrapeDuration":0.016734081,"health":"up", ...}]}}

"health":"up" and an empty "lastError" confirm Prometheus successfully connected to the app’s /metrics endpoint on its own — the app did nothing to make this scrape happen beyond existing and answering when asked.

Querying what was scraped

curl -s "http://localhost:9090/api/v1/query?query=app_requests_total"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"app_requests_total","instance":"metrics-app:8000","job":"demo-app"},"value":[1789368946.111,"0"]}]}}

The value Prometheus reports ("0") is exactly the number the app’s /metrics endpoint printed at the last scrape — Prometheus stores what it pulled, on its own timeline, not whatever the app’s value happens to be right now.

The bug: a wrong port doesn’t error where you’d expect

# prometheus.yml (wrong port on purpose)
scrape_configs:
  - job_name: "demo-app"
    static_configs:
      - targets: ["metrics-app:9999"]
curl -s http://localhost:9090/api/v1/targets
{"status":"success","data":{"activeTargets":[{"discoveredLabels":{"__address__":"metrics-app:9999", ...},"labels":{"instance":"metrics-app:9999","job":"demo-app"},"scrapePool":"demo-app","scrapeUrl":"http://metrics-app:9999/metrics","lastError":"Get \"http://metrics-app:9999/metrics\": dial tcp 172.19.0.2:9999: connect: connection refused","lastScrape":"2026-09-14T06:56:40.825577992Z","lastScrapeDuration":0.001024624,"health":"down", ...}]}}

The application never saw this request at all — there’s nothing in the app’s own logs because the connection was refused before it ever reached the app’s port. The failure is entirely on Prometheus’s side, visible only by checking Prometheus’s own /api/v1/targets — "health": "down" with the real connection refused error naming the exact address it tried.

Practice

1. Query app_uptime_seconds directly and confirm it increases between two queries a few seconds apart. 
curl -s "http://localhost:9090/api/v1/query?query=app_uptime_seconds"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"app_uptime_seconds","instance":"metrics-app:8000","job":"demo-app"},"value":[1789369059.657,"179.23"]}]}}

A gauge like app_uptime_seconds is free to report any value each scrape — here it climbs steadily because the app just reports real elapsed time, but a gauge could just as easily go down (queue depth, active connections).

2. Hit the app's /work endpoint a few times, then confirm app_requests_total went up by exactly that many. 
curl -s http://localhost:8000/work
curl -s http://localhost:8000/work
curl -s http://localhost:8000/work
curl -s "http://localhost:9090/api/v1/query?query=app_requests_total"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"app_requests_total","instance":"metrics-app:8000","job":"demo-app"},"value":[1789369301.111,"3"]}]}}

The counter only reflects what the app itself counted — Prometheus doesn’t count requests to /work on its own; it just reports whatever number the app’s /metrics endpoint prints at the next scrape.

3. Query a metric name that doesn't exist and see what Prometheus returns. 
curl -s "http://localhost:9090/api/v1/query?query=nonexistent_metric_name"
{"status":"success","data":{"resultType":"vector","result":[]}}

Still "status":"success" — an empty result set, not an error. A PromQL query for a metric that was never scraped isn’t invalid syntax, it’s just a query that correctly finds nothing.

Next: PromQL’s most common gotcha — a raw counter value is almost never the number you actually want.

Frequently Asked Questions

Is Prometheus push-based or pull-based?
Pull-based, by design. Your application does nothing except expose a /metrics HTTP endpoint in a specific text format; Prometheus is the one that initiates the connection, on a schedule it controls (scrape_interval). This is the opposite of push-based systems like StatsD, where the application actively sends data out — and it's the single most common point of confusion for anyone coming from a push-based background.
What does the # HELP and # TYPE syntax in the metrics output mean?
They're comment lines in the Prometheus text exposition format, not decoration — # HELP gives a human-readable description and # TYPE declares whether the metric is a counter (only goes up), gauge (goes up or down), histogram, or summary. Prometheus's own tooling reads these lines to know how to handle the metric correctly, particularly for functions like rate() that only make sense on a counter.
Why did the target show 'down' instead of my app returning an error?
Because the failure isn't in your application at all — the app was never even reached. When Prometheus can't connect to the address in its own scrape config, that's a networking-level failure (connection refused, timeout, DNS failure) that Prometheus records on its own side, visible only through Prometheus's own targets page or API, not through anything your app logs.
How often does Prometheus actually scrape a target?
Whatever scrape_interval says in the config — 5s in this lesson's example, 15s-60s is more typical in production. Every scrape is independent; there's no persistent connection or streaming involved, just a plain HTTP GET repeated on that interval.
Do I need a client library to expose metrics?
No — the exposition format is just plain text over HTTP, which is why this lesson's example app uses nothing but Python's standard library. Client libraries (prometheus_client for Python, the Go/Java/Node equivalents) exist to make instrumenting a real application more convenient — tracking counters and histograms across many code paths — not because the format itself requires one.