Load Testing with Locust: Percentiles, the Knee and a CI Gate

A load test answers a question unit tests can’t: what happens when many people use the system at once? This lesson points Locust 2.46.7, a Python load-testing tool, at a small local API in Docker. You’ll read percentiles, find the load where latency starts to climb, see why “number of users” is a misleading measure, and turn a latency budget into a failing CI step. Everything runs on one machine, with no cloud account.

The lab: an API with one fast and one limited endpoint

/items is fast, except that 1 request in 25 hits a cold cache and takes 400 ms. /report does 100 ms of work holding one of only two “database connections”, so it can serve at most 20 requests per second.

# server.py - the system under test: one fast endpoint with a rare slow path, one endpoint behind a small pool
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

POOL = threading.BoundedSemaphore(2)  # like a database with 2 connections
lock = threading.Lock()
item_requests = 0


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        global item_requests
        if self.path == "/items":
            with lock:
                item_requests += 1
                slow = item_requests % 25 == 0  # 1 request in 25 hits a cold cache
            time.sleep(0.4 if slow else 0.005)
        elif self.path == "/report":
            with POOL:  # wait for a free connection, then do 100 ms of work
                time.sleep(0.1)
        else:
            self.send_error(404)
            return
        body = b'{"ok": true}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):  # keep the console quiet under load
        pass


ThreadingHTTPServer.request_queue_size = 128
print("api listening on :8000", flush=True)
ThreadingHTTPServer(("", 8000), Handler).serve_forever()

A Locust test is a Python class. Each simulated user runs its @task methods in a loop, picked at random by weight, and pauses for wait_time between them. self.client is a requests session that times every call.

# locustfile.py - one simulated user: lists items three times as often as it opens the report
from locust import HttpUser, constant, task


class ApiUser(HttpUser):
    wait_time = constant(0.5)  # half a second between one request and the next

    @task(3)
    def list_items(self):
        self.client.get("/items")

    @task(1)
    def open_report(self):
        self.client.get("/report")

Three small scripts run it. start.sh starts the API on a private Docker network, and load.sh runs Locust headless (no web UI) in a second container: -u users, all started at once (-r, the spawn rate per second), for -t seconds, writing CSV results to out/.

# start.sh - (re)start the API on a private Docker network called loadlab
export MSYS_NO_PATHCONV=1  # Git Bash on Windows: don't rewrite the /lab paths
docker rm -f api >/dev/null 2>&1
docker network create loadlab >/dev/null 2>&1
docker run -d --rm --name api --network loadlab -v "$PWD:/lab" python:3.13-slim python /lab/server.py >/dev/null
sleep 2
docker logs api

# load.sh USERS SECONDS [LOCUSTFILE] - run Locust headless against the API; summary only, CSVs in ./out
export MSYS_NO_PATHCONV=1
file="${3:-locustfile.py}"
mkdir -p out
docker run --rm --network loadlab -v "$PWD:/lab" -w /lab locustio/locust:2.46.7 \
  -f "$file" --host http://api:8000 \
  --headless -u "$1" -r "$1" -t "${2}s" --csv "out/$(basename "$file" .py)-u$1" --only-summary --loglevel WARNING

Locust’s own summary is wide, so summary.py prints the CSV numbers that matter, one line per endpoint.

# summary.py CSV... - one compact line per endpoint from Locust's *_stats.csv files (times in ms)
import csv
import sys

print(f"{'run':<12}{'endpoint':<12}{'reqs':>6}{'req/s':>7}{'avg':>6}{'p50':>6}{'p95':>6}{'p99':>6}{'max':>6}")
for path in sys.argv[1:]:
    run = path.split("/")[-1].removesuffix("_stats.csv").removeprefix("locustfile").lstrip("_-")
    for row in csv.DictReader(open(path)):
        print(f"{run:<12}{row['Name']:<12}{row['Request Count']:>6}{float(row['Requests/s']):>7.1f}"
              f"{float(row['Average Response Time']):>6.0f}{row['50%']:>6}{row['95%']:>6}{row['99%']:>6}"
              f"{float(row['Max Response Time']):>6.0f}")

Save the five files in one folder and run everything from it.

$ sh start.sh
api listening on :8000

1. A smoke test: one user

Before any real load, run one user for 15 seconds. It checks that the script, the URLs and the server all work.

$ sh load.sh 1 15
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /items                                                                            23     0(0.00%) |      9       7      13      9 |    1.61        0.00
GET      /report                                                                            5     0(0.00%) |    107     103     117    103 |    0.35        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                        28     0(0.00%) |     26       7     117      9 |    1.96        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /items                                                                                  9      9     10     10     10     11     13     13     13     13     13     23
GET      /report                                                                               100    110    110    120    120    120    120    120    120    120    120      5
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                              9     10     11     13    100    110    120    120    120    120    120     28

No failures. Tasks are picked at random by weight, so the 3-to-1 split only shows over many requests: across three runs this smoke test made 18 to 23 /items requests and 5 to 10 reports, all under 120 ms. /items takes about 9 ms, not the 5 ms the server sleeps, because each request also crosses the Docker network. The smoke run never hit the slow path, since fewer than 25 requests reached /items. Note “approximated”: Locust rounds response times into buckets to keep percentiles cheap, so for example a p99 can read slightly above the max.

2. The average hides the slow requests

Now 10 users for 30 seconds.

$ sh load.sh 10 30 2>/dev/null
$ python summary.py out/locustfile-u10_stats.csv
run         endpoint      reqs  req/s   avg   p50   p95   p99   max
u10         /items         408   14.0    26     9    14   410   412
u10         /report        132    4.5   108   100   110   210   209
u10         Aggregated     540   18.5    46    10   110   410   412

/items averaged 26 ms. Almost no request took 26 ms: half took 9 ms or less (p50, the median), 95% took 14 ms or less (p95), and the slowest 1 in 100 took about 410 ms (p99). The average is pulled up by the cold-cache requests, but it still makes them look like a small slowdown for everyone, when really 1 user in 25 waits 40 times longer than the rest. This is why latency budgets are written as percentiles. And with 4% slow requests, p95 misses them too; only p99 sees them. Pick the percentile from how many users you can afford to fail.

Across three runs, the 10-user /items line was always avg 26 ms, p50 9 to 10, p95 14 to 16 and p99 410 ms.

3. Ramp up until latency climbs

ramp.sh runs 30 seconds at each user count and prints one line per endpoint per run.

# ramp.sh USERS... - one 30-second run per user count, then one summary line per run for each endpoint
for users in "$@"; do
  sh load.sh "$users" 30 > /dev/null 2>&1
done
python summary.py $(for users in "$@"; do echo "out/locustfile-u${users}_stats.csv"; done) | grep -v Aggregated

$ sh ramp.sh 10 20 40 60 80
run         endpoint      reqs  req/s   avg   p50   p95   p99   max
u10         /items         413   14.2    25    10    15   400   407
u10         /report        125    4.3   110   110   130   210   215
u20         /items         767   26.3    25     9    18   410   407
u20         /report        295   10.1   123   110   200   410   511
u40         /items        1523   52.2    25     9    29   410   425
u40         /report        509   17.4   238   210   440   500   608
u60         /items        1799   61.5    26     9    39   410   437
u60         /report        580   19.8   897   910  1400  1500  2101
u80         /items        1811   62.0    30    10   120   410   509
u80         /report        578   19.8  1819  1900  2200  2300  3789

Read the /report lines from top to bottom. From 10 to 40 users, its throughput rises (4.3, 10.1, 17.4 requests per second) and its median stays near the 100 ms of real work. Then throughput stops at 19.8, which is the pool’s limit: 2 connections times 10 requests per second each. From 60 to 80 users, adding users adds no throughput, only waiting: the median goes from 910 ms to 1.9 s. That bend is the knee. Past it, every extra user joins a queue for a connection.

Two more things in this table. /items has no pool, and its median stayed at 9 to 10 ms throughout, so one saturated endpoint doesn’t slow every endpoint. But /items throughput almost stopped growing too (61.5, then 62.0 requests per second), because the simulated users spent their time waiting on reports instead of sending more requests. The knee was in the same place in all three runs: report throughput 19.8 at both 60 and 80 users, with a median of 810 to 910 ms at 60 users and 1.8 to 1.9 s at 80.

4. “Users” isn’t a load level: wait time

Here’s the same user with no pause between requests.

# locustfile_nowait.py - the same user with no pause: each one sends its next request the moment the last returns
from locust import HttpUser, constant, task


class ApiUser(HttpUser):
    wait_time = constant(0)

    @task(3)
    def list_items(self):
        self.client.get("/items")

    @task(1)
    def open_report(self):
        self.client.get("/report")

$ sh load.sh 10 30 locustfile_nowait.py 2>/dev/null
$ python summary.py out/locustfile-u10_stats.csv out/locustfile_nowait-u10_stats.csv
run         endpoint      reqs  req/s   avg   p50   p95   p99   max
u10         /items         413   14.2    25    10    15   400   407
u10         /report        125    4.3   110   110   130   210   215
u10         Aggregated     538   18.5    45    10   110   400   407
nowait-u10  /items        1655   56.7    24     8    15   400   408
nowait-u10  /report        580   19.9   431   420   500   500   785
nowait-u10  Aggregated    2235   76.6   130     9   490   500   785

The first three lines are the 10-user run from the ramp. With no pause, the same 10 users sent four times as many requests, saturated the report pool (19.9 per second in all three runs) and made reports wait 410 to 420 ms at the median. “10 users” described two very different loads. Always report requests per second next to the user count, and set wait_time from how real users behave.

Look at that report median again: about 400 ms, not seconds. Each Locust user waits for its response before sending the next request, so a slow server automatically slows the load down. Real users don’t stop arriving because the site is slow, so this kind of test (a “closed” model) can make an overloaded system look better than it would be in production. Locust’s constant_throughput(n) wait time, which aims for a fixed number of task runs per second for each user, gets closer to a fixed arrival rate.

5. A latency budget that fails the build

Locust fires a quitting event at the end of a run. A listener can read the final stats and set the exit code.

# locustfile_gate.py - the same user, plus a latency budget: fail the run if p95 is over 300 ms
from locust import HttpUser, constant, events, task

P95_BUDGET_MS = 300


class ApiUser(HttpUser):
    wait_time = constant(0.5)

    @task(3)
    def list_items(self):
        self.client.get("/items")

    @task(1)
    def open_report(self):
        self.client.get("/report")


@events.quitting.add_listener
def check_budget(environment, **kwargs):
    p95 = environment.stats.total.get_response_time_percentile(0.95)
    if p95 > P95_BUDGET_MS:
        print(f"FAIL: p95 {p95:.0f} ms is over the {P95_BUDGET_MS} ms budget")
        environment.process_exit_code = 1
    else:
        print(f"PASS: p95 {p95:.0f} ms is within the {P95_BUDGET_MS} ms budget")

$ sh load.sh 20 30 locustfile_gate.py 2>/dev/null; echo "exit code $?"
PASS: p95 170 ms is within the 300 ms budget
exit code 0
$ sh load.sh 60 30 locustfile_gate.py 2>/dev/null; echo "exit code $?"
FAIL: p95 1100 ms is over the 300 ms budget
exit code 1

At 20 users, below the knee, the run passed and exited 0. At 60 users, past it, p95 was over a second and the run exited 1, which is all a CI system needs to fail the step. In a pipeline you’d run the same command after deploying to a test environment. Over three runs, p95 was 170 to 180 ms at 20 users and 1100 ms every time at 60, so the budget has room on both sides. Set yours from what users need, then check the margin the same way: a budget that passes or fails depending on the run is noise.

The 2>/dev/null drops Locust’s summary, which it prints to stderr. Locust also exits 1 on its own when any request fails, without a listener (Practice 2).

Load-test types

The same locustfile covers every kind of load test. What changes is the number of users, how fast they arrive and how long the run lasts.

Common gotchas

• The load generator shares the machine. Here Locust and the API run on one computer, so they compete for CPU. The numbers are good for comparing runs with each other, not for stating the API’s real capacity. A real test runs the generator on separate machines, close to the system, not over a home connection.
• Percentiles are approximate. Locust buckets response times, so a p99 can read above the max (section 3, /items at 20 users: p99 410, max 407).
• Short runs don’t see rare events. The smoke test never hit the 1-in-25 slow path. Run long enough for each endpoint to collect hundreds of requests.
• Warm-up. Caches, connection pools and JIT compilers start cold. Leave the first minute out of the numbers, or run long enough that it doesn’t matter.
• User counts mislead. Report requests per second, and set wait_time from real behaviour (section 4).
• Only load-test systems you own or have permission to test. A load test against someone else’s server looks like a denial-of-service attack.
• On Git Bash for Windows, Docker volume paths need MSYS_NO_PATHCONV=1, which the scripts set.

Practice

1. Give the report endpoint 4 connections instead of 2 (BoundedSemaphore(4) in server.py), restart the API, and ramp 40, 60 and 80 users. Predict where the report throughput will stop growing.

$ sh start.sh
api listening on :8000
$ sh ramp.sh 40 60 80
run         endpoint      reqs  req/s   avg   p50   p95   p99   max
u40         /items        1596   54.7    25     9    32   400   424
u40         /report        552   18.9   111   100   140   220   314
u60         /items        2326   79.7    25     9    38   410   437
u60         /report        850   29.1   134   110   230   420   611
u80         /items        3135  107.3    27     9    97   400   458
u80         /report       1039   35.6   172   160   300   430   824

2. Make a report that takes over a second count as a failure, using catch_response=True, and run 80 users. What exit code does Locust return?

# locustfile_slow.py - the same user, but a report that takes over a second counts as a failure
from locust import HttpUser, constant, task


class ApiUser(HttpUser):
    wait_time = constant(0.5)

    @task(3)
    def list_items(self):
        self.client.get("/items")

    @task(1)
    def open_report(self):
        with self.client.get("/report", catch_response=True) as response:
            if response.elapsed.total_seconds() > 1:
                response.failure("report took over 1 s")

$ sh load.sh 80 30 locustfile_slow.py; echo "exit code $?"
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /items                                                                          2028     0(0.00%) |     27       6     442      9 |   68.10        0.00
GET      /report                                                                          590  537(91.02%) |   1679     102    3818   1800 |   19.81       18.03
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      2618  537(20.51%) |    399       6    3818     10 |   87.92       18.03

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
GET      /items                                                                                  9     10     11     12     16     83    400    410    440    440    440   2028
GET      /report                                                                              1800   1900   1900   1900   2000   2100   2200   2800   3800   3800   3800    590
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             10     13    400   1200   1800   1900   2000   2100   3500   3800   3800   2618

Error report
# occurrences      Error                                                                                               
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
537                GET /report: report took over 1 s                                                                   
------------------|---------------------------------------------------------------------------------------------------------------------------------------------

exit code 1

3. The total p95 mixes cheap and expensive endpoints. Give each endpoint its own budget, /items 100 ms and /report 150 ms, and run 20 users.

# locustfile_budgets.py - the same user, with a p95 budget for each endpoint instead of one for the total
from locust import HttpUser, constant, events, task

P95_BUDGETS_MS = {"/items": 100, "/report": 150}


class ApiUser(HttpUser):
    wait_time = constant(0.5)

    @task(3)
    def list_items(self):
        self.client.get("/items")

    @task(1)
    def open_report(self):
        self.client.get("/report")


@events.quitting.add_listener
def check_budgets(environment, **kwargs):
    for name, budget in P95_BUDGETS_MS.items():
        p95 = environment.stats.get(name, "GET").get_response_time_percentile(0.95)
        verdict = "PASS" if p95 <= budget else "FAIL"
        print(f"{verdict}: {name} p95 {p95:.0f} ms (budget {budget} ms)")
        if p95 > budget:
            environment.process_exit_code = 1
    total = environment.stats.total.get_response_time_percentile(0.95)
    print(f"for comparison, the total p95 is {total:.0f} ms")

$ sh load.sh 20 30 locustfile_budgets.py 2>/dev/null; echo "exit code $?"
PASS: /items p95 29 ms (budget 100 ms)
FAIL: /report p95 190 ms (budget 150 ms)
for comparison, the total p95 is 160 ms
exit code 1