Sync vs Async Endpoints in FastAPI, Measured

What This Measures

The usual advice is “use async def for I/O, def for everything else”, usually with no numbers attached. This lesson runs three versions of the same 100 ms wait under real concurrent load and records requests per second and latency for each. It also measures what the advice leaves out: the thread pool behind def routes has a size, a single blocking call can freeze every other route, and async does nothing for CPU-bound work.

pip install fastapi "uvicorn[standard]"
sudo apt install apache2-utils      # provides ab; macOS already ships it

The numbers below come from fastapi 0.143.0, starlette 1.7.0, anyio 4.15.1 and uvicorn 0.54.0 with uvloop, on Python 3.12, in a Docker container limited to 4 CPUs. ApacheBench (ab) ran in the same container. Expect different absolute numbers on your machine. The ratios between them are the point.

Three Ways to Wait 100 ms

bench.py has three routes that each spend 100 ms waiting, written three different ways, plus a route that does nothing:

import asyncio
import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/sync")                  # plain def: FastAPI runs it in a worker thread
def sync_endpoint():
    time.sleep(0.1)                # stands in for a blocking call: requests.get, psycopg2, open()
    return {"ok": True}

@app.get("/async-blocking")        # async def that still makes a blocking call
async def async_blocking():
    time.sleep(0.1)
    return {"ok": True}

@app.get("/async")                 # async def with a non-blocking wait
async def async_endpoint():
    await asyncio.sleep(0.1)       # stands in for an awaited call: httpx, asyncpg
    return {"ok": True}

@app.get("/health")
async def health():
    return {"status": "ok"}

Every run in this lesson starts uvicorn with --log-level warning, which turns off the per-request access log so logging doesn’t skew the numbers. Start uvicorn bench:app --log-level warning and take a baseline from the route that does no work (-k reuses connections, -c 100 keeps 100 requests in flight):

$ ab -q -k -n 2000 -c 100 http://127.0.0.1:8000/health | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   0.458 seconds
Failed requests:        0
Requests per second:    4366.31 [#/sec] (mean)
  50%     20
  99%     64

One uvicorn worker handles about 4,400 trivial requests per second here, with a median latency of 20 ms (the 50% row, in milliseconds) that is all queueing behind 100 concurrent clients. FastAPI’s own overhead is about 0.2 ms per request, so it won’t be the bottleneck in anything that follows.

async def with an Awaited Wait

$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/async | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   1.243 seconds
Failed requests:        0
Requests per second:    804.54 [#/sec] (mean)
  50%    110
  99%    134

The ceiling is 100 clients each waiting 0.1 s, so 1,000 requests per second, and one process with one thread got 805 of them. Median latency is 110 ms, only 10 ms above the sleep itself. await asyncio.sleep(0.1) hands control back to the event loop, which starts the other 99 requests while the first one waits. This is the case async is built for: many requests, each mostly waiting.

def: Fast Until the Thread Pool Runs Out

$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   2.731 seconds
Failed requests:        0
Requests per second:    366.22 [#/sec] (mean)
  50%    239
  99%    331

time.sleep is blocking, but this route still served 366 requests per second, because FastAPI runs every plain def route in a worker thread so it can’t block the event loop. The pool has a size, though: 40 threads, so at most 40 sleeps run at once. 40 threads × 10 sleeps per second gives a ceiling of 400 requests per second, and 366 is close to it. The other 60 clients wait in line, which is why median latency more than doubled to 239 ms for the same 100 ms of work.

The Failure: async def Around a Blocking Call

/async-blocking is the version people write when they’ve heard “async is faster”: async def on the outside, a blocking call on the inside. It gets only 10 concurrent clients and 100 requests here, because more would take minutes:

$ ab -q -k -n 100 -c 10 http://127.0.0.1:8000/async-blocking | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   10.195 seconds
Failed requests:        0
Requests per second:    9.81 [#/sec] (mean)
  50%   1019
  99%   1022

9.81 requests per second is one request at a time: 100 requests × 0.1 s = 10 s, and the run took 10.195 s. An async def route runs directly on the event loop, and time.sleep never yields, so the loop can’t start anything else until the sleep returns. Each client waited behind the nine others, which is why the median is 1,019 ms. More clients can’t help, because one event loop doing a blocking 0.1 s call is capped at 1 / 0.1 s = 10 requests per second. The same route at the 100 clients the other routes faced:

$ ab -q -k -n 200 -c 100 http://127.0.0.1:8000/async-blocking | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   20.352 seconds
Failed requests:        0
Requests per second:    9.83 [#/sec] (mean)
  50%   7559
  99%  10156

Throughput stayed at 9.83 requests per second, and the extra clients only lengthened the queue: the median went to 7.6 seconds. At equal concurrency that is 37 times slower than plain def (366) and 82 times slower than /async (805).

It also affects routes that have nothing to do with it. Keep that load running in one terminal and time /health from another, then do the same while /sync is under load:

$ ab -q -k -n 100 -c 10 http://127.0.0.1:8000/async-blocking > /dev/null &
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.988045s
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.972208s
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.973480s

$ ab -q -k -n 100 -c 10 http://127.0.0.1:8000/sync > /dev/null &
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.000966s
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.000659s
$ curl -s -o /dev/null -w '%{http_code} in %{time_total}s\n' 127.0.0.1:8000/health
200 in 0.000693s

The health check does no work, yet it took almost a second because it queued behind ten blocking sleeps on the one event loop. Under the same load on /sync it answered in under a millisecond, because the sleeps were in threads and the loop was free. In production, a Kubernetes liveness probe with a 1-second timeout would start failing here, and the pod would be restarted for being slow even though nothing had crashed. The fix is to never call blocking code from async def: use def, or an async library (httpx.AsyncClient instead of requests, asyncpg instead of psycopg2), or push the call to a thread, as Practice 1 does.

Raising the 40-Thread Limit

The 40 comes from AnyIO, which Starlette uses to run def routes. threads.py reads the limit at startup and lets an environment variable change it:

import os
import time
from contextlib import asynccontextmanager
import anyio.to_thread
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI):
    limiter = anyio.to_thread.current_default_thread_limiter()
    print("default thread limit:", limiter.total_tokens)
    limiter.total_tokens = int(os.environ.get("THREADS", limiter.total_tokens))
    print("thread limit now:", limiter.total_tokens)
    yield

app = FastAPI(lifespan=lifespan)

@app.get("/sync")
def sync_endpoint():
    time.sleep(0.1)
    return {"ok": True}

@app.get("/health")
async def health():
    return {"status": "ok"}

First with the default, then with THREADS=100, both with the access log off as before:

$ uvicorn threads:app --log-level warning
default thread limit: 40
thread limit now: 40
$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   2.784 seconds
Failed requests:        0
Requests per second:    359.14 [#/sec] (mean)
  50%    264
  99%    410

$ THREADS=100 uvicorn threads:app --log-level warning
default thread limit: 40
thread limit now: 100
$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   1.332 seconds
Failed requests:        0
Requests per second:    750.62 [#/sec] (mean)
  50%    106
  99%    265

With 100 threads for 100 clients, nobody queues for a thread. Throughput doubled from 359 to 751 requests per second, and the median fell from 264 ms to 106 ms, close to /async’s 805 and 110 ms. The p99 of 265 ms, against 134 ms for /async, is what remains, likely because 100 OS threads take turns on the GIL and the OS scheduler where 100 coroutines don’t. One run can’t separate those causes. Raise the limit only when the blocking calls are I/O waits, and remember what is behind them: 100 threads each holding a database connection need a pool and a server that allow 100 connections.

CPU-Bound Work: async Doesn’t Help, Processes Do

cpu.py replaces the wait with arithmetic, once as def and once as async def:

from fastapi import FastAPI

app = FastAPI()

def work():
    return sum(i * i for i in range(200_000))     # pure-Python CPU work, holds the GIL

@app.get("/cpu-sync")
def cpu_sync():
    return {"result": work()}

@app.get("/cpu-async")
async def cpu_async():
    return {"result": work()}

@app.get("/health")
async def health():
    return {"status": "ok"}

Time work() on its own, then load both routes on one worker with uvicorn cpu:app --log-level warning:

$ python -c "import time; from cpu import work; t = time.perf_counter(); work(); print(f\"{(time.perf_counter() - t) * 1000:.1f} ms\")"
12.7 ms
$ ab -q -k -n 200 -c 10 http://127.0.0.1:8000/cpu-sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   2.863 seconds
Failed requests:        0
Requests per second:    69.86 [#/sec] (mean)
  50%    139
  99%    212
$ ab -q -k -n 200 -c 10 http://127.0.0.1:8000/cpu-async | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   2.896 seconds
Failed requests:        0
Requests per second:    69.05 [#/sec] (mean)
  50%    143
  99%    158

69.86 and 69.05 requests per second: the same throughput, about one work() call (12.7 ms plus request handling) at a time. The async def version blocks the loop just as time.sleep did, because computing never awaits. The def version runs in 40 threads, but CPython’s GIL lets only one thread run Python code at a time, so the threads take turns on one core. To use more cores you need more processes, which is what --workers starts:

Run the server in one terminal. With the access log off, it prints nothing while it runs:

$ uvicorn cpu:app --workers 4 --log-level warning

Then, from a second terminal, list the processes and rerun the load:

$ ps -eo pid,ppid,args
    PID    PPID COMMAND
      1       0 /usr/local/bin/python3.12 /usr/local/bin/uvicorn cpu:app --workers 4 --log-level warning
      7       1 /usr/local/bin/python3.12 -c from multiprocessing.resource_tracker import main;main(6)
      8       1 /usr/local/bin/python3.12 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=9) --multiprocessing-fork
      9       1 /usr/local/bin/python3.12 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=13) --multiprocessing-fork
     10       1 /usr/local/bin/python3.12 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=17) --multiprocessing-fork
     11       1 /usr/local/bin/python3.12 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=21) --multiprocessing-fork
     28       0 ps -eo pid,ppid,args
$ ab -q -k -n 200 -c 10 http://127.0.0.1:8000/cpu-sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   1.108 seconds
Failed requests:        0
Requests per second:    180.54 [#/sec] (mean)
  50%     53
  99%    116
$ ab -q -k -n 200 -c 10 http://127.0.0.1:8000/cpu-async | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   0.994 seconds
Failed requests:        0
Requests per second:    201.24 [#/sec] (mean)
  50%     49
  99%     80

The server ran as the container’s main process, which is why uvicorn’s parent is pid 1 here; on a normal machine it gets an ordinary pid. The parent only supervises. Its children are a multiprocessing resource tracker (pid 7) and four spawned workers (8-11), each with its own interpreter, GIL and event loop. Throughput went from about 70 to 181-201 requests per second, roughly 2.6-2.9× rather than 4×, because the container’s four CPUs were also running ab. Every per-process resource is multiplied too: four workers each open their own database pool and their own 40-thread pool.

What the Numbers Say

Use async def only when everything slow inside it is awaited. Use def for blocking libraries, and size the thread pool to the I/O concurrency you need. Use worker processes for CPU. The one combination to avoid entirely is a blocking call inside async def, which caps a worker at one request at a time (10 per second for a 100 ms call, however many clients connect) and slows down every other route as well. Parallel programming covers the GIL-versus-processes tradeoff outside web servers.

Practice

1. A library you depend on only has a blocking API, and the route must stay async def. Add two routes that push time.sleep(0.1) to a thread, one with asyncio.to_thread and one with starlette.concurrency.run_in_threadpool. Predict which is faster at 100 clients, then measure.

import asyncio
import time
from fastapi import FastAPI
from starlette.concurrency import run_in_threadpool

app = FastAPI()

@app.get("/to-thread")
async def to_thread():
    await asyncio.to_thread(time.sleep, 0.1)       # the event loop's default executor
    return {"ok": True}

@app.get("/threadpool")
async def threadpool():
    await run_in_threadpool(time.sleep, 0.1)       # the same pool FastAPI uses for def routes
    return {"ok": True}

@app.get("/health")
async def health():
    return {"status": "ok"}

$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/to-thread | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   6.485 seconds
Failed requests:        0
Requests per second:    154.19 [#/sec] (mean)
  50%    609
  99%    662
$ ab -q -k -n 1000 -c 100 http://127.0.0.1:8000/threadpool | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   2.735 seconds
Failed requests:        0
Requests per second:    365.68 [#/sec] (mean)
  50%    247
  99%    327
$ python -c "import os; print('cpu_count', os.cpu_count(), 'default executor', min(32, os.cpu_count() + 4))"
cpu_count 12 default executor 16

2. Rerun /async and /sync from bench.py with 200 concurrent clients and 2,000 requests. Predict each route’s requests per second from its ceiling first.

$ ab -q -k -n 2000 -c 200 http://127.0.0.1:8000/async | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   1.329 seconds
Failed requests:        0
Requests per second:    1504.60 [#/sec] (mean)
  50%    110
  99%    234
$ ab -q -k -n 2000 -c 200 http://127.0.0.1:8000/sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   5.288 seconds
Failed requests:        0
Requests per second:    378.20 [#/sec] (mean)
  50%    510
  99%    548

3. Can more processes rescue /async-blocking? Start bench.py with uvicorn bench:app --workers 2 --log-level warning and rerun the 10-client test. Predict the requests per second.

$ ab -q -k -n 100 -c 10 http://127.0.0.1:8000/async-blocking | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   5.197 seconds
Failed requests:        0
Requests per second:    19.24 [#/sec] (mean)
  50%    509
  99%    511

4. Start cpu.py with uvicorn cpu:app --workers 2 --log-level warning and measure /cpu-sync at 10 clients. Does it get closer to a perfect 2× than four workers got to 4×?

$ ab -q -k -n 200 -c 10 http://127.0.0.1:8000/cpu-sync | grep -E 'Failed|Time taken|Requests per|  50%|  99%'
Time taken for tests:   1.687 seconds
Failed requests:        0
Requests per second:    118.58 [#/sec] (mean)
  50%     80
  99%    150