What does an API Gateway actually do, mechanically?

It's a service whose job is making other services' calls on behalf of a client and combining the results — the client makes one request to the gateway instead of one request per backend service. This lesson's `gateway_dashboard` function is the whole pattern in miniature.

Isn't a fallback just hiding a real outage?

It's a deliberate tradeoff, not a coverup — a dashboard showing `{'orders': 0}` because orders-service is slow is more useful to a user than the entire page failing to load. Practice 4 in this lesson shows the opposite choice (fail the whole request) for cases where a fallback value would be actively wrong, like a payment confirmation.

Why is one backend's timeout (0.5s) different from another's (0.3s)?

Because not every dependency deserves the same patience. A user-profile lookup that's merely decorative can wait less time than a call whose data is more central to the page — this lesson picks different budgets for `user` and `orders` on purpose, not by accident.

Would a real gateway make these calls one at a time like this example does?

No — a production gateway typically fires backend calls concurrently (with `asyncio`, threads, or a thread pool) so the total latency is the slowest single call, not the sum of all of them. This lesson keeps the calls sequential for clarity; that concurrency upgrade is a natural next exercise once the fallback logic itself is understood.

API Gateway: Aggregating Calls With a Fallback

A dashboard page usually needs data from more than one service — a gateway is the one place that calls all of them and decides what to do when one is too slow.

Combine two calls, fall back on the slow one

import http.server
import json
import threading
import time
import urllib.error
import urllib.request

class UserHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({"name": "Asha"}).encode())

    def log_message(self, *args):
        pass

class SlowOrdersHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(1)
        try:
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({"orders": 3}).encode())
        except OSError:
            pass  # the gateway already gave up and closed the connection

    def log_message(self, *args):
        pass

def fetch_json(url, timeout, fallback):
    try:
        response = urllib.request.urlopen(url, timeout=timeout)
        return json.loads(response.read())
    except (urllib.error.URLError, TimeoutError):
        print(f"  {url} timed out, using fallback {fallback}")
        return fallback

def gateway_dashboard():
    user = fetch_json("http://localhost:8003/user", timeout=0.5, fallback={"name": "unknown"})
    orders = fetch_json("http://localhost:8004/orders", timeout=0.3, fallback={"orders": 0})
    return {"user": user, "orders": orders}

user_server = http.server.HTTPServer(("localhost", 8003), UserHandler)
orders_server = http.server.HTTPServer(("localhost", 8004), SlowOrdersHandler)
threading.Thread(target=user_server.serve_forever, daemon=True).start()
threading.Thread(target=orders_server.serve_forever, daemon=True).start()
time.sleep(0.2)

print("Requesting aggregated dashboard...")
dashboard = gateway_dashboard()
print(f"Gateway response: {dashboard}")

user_server.shutdown()
orders_server.shutdown()

$ python api_gateway.py
Requesting aggregated dashboard...
  http://localhost:8004/orders timed out, using fallback {'orders': 0}
Gateway response: {'user': {'name': 'Asha'}, 'orders': {'orders': 0}}

user-service answers immediately, so user comes back real. orders-service sleeps for a full second before it even tries to respond, but the gateway only gives it 0.3 seconds — so fetch_json gives up, prints that it’s falling back, and the gateway’s final response substitutes {'orders': 0} instead of failing the whole request. The slow handler’s own try/except OSError around its response matters too: by the time it wakes up and tries to write, the gateway has already closed that connection, and without the except this would print a noisy, irrelevant traceback in the server thread.

Practice

1. Add a third backend service (reviews) and aggregate all three into one dashboard.

import http.server
import json
import threading
import time

class Handler(http.server.BaseHTTPRequestHandler):
    payload = {}

    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(self.payload).encode())

    def log_message(self, *args):
        pass

def make_handler(payload):
    class H(Handler):
        pass
    H.payload = payload
    return H

user_server = http.server.HTTPServer(("localhost", 8020), make_handler({"name": "Asha"}))
orders_server = http.server.HTTPServer(("localhost", 8021), make_handler({"orders": 3}))
reviews_server = http.server.HTTPServer(("localhost", 8022), make_handler({"reviews": 12}))
for s in (user_server, orders_server, reviews_server):
    threading.Thread(target=s.serve_forever, daemon=True).start()
time.sleep(0.2)

dashboard = {
    "user": fetch_json("http://localhost:8020/user", 0.5, {"name": "unknown"}),
    "orders": fetch_json("http://localhost:8021/orders", 0.5, {"orders": 0}),
    "reviews": fetch_json("http://localhost:8022/reviews", 0.5, {"reviews": 0}),
}
print(f"Gateway response: {dashboard}")

for s in (user_server, orders_server, reviews_server):
    s.shutdown()

$ python three_services.py
Gateway response: {'user': {'name': 'Asha'}, 'orders': {'orders': 3}, 'reviews': {'reviews': 12}}

make_handler is a small factory that stashes a different payload on a fresh subclass per service — three services, three ports, one shared fetch_json doing the aggregation.

2. Make both backends slow at once and confirm both fallbacks trigger independently.

class SlowHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(1)
        try:
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"{}")
        except OSError:
            pass

    def log_message(self, *args):
        pass

s1 = http.server.HTTPServer(("localhost", 8023), SlowHandler)
s2 = http.server.HTTPServer(("localhost", 8024), SlowHandler)
threading.Thread(target=s1.serve_forever, daemon=True).start()
threading.Thread(target=s2.serve_forever, daemon=True).start()
time.sleep(0.2)

dashboard = {
    "user": fetch_json("http://localhost:8023/user", 0.3, {"name": "unknown"}),
    "orders": fetch_json("http://localhost:8024/orders", 0.3, {"orders": 0}),
}
print(f"Gateway response: {dashboard}")

s1.shutdown()
s2.shutdown()

$ python both_slow.py
  http://localhost:8023/user timed out, using fallback {'name': 'unknown'}
  http://localhost:8024/orders timed out, using fallback {'orders': 0}
Gateway response: {'user': {'name': 'unknown'}, 'orders': {'orders': 0}}

Each fetch_json call fails on its own terms — nothing in the gateway function needs to change for two failures instead of one, because the fallback logic already lives at the level of a single call, not the whole aggregation.

3. Measure and print how long a real gateway call actually takes.

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"{}")

    def log_message(self, *args):
        pass

server = http.server.HTTPServer(("localhost", 8025), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
time.sleep(0.2)

start = time.perf_counter()
result = fetch_json("http://localhost:8025/x", 0.5, {})
elapsed_ms = (time.perf_counter() - start) * 1000
print(f"result: {result}, took {elapsed_ms:.1f}ms")

server.shutdown()

$ python measure_latency.py
result: {}, took 555.9ms

The handler itself does no work at all, yet a single loopback HTTP round trip on this machine still costs over half a second — measuring the real number here matters precisely because it’s so far from the “should be a few milliseconds” instinct. Whatever the actual overhead turns out to be on your machine, it’s real production behavior a “should be fast” assumption would miss.

4. Make a critical call fail the whole request instead of silently falling back.

class CriticalDependencyError(Exception):
    pass

def fetch_json_strict(url, timeout):
    try:
        response = urllib.request.urlopen(url, timeout=timeout)
        return json.loads(response.read())
    except (urllib.error.URLError, TimeoutError) as e:
        raise CriticalDependencyError(f"{url} is unavailable") from e

class SlowHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(1)
        try:
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"{}")
        except OSError:
            pass

    def log_message(self, *args):
        pass

server = http.server.HTTPServer(("localhost", 8026), SlowHandler)
threading.Thread(target=server.serve_forever, daemon=True).start()
time.sleep(0.2)

try:
    fetch_json_strict("http://localhost:8026/payment-status", timeout=0.3)
except CriticalDependencyError as e:
    print(f"Gateway refuses to respond: {e}")

server.shutdown()

$ python strict_mode.py
Gateway refuses to respond: http://localhost:8026/payment-status is unavailable

A fallback value for payment-status would be worse than no answer at all — showing “not paid” when the truth is merely “unknown” could trigger a duplicate charge somewhere downstream. raise ... from e preserves the original timeout as the traceback’s cause while still giving the caller a gateway-specific exception to catch.

Next: a multi-step transaction across services, and what happens when one step in the middle fails.

Combine two calls, fall back on the slow one

Practice

Frequently Asked Questions