Skip to main content
Networking · lesson 7 of 8

Beginner

4

Intermediate

4

Also part of Networking

11
Open the Networking page →
Networking intermediate Lesson 7 of 8

Troubleshooting a Broken Network, Layer by Layer

Debug a Docker Compose app with three planted faults, a DNS typo, the wrong network and the wrong port, using one tool per layer until curl returns 200.

By SkillByExample · Updated

networking troubleshooting docker-compose dns curl debugging

When “the app can’t reach the API”, the fault could be in any layer, and each layer’s tool only sees its own layer. The method that works is a ladder: check name resolution, then the route, then the port, then the application, and stop at the first rung that fails. This lesson plants three faults in a small Docker Compose app and climbs the ladder until the request returns 200. Fixing each fault reveals the next one, the way it usually happens in real life. Tested with Docker Engine 29.8.2, Docker Compose 5.5.1, Python 3.12.15 (python:3.12-slim), Alpine 3.22.6 and curl 8.14.1.

The broken app

The API is a tiny health endpoint on port 8080:

# app.py
from http.server import BaseHTTPRequestHandler, HTTPServer

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        ok = self.path == "/health"
        body = b'{"status": "ok"}\n' if ok else b'{"error": "not found"}\n'
        self.send_response(200 if ok else 404)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("0.0.0.0", 8080), Health).serve_forever()

Its image, in api/Dockerfile, adds ss. The client image, in client/Dockerfile, is Alpine with curl:

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends iproute2 \
    && rm -rf /var/lib/apt/lists/*
COPY app.py /app.py
CMD ["python", "/app.py"]
FROM alpine:3.22
RUN apk add --no-cache curl

The Compose file contains all three faults. Try to spot them before reading on:

# compose.yaml
name: netdebug
services:
  api:
    build: ./api
    networks: [back]
  client:
    build: ./client
    command: sleep infinity
    environment:
      API_URL: http://apii:8000/health
    networks: [front]
networks:
  front:
    internal: true
  back:
    internal: true
docker compose up -d --build

The symptom

$ docker compose exec client sh -c 'curl -sS "$API_URL"'; echo "exit=$?"
curl: (6) Could not resolve host: apii
exit=6

curl’s exit code names the rung. 6 is “couldn’t resolve the host”, so you start at DNS. (7 would be “couldn’t connect”, and 28 would be a timeout.)

Rung 1: does the name resolve?

$ docker compose exec client getent hosts apii; echo "exit=$?"
exit=2
$ docker compose exec client getent hosts client; echo "exit=$?"
172.22.0.2        client  client
exit=0
$ docker compose config --services
api
client

getent hosts asks the same system resolver curl uses. It finds nothing for apii, but it does resolve the client’s own name, so the resolver itself works. The name is what’s wrong. Compose names each service’s DNS entry after the service, and the service is called api. Fault 1 is a typo. Change apii to api in API_URL, and recreate the client:

$ docker compose up -d
$ docker compose exec client sh -c 'curl -sS "$API_URL"'; echo "exit=$?"
curl: (6) Could not resolve host: api
exit=6
$ docker compose exec client getent hosts api; echo "exit=$?"
exit=2

The spelling is right now, and it still doesn’t resolve. Don’t stop at the first thing you fixed: the ladder says to stay on this rung until it passes.

Rung 2: is there a path?

Docker’s DNS only answers for containers that share a network with the one asking. So check which networks each container is on:

$ docker inspect -f "{{.Name}} {{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{$v.IPAddress}}{{end}}" netdebug-api-1 netdebug-client-1
/netdebug-api-1 netdebug_back 172.21.0.2
/netdebug-client-1 netdebug_front 172.22.0.2
$ docker compose exec client ip route
172.22.0.0/16 dev eth0 scope link  src 172.22.0.2
$ docker compose exec client sh -c 'curl -sSv http://172.21.0.2:8000/health 2>&1 | grep -E "Trying|connect to|^curl:"'
*   Trying 172.21.0.2:8000...
* Failed to connect to 172.21.0.2 port 8000 after 0 ms: Could not connect to server
curl: (7) Failed to connect to 172.21.0.2 port 8000 after 0 ms: Could not connect to server

The two containers are on different networks, back and front, in different subnets. The client’s route table knows only its own 172.22.0.0/16, and as an internal network it has no default route, so even the API’s IP address fails in 0 ms. No packet could be sent. The missing DNS answer was a symptom of this rung, not of the name. Fault 2 is the wrong network. Attach the API to both networks, so it stays on back for whatever else lives there:

$ docker compose up -d
$ docker compose exec client getent hosts api; echo "exit=$?"
172.22.0.3        api  api
exit=0

The name now resolves, to the API’s new address on front.

Rung 3: does anything accept the connection?

$ docker compose exec client sh -c 'curl -sSv "$API_URL" 2>&1 | grep -E "Trying|connect to|^curl:|^< HTTP|status"'
*   Trying 172.22.0.3:8000...
* connect to 172.22.0.3 port 8000 from 172.22.0.2 port 41282 failed: Connection refused
* Failed to connect to api port 8000 after 0 ms: Could not connect to server
curl: (7) Failed to connect to api port 8000 after 0 ms: Could not connect to server

Exit code 7 again, with the same vague final message as the route failure. Recent curl versions summarise both as “Could not connect to server”. The -v line underneath has the real error: Connection refused. As the sockets lesson showed, a refusal means the host is up and reachable, so the problem is the port. Ask the API what it’s actually listening on:

$ docker compose exec api ss -tlnp
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      5            0.0.0.0:8080       0.0.0.0:*    users:(("python",pid=1,fd=3))
LISTEN 0      4096      127.0.0.11:33157      0.0.0.0:*

Python is listening on 8080 on every interface. The client was calling 8000. (The second line is Docker’s embedded DNS resolver, which every container on a user-defined network has.) Fault 3 is the wrong port. Here’s the fixed file:

# compose.yaml
name: netdebug
services:
  api:
    build: ./api
    networks: [back, front]
  client:
    build: ./client
    command: sleep infinity
    environment:
      API_URL: http://api:8080/health
    networks: [front]
networks:
  front:
    internal: true
  back:
    internal: true

Rung 4: does the application answer correctly?

$ docker compose up -d
$ docker compose exec client sh -c 'curl -sS -w "HTTP %{http_code}\n" "$API_URL"'
{"status": "ok"}
HTTP 200

200 with the expected body. Each fault was only visible once the one below it was fixed, and each one was found by a different tool: getent for names, docker inspect and ip route for paths, ss for ports, and curl’s status code for the application. Guessing in a different order would have meant, for example, staring at port numbers while the name still didn’t resolve. More on how Docker networks isolate containers in Docker networking, and on route tables in Linux networking basics. When you’re done, run docker compose down.

Practice

  1. A fourth fault, at the top. Change the path in API_URL to /healthz. Which rung fails, and where do you look to prove it?
Solution 
$ docker compose exec client sh -c 'curl -sS -w "HTTP %{http_code}\n" "$API_URL"'
{"error": "not found"}
HTTP 404
$ docker compose logs api
api-1  | 172.22.0.2 - - [13/Oct/2026 19:36:53] "GET /health HTTP/1.1" 200 -
api-1  | 172.22.0.2 - - [13/Oct/2026 19:37:04] "GET /healthz HTTP/1.1" 404 -

The application rung. A 404 is an HTTP answer, so DNS, the route and the port all worked. The server’s access log shows the exact path it received: the earlier /health got 200, and /healthz got 404. Both came from 172.22.0.2, the client’s address on front. No network tool would have found this, and none was needed.

  1. Turn the ladder into a script. Write ladder.sh that checks DNS, then the port, then the status code, and prints the first rung that fails. Run it against the broken file, each partial fix, and the final file.
Solution 
# ladder.sh
host=$(echo "$API_URL" | sed -E 's#^[a-z]+://([^:/]+).*#\1#')
port=$(echo "$API_URL" | sed -E 's#^[a-z]+://[^:/]+:([0-9]+).*#\1#')
getent hosts "$host" > /dev/null || { echo "DNS:  $host does not resolve"; exit 1; }
nc -z -w 2 "$host" "$port" || { echo "PORT: nothing accepts $host:$port"; exit 2; }
code=$(curl -s -o /dev/null -w '%{http_code}' "$API_URL")
[ "$code" = 200 ] || { echo "APP:  $API_URL returned $code"; exit 3; }
echo "OK:   $API_URL returned 200"

-T lets the script be piped into the container’s shell on standard input. Each block below is the result with a different version of compose.yaml: the original, with the typo fixed, with the network fixed, fully fixed, and with the /healthz path from task 1.

$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
DNS:  apii does not resolve
exit=1
$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
DNS:  api does not resolve
exit=1
$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
PORT: nothing accepts api:8000
exit=2
$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
OK:   http://api:8080/health returned 200
exit=0
$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
APP:  http://api:8080/healthz returned 404
exit=3

On Docker, the wrong network shows up at the DNS rung, because the embedded resolver only answers for shared networks, so a script can’t tell it apart from a typo. That’s why you still need the docker inspect step. A distinct exit code per rung lets a health check or CI job report which layer broke, not just that something did.

  1. The firewall version. With everything fixed, drop port 8080 inside the API’s network namespace, using the drop.nft idea from the firewall lesson. Predict curl’s exit code and what ladder.sh prints, then check. The fw-lab image from that lesson can join the API container’s namespace with --network container:netdebug-api-1.
Solution 
# drop8080.nft
table inet lab
delete table inet lab
table inet lab {
    chain input {
        type filter hook input priority 0; policy accept;
        tcp dport 8080 counter drop
    }
}
$ docker run --rm --network container:netdebug-api-1 --cap-add NET_ADMIN -v "$PWD":/work fw-lab nft -f drop8080.nft
$ docker compose exec client sh -c 'time curl -sS --connect-timeout 3 "$API_URL"; echo "exit=$?"'
curl: (28) Connection timed out after 3003 milliseconds
Command exited with non-zero status 28
real	0m 3.00s
user	0m 0.00s
sys	0m 0.00s
exit=28
$ docker compose exec -T client sh < ladder.sh; echo "exit=$?"
PORT: nothing accepts api:8080
exit=2

DNS and the route still pass, and the port rung fails again, but as silence, not a refusal. curl waits out its connect timeout and exits 28 instead of 7. ladder.sh prints the same PORT message it gave for the wrong port, because nc -z only reports failure. To tell the two apart, time the check: a refusal fails in milliseconds, and a drop takes the whole timeout. The rule lives in the API container’s namespace, so docker compose down removes it along with the container.

Frequently Asked Questions

What order should I check network problems in?
Work up the layers: first name resolution (does the name turn into an address?), then routing (is there a path to that address?), then the port (does anything accept a connection there?), and finally the application (does it return the response you expect?). Each layer depends on the one below it, so a fault low down hides everything above it, and fixing it often reveals the next fault.
Why can't one Docker container resolve another container's name?
Docker's embedded DNS only answers names of containers that share a user-defined network with the asking container. If two Compose services are on different networks, the name doesn't resolve even when it's spelled correctly. Check with docker inspect which networks each container is attached to, and attach both services to a common network.
What does curl error 6, 7 or 28 mean?
curl exit code 6 means it could not resolve the host name, so DNS is the problem. Exit code 7 means it failed to connect: the name resolved but the TCP connection was refused or had no route. Exit code 28 means an operation timed out, which often points to a firewall drop. Run curl with -v to see the underlying error, such as Connection refused.
How do I see which port a service is really listening on in a container?
Run ss -tlnp inside the container, for example docker compose exec api ss -tlnp. It lists listening TCP sockets with their local address, port and owning process. If the client connects to a different port than the one shown, you get Connection refused. Slim images may need the iproute2 package installed for ss.
Is a 404 a network problem?
No. A 404 Not Found is an HTTP response, so DNS, routing and the TCP connection all worked and the application itself answered. The fault is in the request, usually the path, or in the application's routing. Check the server's access log to see exactly which path it received.