Bash Networking

The HTTP examples below run against a small local test API on 127.0.0.1:8000, so they work offline and their output is repeatable. Everything was captured with curl 8.5, jq 1.7 and rsync 3.2 on Ubuntu 24.04.

import json, time
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def reply(self, code, body, headers=()):
        self.send_response(code)
        for k, v in headers:
            self.send_header(k, v)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(body).encode())
    def do_GET(self):
        if self.path == "/health":   self.reply(200, {"status": "ok"})
        elif self.path == "/users":  self.reply(200, [{"id": 1, "name": "alice"}, {"id": 2, "name": "bob"}])
        elif self.path == "/old":    self.reply(301, {}, [("Location", "/health")])
        elif self.path == "/slow":   time.sleep(5); self.reply(200, {"slow": True})
        elif self.path == "/me":
            auth = self.headers.get("Authorization", "")
            self.reply(200 if auth == "Bearer t0ken" else 401, {"auth": auth or None})
        else:                        self.reply(404, {"error": "not found"})
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        data = json.loads(self.rfile.read(n) or b"{}")
        if "name" not in data:       self.reply(400, {"error": "name is required"})
        else:                        self.reply(201, {"id": 3, **data})
    def log_message(self, *a): pass

HTTPServer(("127.0.0.1", 8000), H).serve_forever()

curl: Requests, Status Codes, Exit Codes

api=http://127.0.0.1:8000
echo "-- GET:";            curl -s "$api/users"; echo
echo "-- status only:";    curl -s -o /dev/null -w '%{http_code}\n' "$api/nope"
echo "-- redirect:";       curl -s -o /dev/null -w '%{http_code} -> ' "$api/old"; curl -sL "$api/old"; echo
echo "-- header auth:";    curl -s -H "Authorization: Bearer t0ken" "$api/me"; echo
echo "-- exit codes:"
curl -s "$api/nope" > /dev/null;  echo "  curl  on 404: exit $?"
curl -sf "$api/nope" > /dev/null; echo "  curl -f on 404: exit $?"
curl -s http://127.0.0.1:9 > /dev/null; echo "  refused port: exit $?"
curl -s --max-time 1 "$api/slow" > /dev/null; echo "  --max-time 1 on /slow: exit $?"

$ bash curl-basics.sh
-- GET:
[{"id": 1, "name": "alice"}, {"id": 2, "name": "bob"}]
-- status only:
404
-- redirect:
301 -> {"status": "ok"}
-- header auth:
{"auth": "Bearer t0ken"}
-- exit codes:
  curl  on 404: exit 0
  curl -f on 404: exit 22
  refused port: exit 7
  --max-time 1 on /slow: exit 28

The line to remember is exit 0 on a 404. As far as curl is concerned, the transfer worked. -f turns HTTP errors into exit code 22. Exit 7 means the connection was refused, and 28 means a timeout. -s hides the progress meter, -L follows redirects, and -w '%{http_code}' prints the status. Scripts should always set --connect-timeout and --max-time, or a hung server hangs the script too.

Building JSON Safely

api=http://127.0.0.1:8000
name='Alice "Al" O'\''Brien'

echo "-- hand-built JSON:"
curl -s -X POST -H 'Content-Type: application/json' -d "{\"name\":\"$name\"}" "$api/users" \
  -w ' -> %{http_code}\n'

echo "-- jq-built JSON:"
payload=$(jq -nc --arg name "$name" '{name: $name, role: "admin"}')
echo "payload: $payload"
curl -s -X POST -H 'Content-Type: application/json' -d "$payload" "$api/users" \
  -w ' -> %{http_code}\n'

$ bash post-json.sh
-- hand-built JSON:
 -> 000
-- jq-built JSON:
payload: {"name":"Alice \"Al\" O'Brien","role":"admin"}
{"id": 3, "name": "Alice \"Al\" O'Brien", "role": "admin"} -> 201

The quotes in the name produced invalid JSON. The test server’s JSON parser failed and it dropped the connection, so curl reports status 000, meaning no HTTP response at all. jq --arg escapes the value properly. Never build JSON by pasting variables into a string.

A curl Wrapper That Lies

Here is a common “robust” wrapper: capture the body and status together, then check the status.

api_request() {
  local method="$1" url="$2" data="${3:-}" response status body
  response=$(curl -s -w '\n%{http_code}' -X "$method" \
    -H 'Content-Type: application/json' --connect-timeout 2 --max-time 5 \
    ${data:+-d "$data"} "$url")
  status=$(tail -1 <<< "$response")
  body=$(head -n -1 <<< "$response")
  if [[ $status -ge 400 ]]; then
    echo "ERROR: HTTP $status from $url" >&2
    return 1
  fi
  echo "$body"
}
api=http://127.0.0.1:8000
api_request POST "$api/users" '{"name": "Bob Smith"}' && echo "  ^ ok"
api_request POST "$api/users" '{}' || echo "  ^ failed as expected"
out=$(api_request GET http://127.0.0.1:9/users) && echo "server down but 'success': [$out]"

$ bash wrapper.sh
{"id": 3, "name": "Bob Smith"}
  ^ ok
ERROR: HTTP 400 from http://127.0.0.1:8000/users
  ^ failed as expected
server down but 'success': []

HTTP errors are caught. But when the server is down, curl prints status 000, which is not -ge 400, so the wrapper returns success with an empty body. That is the case you most need to detect. Practice task 1 fixes it. Note that ${data:+-d "$data"} does work: the inner quotes keep Bob Smith together as one argument.

Port Checks: nc and /dev/tcp

check_port() { timeout 2 bash -c "echo > /dev/tcp/$1/$2" 2>/dev/null; }
for port in 8000 9; do
  check_port 127.0.0.1 "$port" && echo "/dev/tcp: $port open" || echo "/dev/tcp: $port closed"
  nc -z -w2 127.0.0.1 "$port" && echo "nc -z:    $port open" || echo "nc -z:    $port closed"
done

$ bash ports.sh
/dev/tcp: 8000 open
Connection to 127.0.0.1 8000 port [tcp/*] succeeded!
nc -z:    8000 open
/dev/tcp: 9 closed
nc -z:    9 closed

/dev/tcp/HOST/PORT isn’t a real file. Bash opens a TCP connection when you redirect to it, so it works on machines without nc. It has no timeout of its own, though. A firewalled host can hang it for minutes, which is why it’s wrapped in timeout 2. The OpenBSD nc prints “succeeded!” on stderr. Add 2>/dev/null if you only want the exit status.

Waiting for a service to come up (a container starting, a database booting) is the same check in a loop with a deadline:

wait_for_port() {
  local host="$1" port="$2" max="${3:-10}" start=$SECONDS
  until timeout 1 bash -c "echo > /dev/tcp/$host/$port" 2>/dev/null; do
    (( SECONDS - start >= max )) && { echo "timeout waiting for $host:$port" >&2; return 1; }
    sleep 0.5
  done
  echo "$host:$port ready after $(( SECONDS - start ))s"
}
( sleep 2; nc -l 7000 > /dev/null ) &
wait_for_port 127.0.0.1 7000 10
wait_for_port 127.0.0.1 7001 2 || echo "gave up on 7001"
kill %1 2>/dev/null

$ bash wait-port.sh
127.0.0.1:7000 ready after 2s
timeout waiting for 127.0.0.1:7001
gave up on 7001

SSH Heredocs: Whose Variables?

ssh user@host bash <<EOF sends a script to the remote shell. The heredoc’s quoting decides whether each $ expands locally, before sending, or remotely. Here the same rules are shown with a local bash standing in for the remote one:

app_dir=/srv/myapp
bash <<EOF
echo "unquoted: app_dir=$app_dir host=$(hostname) remote_home=\$HOME"
EOF
bash <<'EOF'
echo "quoted:   app_dir=${app_dir:-<unset>} home=$HOME"
EOF

$ bash heredoc-remote.sh
unquoted: app_dir=/srv/myapp host=devbox remote_home=/home/alice
quoted:   app_dir=<unset> home=/home/alice

With an unquoted EOF, $app_dir and $(hostname) expanded on your machine, and only the escaped \$HOME was left for the remote side. Over SSH, that hostname would print your laptop’s name, not the server’s. With a quoted 'EOF', nothing expands locally, so local variables don’t exist on the remote side. Pass them as arguments instead (ssh host bash -s -- "$app_dir" <<'EOF', then read $1 remotely). For automation, use key-based auth with -o BatchMode=yes -o ConnectTimeout=5, so a missing key fails immediately instead of waiting at a password prompt.

rsync: Trailing Slashes and the Quick Check

mkdir -p src/app && touch src/app/main.sh
mkdir a b
rsync -a src/app  a/
rsync -a src/app/ b/
echo "rsync src/app  a/ -> $(cd a && find . -type f)"
echo "rsync src/app/ b/ -> $(cd b && find . -type f)"

$ bash trailing-slash.sh
rsync src/app  a/ -> ./app/main.sh
rsync src/app/ b/ -> ./main.sh

Without a trailing slash, rsync copies the directory itself. With one, it copies the directory’s contents. A deploy that gets this wrong ends up at /var/www/html/dist/.

mkdir -p dist/css && echo "v1" > dist/index.html && echo "a{}" > dist/css/site.css && echo "x" > dist/debug.log
mkdir -p www && echo "stale" > www/old.html
echo "-- first sync:"
rsync -a --exclude='*.log' --itemize-changes dist/ www/
echo "-- second sync (nothing changed):"
rsync -a --exclude='*.log' --itemize-changes dist/ www/
echo "-- dry run with --delete:"
echo "v2" > dist/index.html
rsync -a --delete --exclude='*.log' --itemize-changes -n dist/ www/
ls www

$ bash rsync.sh
-- first sync:
>f+++++++++ index.html
cd+++++++++ css/
>f+++++++++ css/site.css
-- second sync (nothing changed):
-- dry run with --delete:
*deleting   old.html
css
index.html
old.html

The second sync transferred nothing, which is the point of rsync. -n (dry run) showed that --delete would remove old.html without actually doing it. But the dry run didn’t list index.html, even though it had just changed from v1 to v2. rsync’s quick check compares size and modification time. The new file is the same size, and it was written in the same second as the last sync, so rsync considered it unchanged. Add -c to compare contents (Practice task 2). The same commands work between hosts: rsync -az dist/ user@host:/var/www/html/ runs over SSH.

Practice

1. Fix api_request so a server that is down is reported as a failure, and so only 2xx responses count as success.

   Solution

   api_request() {
     local method="$1" url="$2" data="${3:-}" response status rc
     response=$(curl -s -w '\n%{http_code}' -X "$method" \
       -H 'Content-Type: application/json' --connect-timeout 2 --max-time 5 \
       ${data:+-d "$data"} "$url") || { rc=$?; echo "ERROR: curl exit $rc for $url" >&2; return 1; }
     status=$(tail -1 <<< "$response")
     if [[ ! $status =~ ^2 ]]; then
       echo "ERROR: HTTP $status from $url" >&2
       return 1
     fi
     head -n -1 <<< "$response"
   }
   api_request GET http://127.0.0.1:8000/users && echo "  ^ ok"
   out=$(api_request GET http://127.0.0.1:9/users) || echo "  ^ correctly reported as a failure"

   $ bash p1.sh
   [{"id": 1, "name": "alice"}, {"id": 2, "name": "bob"}]
     ^ ok
   ERROR: curl exit 7 for http://127.0.0.1:9/users
     ^ correctly reported as a failure

2. Get rsync to notice a same-size change made within the same second as the last sync.

   Solution

   mkdir -p dist www && echo "v1" > dist/index.html
   rsync -a dist/ www/
   echo "v2" > dist/index.html
   echo "-- quick check:";  rsync -a  --itemize-changes -n dist/ www/
   echo "-- checksum:";     rsync -ac --itemize-changes -n dist/ www/

   $ bash p2.sh
   -- quick check:
   -- checksum:
   >fc........ index.html

   The c in >fc means “checksum differs”. -c reads every file on both sides, so it is slower on large trees.

3. Print each user’s id and name from /users as tab-separated lines, then the number of users.

   Solution

   curl -s http://127.0.0.1:8000/users | jq -r '.[] | "\(.id)\t\(.name)"'
   count=$(curl -s http://127.0.0.1:8000/users | jq 'length')
   echo "users: $count"

   $ bash p3.sh
   1	alice
   2	bob
   users: 2