Bash DevOps Scripts

Every script below was run in an Ubuntu 24.04 container with bash 5.2, GNU tar and coreutils. The health check runs against the same small local test API as the networking lesson. Four of the six are production scripts as they’re commonly written, and each one has a real bug that shows up in its output.

Deploy: Releases, Atomic Switch, Rollback

A releases directory keeps each version on disk, and a current symlink points at the live one. Rollback is just moving the symlink back. In this runnable version, the “build” step writes a VERSION file, and the “health check” confirms that current serves the new version.

set -euo pipefail
readonly DEPLOY_DIR="$PWD/app" KEEP=3
readonly RELEASES="$DEPLOY_DIR/releases" CURRENT="$DEPLOY_DIR/current"
log() { echo "[deploy] $*"; }

deploy() {
  local version="$1" rel="$RELEASES/$1"
  mkdir -p "$rel"
  echo "$version" > "$rel/VERSION"                       # stands in for clone + build
  ln -sfn "$rel" "$CURRENT.next" && mv -Tf "$CURRENT.next" "$CURRENT"
  [[ $(cat "$CURRENT/VERSION") == "$version" ]] || { log "health check failed"; return 1; }
  log "live: $version"
  ls -1 "$RELEASES" | sort | head -n -"$KEEP" | while IFS= read -r old; do
    rm -rf -- "${RELEASES:?}/$old" && log "pruned $old"
  done
}
rollback() {
  local prev
  prev=$(ls -1 "$RELEASES" | sort | grep -B1 -x "$(basename "$(readlink "$CURRENT")")" | head -1)
  [[ -n $prev && $prev != "$(basename "$(readlink "$CURRENT")")" ]] || { log "nothing to roll back to"; return 1; }
  ln -sfn "$RELEASES/$prev" "$CURRENT.next" && mv -Tf "$CURRENT.next" "$CURRENT"
  log "rolled back to $prev"
}

for v in v001 v002 v003 v004; do deploy "$v"; done
echo "releases on disk: $(ls "$RELEASES" | paste -sd' ')"
rollback
echo "current -> $(readlink "$CURRENT" | sed "s#$PWD/##")"

$ bash deploy.sh
[deploy] live: v001
[deploy] live: v002
[deploy] live: v003
[deploy] live: v004
[deploy] pruned v001
releases on disk: v002 v003 v004
[deploy] rolled back to v003
current -> app/releases/v003

The switch creates current.next, then renames it over current with mv -T. A rename is a single atomic step, so a request never finds current missing or half-updated. ln -sfn straight onto current would unlink the old link and then create the new one, leaving a brief gap. head -n -3 lists everything except the newest three releases for pruning. The names sort by version because they are zero-padded. Timestamped release names such as 20250301_1430 sort the same way. In a real deploy, git clone --depth=1, the build, the migrations, systemctl restart and a curl -sf health poll go where the VERSION line is.

Backup: When the Log Ends Up in the Filename

A backup function returns the path of its archive by printing it, and it logs as it works:

set -euo pipefail
log() { echo "[$(date -u +%H:%M:%S)] $*" | tee -a backup.log; }
backup_files() {
  local archive="uploads_backup.tar.gz"
  log "Backing up files: $1"
  tar -czf "$archive" "$1"
  echo "$archive"
}
mkdir -p uploads && echo data > uploads/a.txt
archive=$(backup_files uploads)
echo "archive variable holds $(wc -l <<< "$archive") lines:"
printf '  [%s]\n' "$archive" | sed 's/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/HH:MM:SS/'
ls "$archive" 2>&1 | sed 's/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/HH:MM:SS/'

$ bash capture-bug.sh
archive variable holds 2 lines:
  [[HH:MM:SS] Backing up files: uploads
uploads_backup.tar.gz]
ls: cannot access '[HH:MM:SS] Backing up files: uploads'$'\n''uploads_backup.tar.gz': No such file or directory

tee writes to stdout, and $(...) captures all of stdout, so the log line became part of the “file name”. The next step (aws s3 cp "$archive" ... in a real script) fails on a path that doesn’t exist, and with set -e the backup job dies. The sed only masks the time. The fix is one redirect (Practice task 1): log helpers write to stderr.

tar —exclude Must Come First

mkdir -p site/tmp site/.git && echo page > site/index.html && echo x > site/app.log && echo t > site/tmp/cache && echo g > site/.git/HEAD
echo "-- excludes after the directory:"
tar -czf a.tar.gz site --exclude="*.log" --exclude="tmp" --exclude=".git"
tar -tzf a.tar.gz | sort
echo "-- excludes first:"
tar -czf b.tar.gz --exclude="*.log" --exclude="tmp" --exclude=".git" site
tar -tzf b.tar.gz | sort

$ bash tar-exclude.sh
-- excludes after the directory:
tar: The following options were used after non-option arguments.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
tar: --exclude ‘*.log’ has no effect
tar: --exclude ‘tmp’ has no effect
tar: --exclude ‘.git’ has no effect
tar: Exiting with failure status due to previous errors
site/
site/.git/
site/.git/HEAD
site/app.log
site/index.html
site/tmp/
site/tmp/cache
-- excludes first:
site/
site/index.html

Current GNU tar (1.30 and later) applies --exclude only to the paths that come after it. With the excludes last, the archive included .git and every log, and tar exited non-zero. Older releases such as 1.27 reordered options and honoured a trailing --exclude, which is how this ordering got into so many scripts that later broke on an upgrade. Put options before paths and the script behaves the same on both.

Log Rotation: cp file - Is Not stdout

This rotation step appears in many scripts:

mkdir -p archive
seq 1 5000 > app.log
before=$(wc -l < app.log)
cp app.log - | gzip -9 > archive/app.log.gz
truncate -s 0 app.log
echo "log had $before lines; archive holds $(zcat archive/app.log.gz | wc -l) lines"
ls
echo "the file named '-' has $(wc -l < ./-) lines"

$ bash rotate-bug.sh
log had 5000 lines; archive holds 0 lines
-
app.log
archive
rotate-bug.sh
the file named '-' has 5000 lines

The archive is empty, and the log was then truncated, so 5,000 lines were lost. cp doesn’t treat - as stdout. It created a file literally named -, and gzip compressed empty input. gzip -c app.log > archive is the correct form (Practice task 2). Truncating instead of deleting is right, because the application keeps its file descriptor open. For real systems, logrotate with copytruncate handles this for you.

Health Check: One Line per Check

A check helper runs any command and reports it the same way. The script’s exit status tells the monitoring system the result.

set -uo pipefail
failures=0 checks=0
check() {
  local name="$1"; shift
  (( ++checks ))
  if "$@" &>/dev/null; then
    printf '[OK]   %s\n' "$name"
  else
    printf '[FAIL] %s\n' "$name"
    (( ++failures ))
  fi
}
check "API /health"      curl -sf --max-time 2 http://127.0.0.1:8000/health
check "API /ready"       curl -sf --max-time 2 http://127.0.0.1:8000/ready
check "port 8000 open"   nc -z -w1 127.0.0.1 8000
check "worker running"   pgrep -f my-worker
check "disk below 90%"   test "$(df --output=pcent / | tail -1 | tr -dc 0-9)" -lt 90
echo "Results: $(( checks - failures ))/$checks passed"
(( failures == 0 )) && echo HEALTHY || { echo "UNHEALTHY"; exit 1; }

$ bash health.sh
[OK]   API /health
[FAIL] API /ready
[OK]   port 8000 open
[FAIL] worker running
[OK]   disk below 90%
Results: 3/5 passed
UNHEALTHY

The test API has no /ready endpoint (so curl -f fails on the 404) and no worker process, so two checks failed and the script exited 1. The script leaves out set -e on purpose, so one failed check can’t stop the rest from running. The ++x pre-increment keeps the counter’s first step from returning status 1. pg_isready, redis-cli ping and systemctl is-active --quiet all plug into check the same way.

CI Helpers and the Swallowed Failure

Helpers for timing steps and grouping log output make pipeline logs easier to read. GitHub Actions folds everything between ##[group] and ##[endgroup] into a collapsible section.

ci_section() { printf '\n##[group]%s\n' "$*"; }
ci_end()     { echo "##[endgroup]"; }
timed() {
  local start=$SECONDS rc=0
  "$@" || rc=$?
  echo "  -> '$*' exit $rc in $(( SECONDS - start ))s"
  return "$rc"
}
ci_section "Lint";  timed bash -c 'sleep 1; echo "  0 problems"'; ci_end
ci_section "Test";  timed bash -c 'sleep 2; echo "  12 passed, 1 failed"; exit 1'; ci_end
echo "pipeline exit: $?"

$ bash ci.sh

##[group]Lint
  0 problems
  -> 'bash -c sleep 1; echo "  0 problems"' exit 0 in 1s
##[endgroup]

##[group]Test
  12 passed, 1 failed
  -> 'bash -c sleep 2; echo "  12 passed, 1 failed"; exit 1' exit 1 in 2s
##[endgroup]
pipeline exit: 0

timed correctly reported the test failure, but the script’s final status is 0, because the last command was ci_end’s echo. The CI system would mark this build green. Either run with set -e or record failures and exit non-zero at the end (Practice task 3).

Practice

1. Fix capture-bug.sh so that $(backup_files ...) gets only the archive path, while backup.log still receives the log line.

   Solution

   set -euo pipefail
   log() { echo "[backup] $*" | tee -a backup.log >&2; }
   backup_files() {
     local archive="uploads_backup.tar.gz"
     log "Backing up files: $1"
     tar -czf "$archive" "$1"
     echo "$archive"
   }
   mkdir -p uploads && echo data > uploads/a.txt
   archive=$(backup_files uploads)
   echo "archive=[$archive]"
   ls -l "$archive" | awk '{print $5, $9}'
   echo "backup.log: $(cat backup.log)"

   $ bash p1.sh
   [backup] Backing up files: uploads
   archive=[uploads_backup.tar.gz]
   158 uploads_backup.tar.gz
   backup.log: [backup] Backing up files: uploads

   The log line still reaches the terminal (on stderr) and the file (through tee), but not the variable.

2. Fix the rotation step so the archive really holds the log before the live file is truncated.

   Solution

   mkdir -p archive
   seq 1 5000 > app.log
   before=$(wc -l < app.log)
   gzip -9 -c app.log > archive/app.log.gz
   truncate -s 0 app.log
   echo "log had $before lines; archive holds $(zcat archive/app.log.gz | wc -l) lines; live log now $(wc -c < app.log) bytes"

   $ bash p2.sh
   log had 5000 lines; archive holds 5000 lines; live log now 0 bytes

   Lines the application writes between the gzip and the truncate are still lost. logrotate’s copytruncate has the same small window, and only signalling the app to reopen its log avoids it.

3. Rewrite the CI helpers so every stage still runs, but the pipeline exits 1 and names the failed stages if any of them failed.

   Solution

   ci_section() { printf '##[group]%s\n' "$*"; }
   ci_end()     { echo "##[endgroup]"; }
   failed=()
   stage() {
     local name="$1"; shift
     ci_section "$name"
     "$@" || failed+=("$name")
     ci_end
   }
   stage Lint bash -c 'echo "  0 problems"'
   stage Test bash -c 'echo "  12 passed, 1 failed"; exit 1'
   stage Build bash -c 'echo "  built"'
   if (( ${#failed[@]} )); then echo "FAILED stages: ${failed[*]}"; exit 1; fi
   echo "all stages passed"

   $ bash p3.sh; echo "pipeline exit: $?"
   ##[group]Lint
     0 problems
   ##[endgroup]
   ##[group]Test
     12 passed, 1 failed
   ##[endgroup]
   ##[group]Build
     built
   ##[endgroup]
   FAILED stages: Test
   pipeline exit: 1