Bash Process Management

Every example below is a complete script, run with bash 5.2 and procps on Ubuntu 24.04. PIDs are small because each run happened in a fresh container.

Background Jobs and wait

& starts a command in the background, and wait blocks until background jobs finish.

start=$SECONDS
task() { sleep "$1"; echo "task $2 done after ${1}s"; }

task 2 A &
task 1 B &
task 3 C &
echo "launched 3 jobs; last PID is a number: $([[ $! =~ ^[0-9]+$ ]] && echo yes)"
wait
echo "all done in $(( SECONDS - start ))s (sequential would be 6s)"

$ bash background.sh
launched 3 jobs; last PID is a number: yes
task B done after 1s
task A done after 2s
task C done after 3s
all done in 3s (sequential would be 6s)

The jobs finished in order of duration, not launch order, and the total time was that of the longest job. $! holds the PID of the most recent background job. Save it right away if you need to wait for or kill that job later.

jobs and Job Specs

jobs lists the shell’s background jobs. %N refers to job number N in kill and wait.

sleep 30 &
sleep 40 &
jobs
kill %1
wait %1 2>/dev/null
echo "job 1 exit status: $?"
jobs
kill %2

$ bash jobs.sh
[1]-  Running                 sleep 30 &
[2]+  Running                 sleep 40 &
job 1 exit status: 143
[2]+  Running                 sleep 40 &

+ marks the current job and - the previous one. The killed job exited with 143, which is 128 + 15 (SIGTERM). fg, bg and Ctrl+Z only work in an interactive shell. Scripts run without job control, so they manage jobs through $!, wait and kill.

Collecting Exit Codes from Parallel Work

wait PID returns that job’s exit status, so you can run checks in parallel and still tell which one failed:

declare -A pids
check() { sleep 1; [[ $1 != db01 ]]; }

for host in web01 web02 db01; do
  check "$host" &
  pids[$host]=$!
done

failures=0
for host in web01 web02 db01; do
  if wait "${pids[$host]}"; then
    echo "$host: UP"
  else
    echo "$host: DOWN (exit $?)"
    (( ++failures ))
  fi
done
echo "failures: $failures"

$ bash exitcodes.sh
web01: UP
web02: UP
db01: DOWN (exit 1)
failures: 1

All three checks ran at the same time, so this took about 1 second, not 3. In a real script, check would be ping -c1 -W2 "$host" or curl -sf. Use (( ++failures )), not failures++: the post-increment evaluates to 0 the first time, and under set -e that would end the script.

Signals with kill

kill sends any signal, not just the ones that terminate. kill -0 only checks that the process exists.

sleep 60 &
pid=$!
kill -0 "$pid" && echo "$pid is alive"
kill -STOP "$pid"; ps -o stat= -p "$pid" | sed 's/^/state after STOP: /'
kill -CONT "$pid"; ps -o stat= -p "$pid" | sed 's/^/state after CONT: /'
kill -TERM "$pid"
wait "$pid"
echo "exit after TERM: $?"
kill -0 "$pid" 2>/dev/null || echo "$pid is gone"

$ bash signals.sh
9 is alive
state after STOP: T
state after CONT: S
exit after TERM: 143
9 is gone

T means stopped and S means sleeping. STOP pauses a process without ending it.

pkill -f "pattern" and killall name select processes by name instead of PID.

Why Your Trap Fires Late

A script that traps TERM should react as soon as the signal arrives. This one doesn’t:

trap 'echo "[$SECONDS s] got TERM, cleaning up"; exit 143' TERM
echo "[$SECONDS s] working (pid $$)"
sleep 5
echo "[$SECONDS s] finished normally"

$ bash trap-delay.sh & sleep 1; kill -TERM $!; wait $!; echo "exit: $?"
[0 s] working (pid 10)
[5 s] got TERM, cleaning up
exit: 143

The TERM arrived at 1 second, but the handler ran at 5 seconds. Bash only runs traps between commands, so it waited for the foreground sleep to finish. When systemd or Kubernetes sends TERM and then SIGKILL after a grace period, that delay can cost you the cleanup entirely. Run the long step in the background and wait for it, because wait is interrupted by signals:

trap 'echo "[$SECONDS s] got TERM, cleaning up"; exit 143' TERM
echo "[$SECONDS s] working (pid $$)"
sleep 5 &
wait $!
echo "[$SECONDS s] finished normally"

$ bash trap-fast.sh & sleep 1; kill -TERM $!; wait $!; echo "exit: $?"
[0 s] working (pid 10)
[1 s] got TERM, cleaning up
exit: 143

The handler now runs at 1 second. By convention, a script that stops because of a signal exits with 128 + the signal number: 143 for TERM, 130 for INT.

Subshells vs Grouping

( ... ) runs its commands in a subshell. { ...; } groups them in the current shell.

x=10
cd /tmp
( x=99; cd /; echo "inside:  x=$x pwd=$PWD" )
echo "outside: x=$x pwd=$PWD"

{ x=42; cd /; }
echo "after { }: x=$x pwd=$PWD"

$ bash subshell.sh
inside:  x=99 pwd=/
outside: x=10 pwd=/tmp
after { }: x=42 pwd=/

Nothing done inside the parentheses survived: not the variable, not the cd. That makes a subshell a safe place to change directory, options (set +e) or IFS for a few lines. Braces change the real shell state. $(...), pipeline stages and & jobs are all subshells too.

Limiting Parallelism

Starting a job for every input can overwhelm a machine or an API. Cap the number of jobs running at once with wait -n, which returns as soon as any one job finishes:

start=$SECONDS
max_jobs=2
work() { sleep 1; echo "[$(( SECONDS - start ))s] finished $1"; }

for item in a b c d e; do
  while (( $(jobs -rp | wc -l) >= max_jobs )); do
    wait -n
  done
  work "$item" &
done
wait
echo "total: $(( SECONDS - start ))s for 5 one-second jobs, 2 at a time"

$ bash throttle.sh
[1s] finished a
[1s] finished b
[2s] finished c
[2s] finished d
[3s] finished e
total: 3s for 5 one-second jobs, 2 at a time

The jobs finished in pairs: five one-second jobs, two at a time, take three rounds. Counting running jobs with jobs -rp is sturdier than keeping your own counter. xargs -P does the same thing without a loop:

start=$SECONDS
printf '%s\n' a b c d e | xargs -P 2 -I{} sh -c 'sleep 1; echo "done $1"' _ {} | sort
echo "total: $(( SECONDS - start ))s"

$ bash xargs.sh
done a
done b
done c
done d
done e
total: 3s

The sort is there because parallel jobs print in whatever order they finish. Each item reaches the inner shell as an argument: _ fills $0, the item becomes $1, and the code inside the quotes never changes. It’s tempting to write sh -c 'echo "done {}"' instead, but that pastes the input into the code, so the inner shell parses each input line as shell syntax. A hostile (but harmless) filename shows the difference:

name='$(echo PWNED >&2)report.txt'   # a hostile but harmless filename

printf '%s\n' "$name" | xargs -I{} sh -c 'echo "unsafe: {}"'
printf '%s\n' "$name" | xargs -I{} sh -c 'echo "safe:   $1"' _ {}

$ bash inject.sh
PWNED
unsafe: report.txt
safe:   $(echo PWNED >&2)report.txt

In the unsafe form, the inner shell ran the $(…) from the filename. Here it only printed PWNED, but it could as easily have been rm -rf ~. The safe form prints the name unchanged, because $1 is expanded as data and never parsed as code. The same rule applies to find … -exec sh -c '…' _ {} \;: put {} after the script, never inside it. xargs -I still treats quotes and backslashes in its input specially, so a filename containing ' fails with “unmatched single quote”; for arbitrary names use xargs -d '\n' (GNU) or printf '%s\0' … | xargs -0.

Practice

1. Write run_with_timeout SECONDS command... that kills the command if it runs too long and returns its exit status either way.

   Solution

   run_with_timeout() {
     local secs="$1"; shift
     "$@" &
     local pid=$!
     ( sleep "$secs"; kill "$pid" 2>/dev/null ) &
     local watchdog=$!
     wait "$pid"
     local rc=$?
     kill "$watchdog" 2>/dev/null
     return "$rc"
   }
   run_with_timeout 1 sleep 10; echo "slow command: exit $?"
   run_with_timeout 5 true;     echo "fast command: exit $?"

   $ bash p1.sh
   slow command: exit 143
   fast command: exit 0

   For external commands, coreutils’ timeout 1 sleep 10 does this for you and exits 124. A function like this one also works for shell functions, which timeout can’t run.

2. When trap-fast.sh gets TERM, its background sleep keeps running as an orphan. Make the trap stop the child too.

   Solution

   child=""
   trap 'echo "got TERM, stopping child $child"; kill "$child"; wait "$child"; exit 143' TERM
   sleep 30 &
   child=$!
   wait "$child"

   $ bash p2.sh & sleep 1; kill -TERM $!; wait $!; echo "exit: $?"; pgrep -x sleep || echo "no sleep left running"
   got TERM, stopping child 12
   exit: 143
   no sleep left running

3. Use xargs -P to run the five one-second jobs three at a time. Predict the total time first.

   Solution

   start=$SECONDS
   printf '%s\n' a b c d e | xargs -P 3 -I{} sh -c 'sleep 1; echo "done $1"' _ {} > /dev/null
   echo "5 jobs, 3 at a time: $(( SECONDS - start ))s"

   $ bash p3.sh
   5 jobs, 3 at a time: 2s

   The first round runs three jobs and the second runs the remaining two.