Bash Input and Output

Every example below is a complete script, run with bash 5.2 on Ubuntu 24.04. Where a script reads input, the command line shows exactly what was piped in.

stdout vs stderr

Every process starts with three file descriptors: 0 is stdin, 1 is stdout and 2 is stderr. > redirects only stdout.

echo "data line"
echo "error line" >&2

$ bash streams.sh > out.txt; echo "--- out.txt:"; cat out.txt
error line
--- out.txt:
data line

error line still reached the terminal because only fd 1 went to the file. That split is the point: data goes to stdout, where the next command in a pipe can use it, and messages go to stderr, where a person sees them. Always send errors with >&2.

echo vs printf

printf applies its format string to the arguments and reuses it for any extras. echo’s handling of escape sequences depends on which shell and system you’re on.

printf "Name: %-8s Age: %3d\n" "Alice" 30 "Bob" 4
printf "Price: \$%.2f  Hex: 0x%X  Padded: %05d\n" 19.999 255 42
printf "%s\n" "one" "two"
echo "Tab:\there"
echo -e "Tab:\there"
printf '%b\n' "Tab:\there"

$ bash printf.sh
Name: Alice    Age:  30
Name: Bob      Age:   4
Price: $20.00  Hex: 0xFF  Padded: 00042
one
two
Tab:\there
Tab:	here
Tab:	here

Four arguments with a two-slot format produced two lines, and %s\n with two arguments printed each on its own line. That makes printf '%s\n' "${arr[@]}" the standard way to print an array one element per line. %.2f rounds, so 19.999 becomes 20.00. Bash’s echo leaves \t alone unless you add -e, but sh’s echo on some systems interprets it by default. printf '%b' behaves the same everywhere.

Keep data out of the format string. A variable used as the format is scanned for % directives like any other format:

msg='Disk 100% full'
printf "$msg\n"
printf '%s\n' "$msg"

$ bash fmt.sh
Disk 100 0.000000ull
Disk 100% full

In the first line, % f (a space flag, then f) became a floating-point conversion. There was no argument for it, so it printed 0.000000 and swallowed the f of “full”. Put a fixed format in single quotes and pass the data as an argument, as the second line does.

Reading Input

read takes one line from stdin. Each named variable gets one field, and the last one gets whatever is left.

read -r name
echo "Hello, $name"
read -r first last
echo "First: $first, Last: $last"
read -ra parts
echo "3 parts? ${#parts[@]} -> second is ${parts[1]}"
if read -r -t 1 extra; then echo "got: $extra"; else echo "no more input (status $?)"; fi

$ printf 'Alice\nAlice van Smith\none two three\n' | bash read.sh
Hello, Alice
First: Alice, Last: van Smith
3 parts? 3 -> second is two
no more input (status 1)

last took “van Smith”, the rest of the line. -a splits the line into an array, and -t sets a timeout in seconds. The final read hit end-of-input, and its failure status is how a while read loop knows to stop. When a person is typing, use -p "prompt: " to show a prompt and -s to hide what they type.

Always use -r:

read -r with_r
read without_r
echo "with -r:    $with_r"
echo "without -r: $without_r"

$ printf '%s\n' 'C:\new\temp' 'C:\new\temp' | bash read-backslash.sh
with -r:    C:\new\temp
without -r: C:newtemp

Without -r, read treats each backslash as an escape and removes it, which silently corrupts Windows paths, regexes, and anything else that contains one.

Redirection Order Matters

Redirections are applied left to right. 2>&1 means “point fd 2 wherever fd 1 points right now”.

ls /nonexistent > a.log 2>&1
ls /nonexistent 2>&1 > b.log
echo "a.log has $(wc -l < a.log) line(s); b.log has $(wc -l < b.log) line(s)"

$ bash order.sh
ls: cannot access '/nonexistent': No such file or directory
a.log has 1 line(s); b.log has 0 line(s)

In the first line, stdout moved to the file and then stderr followed it. In the second, stderr was pointed at the terminal (where stdout was at that moment), and only then was stdout moved. The error leaked to the screen and b.log stayed empty. &> file is Bash shorthand for the correct > file 2>&1. Use 2> /dev/null to discard errors and &> /dev/null to discard everything.

Pipes

Each stage reads the previous stage’s stdout. This example finds the addresses behind failed SSH logins:

printf '%s\n' \
  "Jan 10 sshd: Failed password for root from 10.0.0.5" \
  "Jan 10 sshd: Failed password for admin from 10.0.0.9" \
  "Jan 10 sshd: Accepted password for alice from 10.0.0.2" \
  "Jan 10 sshd: Failed password for root from 10.0.0.5" \
  "Jan 10 sshd: Failed password for test from 10.0.0.5" > auth.log

grep "Failed password" auth.log | awk '{print $NF}' | sort | uniq -c | sort -rn

$ bash pipeline.sh
      3 10.0.0.5
      1 10.0.0.9

uniq -c only counts adjacent duplicates, which is why sort runs before it. The second sort -rn orders the result by count. $NF is awk’s last field, so the command doesn’t depend on how many words come before the address.

tee: See It and Keep It

tee copies its input to a file and also passes it on down the pipe.

{
  echo "=== deploy started"
  echo "migrating"
  ls /missing-dir
  echo "=== deploy finished"
} 2>&1 | tee deploy.log | grep -c "==="
echo "deploy.log has $(wc -l < deploy.log) lines"

$ bash tee.sh
2
deploy.log has 4 lines

The { ...; } 2>&1 group sent everything into the pipe, including the ls error. tee saved all 4 lines and still passed them to grep. tee -a appends instead of overwriting, and tee f1 f2 writes to several files at once.

Process Substitution

<(cmd) exposes a command’s output as a file name, for tools that only accept files:

printf 'b\na\nc\n' > one.txt
printf 'c\nb\nd\n' > two.txt
diff <(sort one.txt) <(sort two.txt)
echo "diff exit: $?"
echo "procsub path: $(echo <(true))"

$ bash procsub.sh
1d0
< a
3a3
> d
diff exit: 1
procsub path: /dev/fd/63

diff compared two sorted streams without any temporary files. The last line shows how it works: <(...) expands to a /dev/fd/N path connected to the command’s output. The same mechanism powers done < <(cmd), which keeps a while loop out of a subshell (see the control-flow lesson).

Custom File Descriptors

exec N> file opens a file on descriptor N for the rest of the script. Writes don’t reopen the file, and reads continue from where the last one stopped.

exec 3> report.txt
echo "header" >&3
for n in 1 2; do echo "row $n" >&3; done
exec 3>&-
cat report.txt

exec 4< report.txt
read -r h <&4
read -r r1 <&4
exec 4<&-
echo "read via fd 4: [$h] then [$r1]"

$ bash fds.sh
header
row 1
row 2
read via fd 4: [header] then [row 1]

The two reads got consecutive lines because fd 4 remembers its position. Running read -r x < report.txt twice would return header both times. Close descriptors with N>&- when you’re done.

PIPESTATUS and pipefail

$? after a pipeline is the status of the last command only. PIPESTATUS holds every stage’s status, but only until the next command runs:

printf 'x\n' | grep nomatch | sort
echo "PIPESTATUS: ${PIPESTATUS[@]}"

printf 'x\n' | grep nomatch | sort
echo "first: ${PIPESTATUS[0]}"
echo "second: ${PIPESTATUS[1]}"

printf 'x\n' | grep nomatch | sort
codes=("${PIPESTATUS[@]}")
echo "saved: ${codes[0]} ${codes[1]} ${codes[2]}"

$ bash pipestatus.sh
PIPESTATUS: 0 1 0
first: 0
second: 
saved: 0 1 0

second is empty because the echo that printed first was itself a pipeline of one command, and it overwrote PIPESTATUS with just (0). Copy the array on the very next line if you need more than one value.

set -o pipefail makes the pipeline itself report failure:

(exit 2) | (exit 3) | true
echo "default: $?"
set -o pipefail
(exit 2) | (exit 3) | true
echo "pipefail: $?"

$ bash pipefail.sh
default: 0
pipefail: 3

Without pipefail, the final true hid both failures. With it, the pipeline returns the status of the rightmost failing command, 3, not the first one. Most scripts only need “did anything fail”, which pipefail answers. Use PIPESTATUS when you need to know which stage failed.

Practice

1. A function logs progress and also returns a value by echoing it. Make sure the log line stays out of the value the caller captures.

   Solution

   Send the log to stderr. $(...) captures stdout only:

   log() { echo "[log] $*" >&2; }
   get_version() {
     log "looking up version"
     echo "1.4.2"
   }
   v=$(get_version)
   echo "captured: [$v]"

   $ bash p1.sh
   [log] looking up version
   captured: [1.4.2]

   With the log on stdout, v would hold both lines.

2. Print the CSV below as aligned columns using read and printf, with no column command.

   NAME,ROLE,DAYS
   alice,admin,120
   bob,dev,7
   carol,analyst,45

   Solution

   while IFS=, read -r name role days; do
     printf '%-8s %-10s %4s\n' "$name" "$role" "$days"
   done <<'EOF'
   NAME,ROLE,DAYS
   alice,admin,120
   bob,dev,7
   carol,analyst,45
   EOF

   $ bash p2.sh
   NAME     ROLE       DAYS
   alice    admin       120
   bob      dev           7
   carol    analyst      45

3. Run a loop whose normal output is appended to results.log, with errors going to errors.log and nothing reaching the terminal.

   Solution

   check() { echo "ok: $1"; [[ $1 != bad ]] || { echo "failed: $1" >&2; return 1; }; }
   for item in one bad two; do
     check "$item"
   done 2>errors.log | tee -a results.log > /dev/null
   echo "results.log:"; cat results.log
   echo "errors.log:"; cat errors.log

   $ bash p3.sh
   results.log:
   ok: one
   ok: bad
   ok: two
   errors.log:
   failed: bad

   A redirection placed after done applies to the whole loop.