Bash File Operations

Every example below is a complete script, run with bash 5.2 and GNU coreutils on Ubuntu 24.04 in an empty working directory, as a normal (non-root) user.

Guarding with File Tests

Check a file before you use it. Your own error message is far clearer than whatever the failing tool would print later.

require_file() {
  local file="$1"
  [[ -f $file ]] || { echo "ERROR: required file not found: $file" >&2; exit 1; }
  [[ -r $file ]] || { echo "ERROR: file not readable: $file" >&2; exit 1; }
  echo "ok: $file"
}
echo "port=5432" > app.conf
touch locked.conf && chmod 000 locked.conf

require_file app.conf
require_file locked.conf
echo "never printed"

$ bash require.sh
ok: app.conf
ERROR: file not readable: locked.conf

locked.conf exists, so -f passed, but mode 000 failed the -r test. Run the same script as root and it passes, because root can read any file. -e (exists), -d (directory), -L (symlink), -s (non-empty), -w and -x complete the set, as the operators lesson showed. -nt and -ot compare modification times, as in [[ a -nt b ]].

Reading Files and the Missing Last Line

while IFS= read -r is the standard line loop. It has one well-known trap:

printf 'first\nsecond\nthird' > data.txt   # no newline after "third"

echo "plain while-read:"
while IFS= read -r line; do echo "  $line"; done < data.txt

echo "fixed:"
while IFS= read -r line || [[ -n $line ]]; do echo "  $line"; done < data.txt

content=$(<data.txt)
echo "\$(<file) holds ${#content} chars"

$ bash lastline.sh
plain while-read:
  first
  second
fixed:
  first
  second
  third
$(<file) holds 18 chars

third vanished from the first loop. read did fill $line with it, but hitting end-of-file without a newline made it return failure, so the loop stopped before the body ran. Files written by editors and APIs often lack that final newline. The || [[ -n $line ]] form handles them. $(<file) reads the whole file into a variable without running cat.

Copying: What -p Preserves

echo "v1" > src.txt
touch -d "2024-01-15 09:30" src.txt

cp src.txt plain.txt
cp -p src.txt preserved.txt
for f in src.txt plain.txt preserved.txt; do
  printf '%-14s %s\n' "$f" "$(date -r "$f" '+%F %H:%M')"
done

$ bash copy.sh
src.txt        2024-01-15 09:30
plain.txt      2026-09-23 13:17
preserved.txt  2024-01-15 09:30

A plain cp stamps the copy with the current time. cp -p keeps the original modification time and mode, and the owner too when run as root. This matters for anything that compares timestamps, such as make, rsync -u, backups and the -nt test. Other flags you’ll use: cp -r for directories, cp -u to copy only when the source is newer, and cp -v to list each file.

Bulk Renames and Unmatched Globs

touch IMG_01.JPG IMG_02.JPG notes.txt
for f in *.JPG; do
  mv -- "$f" "${f%.JPG}.jpg"
done
ls
for f in *.PNG; do mv -- "$f" "${f%.PNG}.png"; done

$ bash rename.sh
IMG_01.jpg
IMG_02.jpg
notes.txt
rename.sh
mv: cannot stat '*.PNG': No such file or directory

The first loop worked. The second found no .PNG files, so it ran once with the literal string *.PNG, and mv failed with status 1. Add shopt -s nullglob or [[ -e $f ]] || continue whenever the glob might match nothing. The -- stops mv from reading a file named -rf.JPG as options.

Deleting Safely with ${var:?}

rm -rf "$dir/" with an empty $dir becomes rm -rf /. The :? expansion prevents that:

set -u
mkdir -p build/out && touch build/out/app.bin
dir="build"
rm -rf -- "${dir:?}/"
[[ -d build ]] || echo "build removed"

dir=""
rm -rf -- "${dir:?refusing to rm with empty dir}/"
echo "never printed"

$ bash guard.sh
build removed
guard.sh: line 8: dir: refusing to rm with empty dir

When dir is empty or unset, ${dir:?} stops the script before rm runs, so no separate [[ -n $dir ]] test is needed. Use it on every variable that ends up in an rm -rf path. For data you might want back, mv the file to a trash directory instead of deleting it.

find: Search by Name, Size, Age

This example ran against a small tree: logs/app.log, a 2 KB logs/big.log, a 10-day-old logs/old/archive.log, and a my docs/ directory whose file names contain spaces.

echo "-- *.log files:"
find . -name "*.log" | sort
echo "-- bigger than 1k:"
find . -type f -size +1k
echo "-- modified more than 7 days ago:"
find . -type f -mtime +7
echo "-- files containing TODO (spaces in names):"
find . -name "*.txt" -print0 | xargs -0 grep -l "TODO"
echo "-- broken without -print0:"
find . -name "*.txt" | xargs grep -l "TODO"

$ bash find.sh
-- *.log files:
./logs/app.log
./logs/big.log
./logs/old/archive.log
-- bigger than 1k:
./logs/big.log
-- modified more than 7 days ago:
./logs/old/archive.log
-- files containing TODO (spaces in names):
./my docs/plan notes.txt
-- broken without -print0:
grep: ./my: No such file or directory
grep: docs/finished.txt: No such file or directory
grep: ./my: No such file or directory
grep: docs/plan: No such file or directory
grep: notes.txt: No such file or directory

Plain xargs splits its input on whitespace, so my docs/plan notes.txt reached grep as three wrong file names, and the TODO file was never searched. -print0 with xargs -0 separates names with NUL bytes, which can’t appear in a file name. Also useful: -exec cmd {} + runs a command on batches of results, and -delete removes them (Practice task 3).

stat: Size, Mode, Age

stat -c prints chosen fields. %s is the size, %a the octal mode, and %Y the modification time in epoch seconds.

echo "hello" > config.yml
touch -d "2 hours ago" config.yml
stat -c 'name=%n size=%s perms=%a' config.yml
age=$(( $(date +%s) - $(stat -c %Y config.yml) ))
echo "age: $(( age / 60 )) minutes"
(( age > 3600 )) && echo "config is stale"

$ bash stat.sh
name=config.yml size=6 perms=644
age: 120 minutes
config is stale

stat -c is GNU (Linux) syntax. On macOS, BSD stat uses -f with different format letters. That is a common reason a script works on a server but fails on a laptop.

Symlinks: The ln -sf Trap

Deploy scripts often point a current symlink at the latest release. Updating that link has a trap:

mkdir -p releases/v1 releases/v2
ln -s releases/v1 current
echo "current -> $(readlink current)"

ln -sf releases/v2 current
echo "after ln -sf:  current -> $(readlink current)"
ls releases/v1

ln -sfn releases/v2 current
echo "after ln -sfn: current -> $(readlink current)"

$ bash symlink.sh
current -> releases/v1
after ln -sf:  current -> releases/v1
v2
after ln -sfn: current -> releases/v2

ln -sf didn’t move the link. current points to a directory, so ln followed it and created a new link called v2 inside releases/v1. The deploy looked successful but was still serving v1. -n treats the existing link as a plain file, so it really gets replaced.

Atomic Writes, and the Permissions They Change

Writing to a temp file and then renaming it with mv means readers always see either the complete old file or the complete new one. The rename is atomic within one filesystem.

umask 022
write_config() {
  local target="$1" tmp
  tmp=$(mktemp "${target}.XXXXXX")
  printf 'host=localhost\nport=5432\n' > "$tmp"
  grep -q '^host=' "$tmp" || { rm -f "$tmp"; return 1; }
  mv "$tmp" "$target"
}
echo "old" > app.conf
stat -c 'before: %a %n' app.conf
write_config app.conf
stat -c 'after:  %a %n' app.conf
cat app.conf

$ bash atomic.sh
before: 644 app.conf
after:  600 app.conf
host=localhost
port=5432

The content is right, but the mode went from 644 to 600. mktemp always creates files readable only by their owner, and the rename kept that mode. A service that reads the config as another user would now get “permission denied”. Practice task 2 fixes it.

Practice

1. Gzip every .log file under logs/ that is more than 7 days old, using a single find command. The test tree has today.log plus two files that are 9 and 30 days old.

   Solution

   find logs -name "*.log" -mtime +7 -exec gzip {} +
   ls logs

   $ bash p1.sh
   old1.log.gz
   old2.log.gz
   today.log

   {} + passes many files to a single gzip run instead of starting one process per file.

2. Fix write_config so the replaced file keeps its original permissions.

   Solution

   Copy the mode onto the temp file before the rename:

   umask 022
   write_config() {
     local target="$1" tmp
     tmp=$(mktemp "${target}.XXXXXX")
     [[ -e $target ]] && chmod --reference="$target" "$tmp"
     printf 'host=localhost\nport=5432\n' > "$tmp"
     mv "$tmp" "$target"
   }
   echo "old" > app.conf
   write_config app.conf
   stat -c 'after: %a %n' app.conf

   $ bash p2.sh
   after: 644 app.conf

3. Delete every *.tmp file in cache/ that is more than a day old, and print each name as it is deleted. a.tmp is new, b.tmp and c.tmp are 3 days old, and keep.dat must survive.

   Solution

   find cache -name "*.tmp" -mtime +1 -print -delete | sort
   echo "left:"; ls cache

   $ bash p3.sh
   cache/b.tmp
   cache/c.tmp
   left:
   a.tmp
   keep.dat

   Put -delete last. find evaluates its tests left to right, and a -delete placed before -name would delete everything it visits.

4. Rename every file in inbox/ whose name contains spaces, replacing each space with _. It must work for any file name.

   Solution

   find inbox -type f -name "* *" -print0 |
     while IFS= read -r -d '' f; do
       dir=${f%/*} base=${f##*/}
       mv -- "$f" "$dir/${base// /_}"
     done
   ls inbox

   $ bash p4.sh
   plain.txt
   q1_report.pdf
   team_photo_2.png

   read -d '' reads up to each NUL byte, matching -print0. Only the base name is changed, so a space in a directory name can’t produce a path that doesn’t exist.