Bash Text Processing

Every example below runs against the same three small files, created fresh for each run. The tools are GNU grep, sed, awk (gawk) and coreutils on Ubuntu 24.04. Here are two of the files; the third, config.ini, is a short key=value file with comments and blank lines.

$ cat app.log sales.csv
2025-03-01 10:00:01 INFO  api GET /users 200 12ms 10.0.0.5
2025-03-01 10:00:02 ERROR db connection refused 10.0.0.9
2025-03-01 10:00:03 WARN  api GET /orders 429 3ms 10.0.0.5
2025-03-01 10:00:04 INFO  api POST /orders 201 88ms 192.168.1.20
2025-03-01 10:00:05 error api GET /users 500 240ms 10.0.0.5
2025-03-01 10:00:06 INFO  api GET /health 200 1ms 192.168.1.20
id,region,rep,amount
1,north,ana,120.50
2,south,ben,80.00
3,north,cara,99.50
4,east,ana,300.00
5,south,dev,20.00

grep: Filter Lines

echo "-- -i error:"
grep -i "error" app.log | cut -c21-50
echo "-- -c, -v:"
grep -c "INFO" app.log
grep -vc "INFO" app.log
echo "-- -E alternation, -n:"
grep -nE "ERROR|WARN" app.log | cut -c1-40
echo "-- -o without -E:"
grep -o "192\.168\.[0-9]+\.[0-9]+" app.log
echo "(exit $?)"
echo "-- -oE:"
grep -oE "192\.168\.[0-9]+\.[0-9]+" app.log
echo "-- -F treats . literally:"
grep -c "10.0.0.5" app.log
grep -c "10.0.0.5" <<< "10x0y0z5"
grep -cF "10.0.0.5" <<< "10x0y0z5"

$ bash grep.sh
-- -i error:
ERROR db connection refused 10
error api GET /users 500 240ms
-- -c, -v:
3
3
-- -E alternation, -n:
2:2025-03-01 10:00:02 ERROR db connectio
3:2025-03-01 10:00:03 WARN  api GET /ord
-- -o without -E:
(exit 1)
-- -oE:
192.168.1.20
192.168.1.20
-- -F treats . literally:
3
1
0

Two results here catch people out. First, grep -o with [0-9]+ found nothing. Plain grep uses basic regular expressions, where + is a literal character, so you need -E for +, ?, | and {n}. Second, the pattern 10.0.0.5 matched 10x0y0z5, because an unescaped . matches any character. -F searches for a fixed string. Other flags you’ll use: -r to search a directory tree, -l to list matching files, -A/-B/-C N for context lines, and -q to only set the exit status.

sed: Edit Streams

sed -e '/^#/d' -e '/^$/d' config.ini
echo "-- s/// vs s///g on one line:"
echo "localhost localhost" | sed 's/localhost/db/'
echo "localhost localhost" | sed 's/localhost/db/g'
echo "-- anchored, in place with backup:"
sed -i.bak 's/^host=localhost$/host=db.internal/' config.ini
grep host config.ini
ls config.ini*
echo "-- lines 2-3:"
sed -n '2,3p' config.ini

$ bash sed.sh
host=localhost
port=5432
cache_host=localhost
-- s/// vs s///g on one line:
db localhost
db db
-- anchored, in place with backup:
host=db.internal
cache_host=localhost
config.ini
config.ini.bak
-- lines 2-3:
host=db.internal
port=5432

Without g, s/// replaces only the first match on each line. The in-place edit uses ^...$ anchors, so cache_host=localhost was left alone. An unanchored s/localhost/db.internal/ would have changed the cache host too. -i.bak keeps the original, and the same syntax works in GNU and BSD sed. -n with p prints only the lines you select.

awk: Fields and Aggregation

awk splits each line into fields ($1, $2, …). Each rule is a condition followed by an action.

echo "-- level, path, status where status >= 400:"
awk '$7 >= 400 {print $3, $6, $7}' app.log
echo "-- per-level counts:"
awk '{count[toupper($3)]++} END {for (l in count) print l, count[l]}' app.log | sort
echo "-- average latency:"
awk '{ sub(/ms$/, "", $8); if ($8 != "") { sum += $8; n++ } } END { printf "%.1f ms over %d requests\n", sum/n, n }' app.log

$ bash awk.sh
-- level, path, status where status >= 400:
WARN /orders 429
error /users 500
-- per-level counts:
ERROR 2
INFO 3
WARN 1
-- average latency:
68.8 ms over 5 requests

toupper($3) merged error and ERROR into one bucket. The latency rule strips the ms suffix before doing arithmetic, and it skips the database error line, which has no eighth field. That line’s seventh field is an IP address, so $7 >= 400 compared it as a string, and "10.0.0.9" sorts below "400", which kept it out of the output. That worked here by luck. On data with mixed columns, check $7 ~ /^[0-9]+$/ before comparing. -F, sets the field separator for CSV, and NR is the current line number.

cut, sort, uniq

echo "-- cut region column:"
cut -d, -f2 sales.csv | tail -n +2 | sort | uniq -c
echo "-- sort by amount (text vs numeric):"
tail -n +2 sales.csv | sort -t, -k4 | cut -d, -f4 | paste -sd' '
tail -n +2 sales.csv | sort -t, -k4 -n | cut -d, -f4 | paste -sd' '
echo "-- uniq without sort:"
printf 'a\nb\na\na\n' | uniq -c

$ bash cut-sort.sh
-- cut region column:
      1 east
      2 north
      2 south
-- sort by amount (text vs numeric):
120.50 20.00 300.00 80.00 99.50
20.00 80.00 99.50 120.50 300.00
-- uniq without sort:
      1 a
      1 b
      2 a

tail -n +2 skips the header row. Without -n, sort compares text, so 120.50 comes before 20.00. uniq only merges adjacent duplicates, which is why a appears twice in the last count. Sort first. sort -h handles du -h sizes like 10K and 2G, and sort -u removes duplicates as it sorts.

tr, wc, column

echo "hello world" | tr 'a-z' 'A-Z'
echo "hello 123 world" | tr -d '0-9'
echo "hello   world" | tr -s ' '
echo "a:b:c" | tr ':' '\n'
printf 'naïve\n' | wc -c
printf 'naïve\n' | wc -m

$ bash tr-wc.sh
HELLO WORLD
hello  world
hello world
a
b
c
7
6

tr works on single characters, not strings. -d deletes the digits but leaves both spaces around them. wc -c counts bytes, and ï takes two bytes in UTF-8, so the byte count is one higher than the character count from wc -m.

column -t lines text up into a table:

cut -d, -f2,3,4 sales.csv | column -t -s,

$ bash column.sh
region  rep   amount
north   ana   120.50
south   ben   80.00
north   cara  99.50
east    ana   300.00
south   dev   20.00

A Report That Sorts Its Own Header

Here is a pattern that shows up in many tutorials: aggregate with awk, then sort by total.

awk -F, '
  NR == 1 { next }
  { totals[$2] += $4; counts[$2]++ }
  END {
    printf "%-8s %8s %6s\n", "Region", "Total", "Count"
    for (r in totals) printf "%-8s %8.2f %6d\n", r, totals[r], counts[r]
  }
' sales.csv | sort -k2 -rn

$ bash report.sh
east       300.00      1
north      220.00      2
south      100.00      2
Region      Total  Count

The totals are right, but the header ended up at the bottom. sort received it as just another line, and under -n the word Total counts as 0, so it sorted last. Print the header outside the sorted stream (Practice task 1). The same fix applies to any command that emits its own header line.

Practice

1. Fix report.sh so the header stays on top.

   Solution

   printf "%-8s %8s %6s\n" "Region" "Total" "Count"
   awk -F, '
     NR == 1 { next }
     { totals[$2] += $4; counts[$2]++ }
     END { for (r in totals) printf "%-8s %8.2f %6d\n", r, totals[r], counts[r] }
   ' sales.csv | sort -k2 -rn

   $ bash p1.sh
   Region      Total  Count
   east       300.00      1
   north      220.00      2
   south      100.00      2

2. List the two IP addresses that appear most often in app.log, with their counts.

   Solution

   awk '{print $NF}' app.log | sort | uniq -c | sort -rn | head -2

   $ bash p2.sh
         3 10.0.0.5
         2 192.168.1.20

   $NF is the last field. It is safer than a fixed column number because the ERROR line has fewer fields than the others.

3. Comment out the port= line of config.ini in place, keeping its value.

   Solution

   sed -i 's/^port=/#&/' config.ini
   grep -n port config.ini

   $ bash p3.sh
   3:#port=5432

   & in the replacement stands for the whole matched text.

4. From sales.csv, print every sales rep whose total is over 100, with the total to two decimal places.

   Solution

   awk -F, 'NR > 1 { t[$3] += $4 } END { for (r in t) if (t[r] > 100) printf "%s %.2f\n", r, t[r] }' sales.csv | sort

   $ bash p4.sh
   ana 420.50