Regex in Bash

Every example below is a complete script, run with bash 5.2, GNU grep and GNU sed on Ubuntu 24.04.

=~ and the Quoting Rule

Inside [[ ]], =~ tests a string against a POSIX extended regular expression. Whether the right-hand side is quoted changes what it does:

re='^a.c$'
for s in abc 'a.c' '^a.c$'; do
  [[ $s =~ $re ]]     && r1=match || r1=no
  [[ $s =~ "$re" ]]   && r2=match || r2=no
  printf '%-6s  unquoted: %-5s  quoted: %s\n' "$s" "$r1" "$r2"
done

$ bash quoting.sh
abc     unquoted: match  quoted: no
a.c     unquoted: match  quoted: no
^a.c$   unquoted: no     quoted: match

Quoted, the pattern is a literal substring to look for. It only matched the string that contains the characters ^a.c$. Store the pattern in a variable and leave the variable unquoted. This also avoids having to escape spaces and parentheses inside [[ ]].

BASH_REMATCH: Capture Groups

After a successful match, BASH_REMATCH[0] holds the whole match and [1], [2], … hold the groups.

log_line="2025-01-15 14:32:07 ERROR database connection failed"
re='^([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9:]{8}) ([A-Z]+) (.+)$'
if [[ $log_line =~ $re ]]; then
  echo "date=${BASH_REMATCH[1]} time=${BASH_REMATCH[2]} level=${BASH_REMATCH[3]}"
  echo "message=${BASH_REMATCH[4]}"
fi

for version in v2.14.3-beta v1.0.0 2.0; do
  if [[ $version =~ ^v([0-9]+)\.([0-9]+)\.([0-9]+)(-(.+))?$ ]]; then
    echo "$version -> major=${BASH_REMATCH[1]} label=${BASH_REMATCH[5]:-stable} groups=${#BASH_REMATCH[@]}"
  else
    echo "$version -> no match, BASH_REMATCH now has ${#BASH_REMATCH[@]} elements"
  fi
done

$ bash capture.sh
date=2025-01-15 time=14:32:07 level=ERROR
message=database connection failed
v2.14.3-beta -> major=2 label=beta groups=6
v1.0.0 -> major=1 label=stable groups=6
2.0 -> no match, BASH_REMATCH now has 0 elements

Groups are numbered by their opening parenthesis, so the nested (-(.+)) makes [4] equal -beta and [5] equal beta. An optional group that doesn’t match still takes a slot, and it is empty. That’s why v1.0.0 also reports six elements. A failed match clears the array, so read captures right after the test.

ERE Syntax Reference

ERE has no \d, \w, lookarounds or lazy quantifiers. Use [0-9] and [[:alnum:]_] instead of \d and \w.

A Too-Loose Pattern: IPv4

A regex can match every string you want and still accept strings you don’t.

naive='^([0-9]{1,3}\.){3}[0-9]{1,3}$'
strict='^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$'
for ip in 192.168.1.100 192.168.1.999 10.0.0 256.1.1.1; do
  [[ $ip =~ $naive ]]  && a=yes || a=no
  [[ $ip =~ $strict ]] && b=yes || b=no
  printf '%-14s naive=%-3s strict=%s\n' "$ip" "$a" "$b"
done

$ bash ipv4.sh
192.168.1.100  naive=yes strict=yes
192.168.1.999  naive=yes strict=no
10.0.0         naive=no  strict=no
256.1.1.1      naive=yes strict=no

“One to three digits” says nothing about the value 0-255. Test your patterns against inputs that should fail, not just the ones that should pass.

The [^\]] Trap

This pattern for nginx access logs appears in many scripts:

line='10.0.0.7 - - [15/Jan/2025:14:32:07 +0000] "GET /api/users HTTP/1.1" 503 512'

broken='^([0-9.]+) - - \[([^\]]+)\] "([A-Z]+) ([^ ]+) HTTP/[0-9.]+" ([0-9]{3}) ([0-9]+)'
[[ $line =~ $broken ]] && echo "broken regex: matched" || echo "broken regex: no match"

fixed='^([0-9.]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) HTTP/[0-9.]+" ([0-9]{3}) ([0-9]+)'
if [[ $line =~ $fixed ]]; then
  echo "fixed regex: ip=${BASH_REMATCH[1]} path=${BASH_REMATCH[4]} status=${BASH_REMATCH[5]}"
fi

$ bash nginx.sh
broken regex: no match
fixed regex: ip=10.0.0.7 path=/api/users status=503

Inside POSIX brackets, a backslash is an ordinary character. [^\] means “anything except a backslash” and ends right there, so the ]+ that follows needs literal ] characters that aren’t in the log line. The pattern never matches, and nothing reports an error. To include ] in a class, put it first: [^]].

grep -P: When You Need PCRE

GNU grep’s -P adds Perl features that ERE lacks. BSD grep on macOS doesn’t have -P.

printf 'total $19.99 paid\nfoo then bar\nbar then foo\n<b>one</b><b>two</b>\n' > page.txt
echo "-- lookbehind:";   grep -oP '(?<=\$)[0-9]+\.[0-9]{2}' page.txt
echo "-- lookahead:";    grep -P 'foo(?=.*bar)' page.txt
echo "-- greedy -E:";    grep -oE '<b>.+</b>' page.txt
echo "-- lazy -P:";      grep -oP '<b>.+?</b>' page.txt
echo "-- \\d in -E:";    grep -cE '\d' <<< "abc 123"

$ bash grep-pcre.sh
-- lookbehind:
19.99
-- lookahead:
foo then bar
-- greedy -E:
<b>one</b><b>two</b>
-- lazy -P:
<b>one</b>
<b>two</b>
-- \d in -E:
0

The greedy .+ matched from the first <b> to the last </b>, while the lazy .+? stopped at each first </b>. The last line is a silent failure. Under -E, \d isn’t a digit class, so a line full of digits gave a count of 0.

sed with Capture Groups

sed -E uses the same ERE syntax. In the replacement, \1, \2 refer to the groups.

echo "John Smith" | sed -E 's/([A-Z][a-z]+) ([A-Z][a-z]+)/\2, \1/'
echo "error: line 42: undefined variable" | sed -E 's/.*line ([0-9]+).*/\1/'
echo "due 2025-03-01, paid 2025-03-09" | sed -E 's#([0-9]{4})-([0-9]{2})-([0-9]{2})#\3/\2/\1#g'
echo "due 2025-03-01" | sed 's/[0-9]{4}/YEAR/'
echo "due 2025-03-01" | sed -E 's/[0-9]{4}/YEAR/'

$ bash sed-regex.sh
Smith, John
42
due 01/03/2025, paid 09/03/2025
due 2025-03-01
due YEAR-03-01

When the replacement contains /, pick another delimiter such as # so you don’t have to escape every slash. The last two lines show the same trap as grep: without -E, {4} is literal text, nothing matches, and sed prints the line unchanged.

Reading Config Values

A regex can read key = value and key: value lines, but a greedy group picks up trailing whitespace:

printf 'db_host = db.local   \nport: 5432\n#db_host=old\n' > app.conf
get_value() {
  local re="^[[:space:]]*$2[[:space:]]*[=:][[:space:]]*(.+)[[:space:]]*$"
  local line
  while IFS= read -r line; do
    [[ $line =~ $re ]] && { printf '[%s]\n' "${BASH_REMATCH[1]}"; return 0; }
  done < "$1"
  return 1
}
get_value app.conf db_host
get_value app.conf port
get_value app.conf missing || echo "missing: not found"

$ bash config.sh
[db.local   ]
[5432]
missing: not found

(.+) is greedy, so it took the trailing spaces before [[:space:]]*$ got a chance. The trailing [[:space:]]* then matched nothing. A host name with spaces on the end fails in confusing ways later. Practice task 1 fixes it. The ^[[:space:]]* anchor is also what kept the commented #db_host=old from matching.

Practice

1. Fix get_value so the value never includes trailing whitespace.

   Solution

   End the capture with a character that isn’t whitespace:

   printf 'db_host = db.local   \nport: 5432\n' > app.conf
   get_value() {
     local re="^[[:space:]]*$2[[:space:]]*[=:][[:space:]]*(.*[^[:space:]])[[:space:]]*$"
     local line
     while IFS= read -r line; do
       [[ $line =~ $re ]] && { printf '[%s]\n' "${BASH_REMATCH[1]}"; return 0; }
     done < "$1"
     return 1
   }
   get_value app.conf db_host
   get_value app.conf port

   $ bash p1.sh
   [db.local]
   [5432]

2. =~ finds only the first match. Loop over "retries=3 timeout=30 workers=12" and print every key and value pair.

   Solution

   Remove the matched text after each match and test again:

   s="retries=3 timeout=30 workers=12"
   while [[ $s =~ ([a-z]+)=([0-9]+) ]]; do
     echo "${BASH_REMATCH[1]} -> ${BASH_REMATCH[2]}"
     s=${s#*"${BASH_REMATCH[0]}"}
   done

   $ bash p2.sh
   retries -> 3
   timeout -> 30
   workers -> 12

3. Mask every 16-digit card number in a line with sed, keeping only the last four digits.

   Solution

   echo "card 4111222233334444 and 5500111122223333 charged" |
     sed -E 's/[0-9]{12}([0-9]{4})/************\1/g'

   $ bash p3.sh
   card ************4444 and ************3333 charged