Skip to main content
Defensive Security · lesson 4 of 8

Beginner

1

Intermediate

6

Advanced

1

Also part of Defensive Security

6
Open the Defensive Security page →
Defensive Security intermediate Lesson 4 of 8

Alert Triage and Tuning: From 41 Alerts to the 3 That Matter

One noisy Suricata rule, 41 alerts and 38 of them benign. Triage by grouping, then tune with suppress and threshold, and measure what each change hides.

By SkillByExample · Published

defensive-security suricata alert-triage false-positives tuning soc

A rule that catches something real usually catches plenty that isn’t. Here you’ll produce a day of traffic to your own web server: a monitoring check, a deploy job, and three requests from a probe. One policy rule turns that into 41 alerts. You’ll triage them by grouping, then tune the rule three ways and count exactly what each tuning removes. Tested with Docker Engine 29.8.2, Suricata 8.0.7 (jasonish/suricata:8.0), Alpine 3.22.6 with tcpdump 4.99.5 and curl 8.14.1, and Python 3.12.15 (python:3.12-slim).

A day of traffic

The web server now serves two real files, so the automation gets 200s.

mkdir site && echo ok > site/health && echo 1.4.2 > site/version
docker run -d --name web --network ds-lab --ip 172.30.0.11 -v "$PWD/site":/site -w /site python:3.12-slim python -m http.server 8000
docker run -d --name tap --network container:web -v "$PWD":/work capture-lab tcpdump -i eth0 -U -w /work/day.pcap port 8000

The ds-lab network and capture-lab image come from the Suricata lesson. --network container:web puts the capture in the web server’s own network namespace, so it sees every client, the way a sensor in front of a server would.

Each client runs the same script with a different role.

# traffic.sh: one client's requests; the role is the first argument
url=http://172.30.0.11:8000
case "$1" in
  monitor) for i in $(seq 30); do curl -s -o /dev/null $url/health; done ;;
  deploy)  for i in $(seq 8); do curl -s -o /dev/null $url/version; done ;;
  probe)   for p in .env admin backup.zip; do curl -s -o /dev/null $url/$p; done ;;
esac
echo "$1 done"
$ docker run --rm --network ds-lab --ip 172.30.0.30 -v "$PWD":/work -w /work capture-lab sh traffic.sh monitor
monitor done
$ docker run --rm --network ds-lab --ip 172.30.0.31 -v "$PWD":/work -w /work capture-lab sh traffic.sh deploy
deploy done
$ docker run --rm --network ds-lab --ip 172.30.0.20 -v "$PWD":/work -w /work capture-lab sh traffic.sh probe
probe done
$ docker stop tap && docker logs tap 2>&1 | tail -3
tap
492 packets captured
492 packets received by filter
0 packets dropped by kernel

172.30.0.30 is the monitoring server, 172.30.0.31 the deploy job, and 172.30.0.20 something looking for files that shouldn’t be there. All three use curl.

One rule, 41 alerts

The rule flags command-line HTTP clients. Rules like it exist in real rule sets, because malware and attackers’ tools often use curl or similar clients. It’s also exactly the kind of rule that automation trips all day.

# local.rules
alert http any any -> any any (msg:"LAB command-line HTTP client (curl)"; flow:established,to_server; http.user_agent; content:"curl/"; startswith; classtype:policy-violation; sid:1000002; rev:1;)

Reading 41 lines of fast.log one by one is how alert fatigue starts. Group them instead: by source, path and status.

# triage.py: group alerts by source and path, so the noise stands out
import json, sys
from collections import Counter

out = sys.argv[1] if len(sys.argv) > 1 else "out"
alerts = [e for line in open(f"{out}/eve.json")
          if (e := json.loads(line))["event_type"] == "alert"]

groups = Counter((a["src_ip"], a["http"]["url"], a["http"].get("status")) for a in alerts)
print(f"{len(alerts)} alerts in {len(groups)} groups")
for (ip, url, status), n in groups.most_common():
    print(f"{n:>4}  {ip:<13} {url:<12} {status}")
$ rm -rf out && mkdir out
$ docker run --rm --entrypoint suricata -v "$PWD":/work -w /work jasonish/suricata:8.0 -r day.pcap -S local.rules -k none -l out
i: suricata: This is Suricata version 8.0.7 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 492 packets, 45964 bytes
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python triage.py
41 alerts in 5 groups
  30  172.30.0.30   /health      200
   8  172.30.0.31   /version     200
   1  172.30.0.20   /admin       404
   1  172.30.0.20   /backup.zip  404
   1  172.30.0.20   /.env        404

41 alerts are really five decisions. Two groups are the same request repeated, with a 200, from hosts you can identify: that’s expected automation. The last three come from one source, each asks for a different sensitive path, and each got a 404. That pattern is reconnaissance. 38 of the 41 alerts are benign, a 93% false-positive rate for this rule on this traffic, and the three that matter were at the bottom of the list.

Tuning 1: suppress the monitoring host

Suricata reads tuning from a threshold.config file. suppress stops one rule from alerting for one source.

# threshold.config: the monitoring host polls /health with curl all day
suppress gen_id 1, sig_id 1000002, track by_src, ip 172.30.0.30
$ rm -rf out && mkdir out
$ docker run --rm --entrypoint suricata -v "$PWD":/work -w /work jasonish/suricata:8.0 -r day.pcap -S local.rules -k none -l out --set threshold-file=threshold.config
i: suricata: This is Suricata version 8.0.7 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 492 packets, 45964 bytes
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python triage.py
11 alerts in 4 groups
   8  172.30.0.31   /version     200
   1  172.30.0.20   /backup.zip  404
   1  172.30.0.20   /.env        404
   1  172.30.0.20   /admin       404

From 41 to 11, and all three probe alerts survived. --set threshold-file= points Suricata at the file for this run; on a sensor you’d set threshold-file in suricata.yaml.

It’s tempting to annotate the line where it is. Don’t:

# threshold.config
suppress gen_id 1, sig_id 1000002, track by_src, ip 172.30.0.30   # monitoring host
$ rm -rf out && mkdir out
$ docker run --rm --entrypoint suricata -v "$PWD":/work -w /work jasonish/suricata:8.0 -r day.pcap -S local.rules -k none -l out --set threshold-file=threshold.config
i: suricata: This is Suricata version 8.0.7 RELEASE running in USER mode
E: threshold-config: pcre2_match parse error, ret -1, string , track by_src, ip 172.30.0.30   # monitoring host
i: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 492 packets, 45964 bytes
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python triage.py
41 alerts in 5 groups
  30  172.30.0.30   /health      200
   8  172.30.0.31   /version     200
   1  172.30.0.20   /backup.zip  404
   1  172.30.0.20   /.env        404
   1  172.30.0.20   /admin       404

The trailing comment makes the line unparseable. Suricata logs one E: line, drops the suppression and carries on, and you’re back at 41 alerts. On a real sensor that error scrolls past in a start-up log nobody reads. Keep comments on their own lines, and compare the alert count after every tuning change.

Tuning 2: rate-limit per source

The deploy job still produces eight identical alerts. Instead of naming another host, cap the rule at one alert per source per hour.

# threshold.config: the monitoring host polls /health with curl all day
suppress gen_id 1, sig_id 1000002, track by_src, ip 172.30.0.30
# at most one alert per source per hour
threshold gen_id 1, sig_id 1000002, type limit, track by_src, count 1, seconds 3600
$ rm -rf out && mkdir out
$ docker run --rm --entrypoint suricata -v "$PWD":/work -w /work jasonish/suricata:8.0 -r day.pcap -S local.rules -k none -l out --set threshold-file=threshold.config
i: suricata: This is Suricata version 8.0.7 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 492 packets, 45964 bytes
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python triage.py
2 alerts in 2 groups
   1  172.30.0.20   /admin       404
   1  172.30.0.31   /version     200

Two alerts, and it looks tidy. But the probe now produces one alert that says /admin, and nothing about /.env or /backup.zip. The analyst sees a single curl request for /admin and may well close it. Which of the three requests survives isn’t even fixed: an earlier run of exactly this command kept /.env, because Suricata spreads flows across worker threads and the limit keeps whichever match it counts first. The other requests are still in the http events in eve.json, but only if someone goes looking. A limit is fine for a source you’ve already investigated; as a general noise filter it hides the shape of an attack.

Tuning 3: name the automation

Go back to suppression, and name both automated hosts, each with a comment on its own line saying what it is.

# threshold.config: known automation that uses curl on purpose
# monitoring host, polls /health
suppress gen_id 1, sig_id 1000002, track by_src, ip 172.30.0.30
# deploy job, reads /version
suppress gen_id 1, sig_id 1000002, track by_src, ip 172.30.0.31
$ rm -rf out && mkdir out
$ docker run --rm --entrypoint suricata -v "$PWD":/work -w /work jasonish/suricata:8.0 -r day.pcap -S local.rules -k none -l out --set threshold-file=threshold.config
i: suricata: This is Suricata version 8.0.7 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 492 packets, 45964 bytes
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python triage.py
3 alerts in 3 groups
   1  172.30.0.20   /.env        404
   1  172.30.0.20   /admin       404
   1  172.30.0.20   /backup.zip  404

Three alerts, all from the probe, all three paths visible. That’s the result you want from this traffic. The price is a blind spot: this rule can no longer see anything 172.30.0.30 or 172.30.0.31 does. If someone takes over the monitoring host, its curl requests are invisible to this rule. The first practice task shows that happening, and the second shows a narrower tuning.

TuningAlertsProbe paths visibleBlind spot
none413 of 3none, but buried in noise
suppress monitor113 of 3the monitor host
suppress monitor + limit 1/h21 of 3the monitor host, and most of every source’s activity
suppress both hosts33 of 3both automation hosts

The next lesson picks up when an alert is real: what to do with the affected container before you stop it.

Practice

  1. Capture a second pcap, day2.pcap, where the monitoring host runs the probe: --ip 172.30.0.30 ... sh traffic.sh probe. Run Suricata over it with the final threshold.config. How many alerts do you get?
Solution

Remove the stopped tap with docker rm tap, start a new one writing to /work/day2.pcap (the web server must still be running), run the probe with --ip 172.30.0.30, stop the tap, and run Suricata with -r day2.pcap. fast.log is empty: 0 alerts for three requests to /.env, /admin and /backup.zip. The suppression applies to everything from that IP, not just /health. This is the cost of allow-listing a host, and why the list needs an owner and a review date.

  1. Instead of suppressing hosts, narrow the rule so it ignores the two automation paths, using negated content on http.uri. Run it with no threshold.config over day.pcap and over day2.pcap.
Solution 
# path.rules: the curl rule, minus the two automation paths
alert http any any -> any any (msg:"LAB command-line HTTP client (curl)"; flow:established,to_server; http.user_agent; content:"curl/"; startswith; http.uri; content:!"/health"; content:!"/version"; classtype:policy-violation; sid:1000002; rev:2;)

It gives 3 alerts on day.pcap and 3 on day2.pcap, so the probe is caught even from the monitoring host. It has its own weakness: a negated content matches anywhere in the URI, so a request for /.env?x=/health would also be skipped. Every exception is a gap someone can aim for, so keep each one as narrow as you can and test its edge cases as well as the traffic it’s meant to silence.

  1. Replace the tuning with threshold gen_id 1, sig_id 1000002, type threshold, track by_src, count 5, seconds 60 and nothing else. Predict the alert count per source, then run it.
Solution

type threshold alerts on every fifth match per source within 60 seconds. The monitor’s 30 requests give 6 alerts, the deploy job’s 8 give 1, and the probe’s 3 give 0. The total is 7, and every one of them is benign. This setting suits rules where a single event is meaningless and volume is the signal, such as failed logins. On a rule where one event matters, it hides exactly the small, quiet activity you care about.

Clean up

docker rm -f web tap

Keep the ds-lab network, the capture-lab image and traffic.sh; the timeline lesson reuses them.

Frequently Asked Questions

What is alert triage in a SOC?
Triage is deciding, for each alert, whether it is a real problem, benign but expected, or a false positive, and how urgently it needs action. Analysts speed it up by grouping alerts by source, destination and rule, because a hundred identical alerts from a monitoring host are one decision, not a hundred. The goal is to spend time on the few alerts that differ from the pattern.
What is the difference between suppress and threshold in Suricata?
suppress stops a rule from alerting for a given source or destination IP entirely, which acts as an allow-list. threshold keeps the rule active for everyone but changes how often it alerts: type limit caps alerts per time window, type threshold alerts only on every Nth match, and type both combines them. Both live in threshold.config and apply by gen_id and sig_id.
Why is allow-listing an IP address risky?
Once an IP is suppressed for a rule, nothing that host does can trigger that rule, including activity by an attacker who has taken over the host. A monitoring server or CI runner is an attractive target precisely because it is trusted. Prefer narrowing the rule to the expected behaviour, such as a specific path, over silencing a whole host.
Can rate limiting alerts hide an attack?
Yes. A limit of one alert per source per hour keeps the first match and drops the rest, so the alert no longer shows everything the source did. A type threshold setting that only alerts on every fifth match can hide a short probe completely. Measure the alert counts before and after each change, per source, so you know what the tuning removed.
Why didn't my Suricata threshold.config take effect?
Suricata rejects a line it can't parse, logs an E: threshold-config error, and carries on without it, so the rule keeps alerting as before. A comment at the end of a line is a common cause; put comments on their own lines. Always check Suricata's start-up output and compare alert counts after editing the file.