Defensive Security · lesson 1 of 8
Intermediate
6- 2 Sigma Detection Rules: Catch SSH Brute Force and Map It to ATT&CK
- 3 Suricata on a Packet Capture: Write an IDS Rule That Fires
- 4 Alert Triage and Tuning: From 41 Alerts to the 3 That Matter
- 5 Incident Response: Preserve the Evidence, Then Contain
- 6 Building an Incident Timeline from Auth, Web and IDS Logs
- 7 Static File Analysis with EICAR and YARA: Hashes, file, strings and Rules
Also part of Defensive Security
6- + A Firing Alert Doesn't Mean Anyone Was Notified from Monitoring & Observability
- + MTTR Isn't a Guess — Compute It From Alertmanager's Own Timestamps from Monitoring & Observability
- + Logging Security Events Without Logging Secrets from Security Fundamentals
- + Watching the TCP Handshake with tcpdump from Networking
- + Host Firewalls with nftables: DROP Times Out, REJECT Refuses from Networking
- + Hunting in AWS CloudTrail Logs with Python from Cloud Security
Also part of: Linux
Reading Linux SSH Auth Logs: Failed Logins You Generate Yourself
Produce real failed and successful SSH logins against your own container, read them in auth.log, and parse them in Python without missing invalid users.
By SkillByExample · Published
Every detection starts with evidence, and the most basic evidence on a Linux server is the authentication
log. Instead of reading someone else’s sample, you’ll run your own SSH server in a container, fail to log
in five times, succeed once, and read exactly what the server wrote down. Tested with Docker Engine 29.8.2,
Debian 13.7 (debian:trixie-slim), OpenSSH 10.0p2, rsyslog 8.2504.0 and Python 3.12.15
(python:3.12-slim).
This is the first lesson of the topic. It assumes you can read a log line and run docker. The
security event logging lesson covers the application
side: what your own code should log. This one is the operating system’s side.
The lab
The server is Debian with openssh-server and rsyslog. A container has no syslog daemon by default,
so without rsyslog there would be no /var/log/auth.log at all. The kernel-log module is switched off
because a container can’t read the kernel log.
# Dockerfile
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends openssh-server rsyslog \
&& rm -rf /var/lib/apt/lists/* \
&& sed -i '/imklog/s/^/#/' /etc/rsyslog.conf \
&& mkdir -p /run/sshd \
&& echo 'PerSourcePenaltyExemptList 172.30.0.0/24' > /etc/ssh/sshd_config.d/lab.conf \
&& useradd -m -s /bin/bash alice && echo 'alice:correct-horse' | chpasswd
CMD rsyslogd && exec /usr/sbin/sshd -D
The PerSourcePenaltyExemptList line turns off one of OpenSSH’s own defences for the lab network.
Since OpenSSH 9.8, sshd penalises a source address that keeps failing to authenticate: after enough
failures it refuses new connections from that address for a while and logs
drop connection ... penalty: failed authentication. That’s exactly what you want on a real server, but
it makes this lab unpredictable, because the final, correct login is sometimes refused. Exempting the
lab subnet means every run produces the same log.
The client is a second image with ssh and sshpass, so a script can type the password.
# client/Dockerfile
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends openssh-client sshpass \
&& rm -rf /var/lib/apt/lists/*
docker build -t sshd-lab .
docker build -t ssh-client client
docker network create --internal --subnet 172.30.0.0/24 ds-lab
docker run -d --name sshd --hostname sshd --network ds-lab --ip 172.30.0.10 sshd-lab
The network is --internal, so neither container can reach anything outside your machine. The rest of
this topic reuses this network and the auth.log you’re about to create.
Making the evidence
The client tries three wrong passwords for the real user alice, then two accounts that don’t exist,
then the right password.
# logins.sh: three wrong passwords, two unknown users, one real login
opts="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"
try() { # try USER PASSWORD
if sshpass -p "$2" ssh $opts -o NumberOfPasswordPrompts=1 "$1@172.30.0.10" true 2>/dev/null
then echo "$1 / $2 -> ok"; else echo "$1 / $2 -> denied"; fi
}
try alice password1
try alice letmein
try alice 123456
try admin admin
try oracle oracle
try alice correct-horse
$ docker run --rm --network ds-lab --ip 172.30.0.20 -v "$PWD":/work -w /work ssh-client sh logins.sh
alice / password1 -> denied
alice / letmein -> denied
alice / 123456 -> denied
admin / admin -> denied
oracle / oracle -> denied
alice / correct-horse -> ok
That took about fifteen seconds. Each wrong password costs roughly two seconds, because sshd and PAM
add a delay after a failure on purpose to slow down guessing. The client got a fixed address,
172.30.0.20, so you’ll recognise it in the log.
What the server wrote
auth.log has about thirty lines for those six attempts. Filter it to the three messages that matter.
$ docker exec sshd grep -E 'Failed|Accepted|Invalid' /var/log/auth.log
2026-10-14T08:58:40.411405+00:00 sshd sshd-session[11]: Failed password for alice from 172.30.0.20 port 52794 ssh2
2026-10-14T08:58:43.534265+00:00 sshd sshd-session[14]: Failed password for alice from 172.30.0.20 port 52802 ssh2
2026-10-14T08:58:47.736645+00:00 sshd sshd-session[17]: Failed password for alice from 172.30.0.20 port 52804 ssh2
2026-10-14T08:58:48.381482+00:00 sshd sshd-session[20]: Invalid user admin from 172.30.0.20 port 36132
2026-10-14T08:58:50.782933+00:00 sshd sshd-session[20]: Failed password for invalid user admin from 172.30.0.20 port 36132 ssh2
2026-10-14T08:58:52.387293+00:00 sshd sshd-session[22]: Invalid user oracle from 172.30.0.20 port 36134
2026-10-14T08:58:53.744186+00:00 sshd sshd-session[22]: Failed password for invalid user oracle from 172.30.0.20 port 36134 ssh2
2026-10-14T08:58:54.158436+00:00 sshd sshd-session[24]: Accepted password for alice from 172.30.0.20 port 36142 ssh2
Each line has a timestamp, the hostname, the program and its PID, then the message. Debian’s rsyslog
writes the timestamp in RFC 3339 form with microseconds and an explicit +00:00 offset. Older systems
and other distributions use the classic Oct 14 08:58:40 form, which has no year and no timezone; the
timelines lesson shows why that matters.
The wrong-password and unknown-user cases look different. A real user gets one line. A user that doesn’t
exist gets an Invalid user line first, and its failure says for invalid user admin. The PID ties the
two together: both admin lines come from sshd-session[20].
The source port changes on every attempt, because each try is a new TCP connection. Group by IP, never by IP and port.
The program name changed
Plenty of older scripts and detection rules filter on the program name, sshd[.
$ docker exec sshd grep 'sshd\[' /var/log/auth.log
2026-10-14T08:58:35.068478+00:00 sshd sshd[1]: Server listening on 0.0.0.0 port 22.
2026-10-14T08:58:35.068525+00:00 sshd sshd[1]: Server listening on :: port 22.
$ docker exec sshd grep -c 'sshd-session\[' /var/log/auth.log
25
It finds the two startup lines and none of the logins. Since OpenSSH 9.8, the listening sshd hands
each connection to a separate program, sshd-session, and that’s the name on every authentication
message. A filter on sshd[ returns a clean result that looks like “no attacks”. This is the most
common way a detection fails: it doesn’t error, it just stops matching. Match on the message text, or on
both program names.
On a systemd host, the same messages are in the journal as well. There you’d use
journalctl -u ssh -g 'Failed password' (the unit is sshd on RHEL-family systems). This lab has no
systemd, so it reads the file.
Parsing it in Python
Copy the log out of the container, so a script can read it.
docker cp sshd:/var/log/auth.log .
A first attempt at counting failures per source and user:
# count_failures.py: failed SSH passwords per source IP and user
import re
from collections import Counter
FAILED = re.compile(r"Failed password for (\S+) from (\S+)")
fails = Counter()
with open("auth.log") as log:
for line in log:
if m := FAILED.search(line):
user, ip = m.groups()
fails[(ip, user)] += 1
for (ip, user), n in fails.most_common():
print(f"{ip:<14} {user:<8} {n}")
print("total failures:", sum(fails.values()))
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python count_failures.py
172.30.0.20 alice 3
total failures: 3
There were five failures, and it found three. For Failed password for invalid user admin from ...,
the (\S+) group takes invalid, then the pattern needs from and finds user instead, so the line
doesn’t match at all. The two failures it drops are the guesses at accounts that don’t exist, which are
the most typical sign of automated guessing. Nothing errors; the count is just low.
The fix makes invalid user an optional, non-capturing part of the pattern. While you’re there, catch
the success too, and say how many failures came before it.
# auth_summary.py: failures per IP and user, then any login that followed them
import re
from collections import Counter
FAILED = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
ACCEPTED = re.compile(r"^(\S+) .*Accepted \S+ for (\S+) from (\S+)")
fails = Counter()
with open("auth.log") as log:
for line in log:
if m := FAILED.search(line):
user, ip = m.groups()
fails[(ip, user)] += 1
elif m := ACCEPTED.search(line):
when, user, ip = m.groups()
prior = sum(n for (i, _), n in fails.items() if i == ip)
print(f"ACCEPTED {user} from {ip} at {when} after {prior} failures")
for (ip, user), n in fails.most_common():
print(f"{ip:<14} {user:<8} {n}")
print("total failures:", sum(fails.values()))
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python auth_summary.py
ACCEPTED alice from 172.30.0.20 at 2026-10-14T08:58:54.158436+00:00 after 5 failures
172.30.0.20 alice 3
172.30.0.20 admin 1
172.30.0.20 oracle 1
total failures: 5
All five failures are counted now. The ACCEPTED line is the one an analyst cares about: one source
guessed five times, then got in. On a real server, failures alone are constant background noise, but a
success from the same IP right after them is worth a look. Accepted \S+ matches password and
publickey, so key logins are caught as well.
This script counts every failure since the start of the file. Real detections add a time window, and the next lesson writes the same idea as a portable Sigma rule.
Practice
- Count how many attempts used a username that doesn’t exist, with a single
grep.
Solution
docker exec sshd grep -c 'Invalid user' /var/log/auth.log
It prints 2, one each for admin and oracle. Each unknown user gets exactly one Invalid user
line per connection, so counting those lines counts the attempts.
- Change the summary so it only reports source IPs with three or more failures, as an
ALERTline.
Solution
# threshold.py: report only source IPs with 3 or more failures
import re
from collections import Counter
FAILED = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
by_ip = Counter()
with open("auth.log") as log:
for line in log:
if m := FAILED.search(line):
by_ip[m[2]] += 1
for ip, n in by_ip.items():
if n >= 3:
print(f"ALERT {ip}: {n} failed passwords")
It prints ALERT 172.30.0.20: 5 failed passwords. Count by IP alone here. Counting by IP and user
would split the five failures into 3, 1 and 1, and the guesses at admin and oracle would never reach
the threshold.
- “After 5 failures” counts every failure since the log began. Make it count only failures in the ten
minutes before the login.
datetime.fromisoformatreads the RFC 3339 timestamps directly.
Solution
# window.py: only count failures in the 10 minutes before the login
import re
from datetime import datetime, timedelta
FAILED = re.compile(r"^(\S+) .*Failed password for (?:invalid user )?(\S+) from (\S+)")
ACCEPTED = re.compile(r"^(\S+) .*Accepted \S+ for (\S+) from (\S+)")
WINDOW = timedelta(minutes=10)
failures = [] # (time, ip)
with open("auth.log") as log:
for line in log:
if m := FAILED.search(line):
failures.append((datetime.fromisoformat(m[1]), m[3]))
elif m := ACCEPTED.search(line):
when, user, ip = datetime.fromisoformat(m[1]), m[2], m[3]
recent = [t for t, i in failures if i == ip and when - t <= WINDOW]
print(f"ACCEPTED {user} from {ip}: {len(recent)} failures in the last 10 min")
It prints ACCEPTED alice from 172.30.0.20: 5 failures in the last 10 min. Run logins.sh again an
hour later and the old failures drop out of the window, so only the new five count.
- Rerun
logins.shfrom a second client address,--ip 172.30.0.21, copy the log out again, and runauth_summary.py. What changes in the output?
Solution
There are two ACCEPTED lines, the table gains three rows for 172.30.0.21, and the total becomes 10.
The first login still reports 5 failures. The second login from 172.30.0.21 also reports 5, not 10, because
prior only sums failures from the same source IP. That’s the point of grouping by IP: two clients
don’t add up to one bigger attack.
Clean up
docker rm -f sshd
docker network rm ds-lab
Keep auth.log; the next lessons read it. If you remove the network, recreate it with the same
--subnet when a later lesson needs it.