Skip to main content
Defensive Security · lesson 1 of 8
Defensive Security beginner Lesson 1 of 8

Also part of: Linux

Reading Linux SSH Auth Logs: Failed Logins You Generate Yourself

Produce real failed and successful SSH logins against your own container, read them in auth.log, and parse them in Python without missing invalid users.

By SkillByExample · Published

defensive-security logs ssh auth-log soc docker

Every detection starts with evidence, and the most basic evidence on a Linux server is the authentication log. Instead of reading someone else’s sample, you’ll run your own SSH server in a container, fail to log in five times, succeed once, and read exactly what the server wrote down. Tested with Docker Engine 29.8.2, Debian 13.7 (debian:trixie-slim), OpenSSH 10.0p2, rsyslog 8.2504.0 and Python 3.12.15 (python:3.12-slim).

This is the first lesson of the topic. It assumes you can read a log line and run docker. The security event logging lesson covers the application side: what your own code should log. This one is the operating system’s side.

The lab

The server is Debian with openssh-server and rsyslog. A container has no syslog daemon by default, so without rsyslog there would be no /var/log/auth.log at all. The kernel-log module is switched off because a container can’t read the kernel log.

# Dockerfile
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends openssh-server rsyslog \
 && rm -rf /var/lib/apt/lists/* \
 && sed -i '/imklog/s/^/#/' /etc/rsyslog.conf \
 && mkdir -p /run/sshd \
 && echo 'PerSourcePenaltyExemptList 172.30.0.0/24' > /etc/ssh/sshd_config.d/lab.conf \
 && useradd -m -s /bin/bash alice && echo 'alice:correct-horse' | chpasswd
CMD rsyslogd && exec /usr/sbin/sshd -D

The PerSourcePenaltyExemptList line turns off one of OpenSSH’s own defences for the lab network. Since OpenSSH 9.8, sshd penalises a source address that keeps failing to authenticate: after enough failures it refuses new connections from that address for a while and logs drop connection ... penalty: failed authentication. That’s exactly what you want on a real server, but it makes this lab unpredictable, because the final, correct login is sometimes refused. Exempting the lab subnet means every run produces the same log.

The client is a second image with ssh and sshpass, so a script can type the password.

# client/Dockerfile
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends openssh-client sshpass \
 && rm -rf /var/lib/apt/lists/*
docker build -t sshd-lab .
docker build -t ssh-client client
docker network create --internal --subnet 172.30.0.0/24 ds-lab
docker run -d --name sshd --hostname sshd --network ds-lab --ip 172.30.0.10 sshd-lab

The network is --internal, so neither container can reach anything outside your machine. The rest of this topic reuses this network and the auth.log you’re about to create.

Making the evidence

The client tries three wrong passwords for the real user alice, then two accounts that don’t exist, then the right password.

# logins.sh: three wrong passwords, two unknown users, one real login
opts="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"
try() {  # try USER PASSWORD
  if sshpass -p "$2" ssh $opts -o NumberOfPasswordPrompts=1 "$1@172.30.0.10" true 2>/dev/null
  then echo "$1 / $2 -> ok"; else echo "$1 / $2 -> denied"; fi
}
try alice password1
try alice letmein
try alice 123456
try admin admin
try oracle oracle
try alice correct-horse
$ docker run --rm --network ds-lab --ip 172.30.0.20 -v "$PWD":/work -w /work ssh-client sh logins.sh
alice / password1 -> denied
alice / letmein -> denied
alice / 123456 -> denied
admin / admin -> denied
oracle / oracle -> denied
alice / correct-horse -> ok

That took about fifteen seconds. Each wrong password costs roughly two seconds, because sshd and PAM add a delay after a failure on purpose to slow down guessing. The client got a fixed address, 172.30.0.20, so you’ll recognise it in the log.

What the server wrote

auth.log has about thirty lines for those six attempts. Filter it to the three messages that matter.

$ docker exec sshd grep -E 'Failed|Accepted|Invalid' /var/log/auth.log
2026-10-14T08:58:40.411405+00:00 sshd sshd-session[11]: Failed password for alice from 172.30.0.20 port 52794 ssh2
2026-10-14T08:58:43.534265+00:00 sshd sshd-session[14]: Failed password for alice from 172.30.0.20 port 52802 ssh2
2026-10-14T08:58:47.736645+00:00 sshd sshd-session[17]: Failed password for alice from 172.30.0.20 port 52804 ssh2
2026-10-14T08:58:48.381482+00:00 sshd sshd-session[20]: Invalid user admin from 172.30.0.20 port 36132
2026-10-14T08:58:50.782933+00:00 sshd sshd-session[20]: Failed password for invalid user admin from 172.30.0.20 port 36132 ssh2
2026-10-14T08:58:52.387293+00:00 sshd sshd-session[22]: Invalid user oracle from 172.30.0.20 port 36134
2026-10-14T08:58:53.744186+00:00 sshd sshd-session[22]: Failed password for invalid user oracle from 172.30.0.20 port 36134 ssh2
2026-10-14T08:58:54.158436+00:00 sshd sshd-session[24]: Accepted password for alice from 172.30.0.20 port 36142 ssh2

Each line has a timestamp, the hostname, the program and its PID, then the message. Debian’s rsyslog writes the timestamp in RFC 3339 form with microseconds and an explicit +00:00 offset. Older systems and other distributions use the classic Oct 14 08:58:40 form, which has no year and no timezone; the timelines lesson shows why that matters.

The wrong-password and unknown-user cases look different. A real user gets one line. A user that doesn’t exist gets an Invalid user line first, and its failure says for invalid user admin. The PID ties the two together: both admin lines come from sshd-session[20].

The source port changes on every attempt, because each try is a new TCP connection. Group by IP, never by IP and port.

The program name changed

Plenty of older scripts and detection rules filter on the program name, sshd[.

$ docker exec sshd grep 'sshd\[' /var/log/auth.log
2026-10-14T08:58:35.068478+00:00 sshd sshd[1]: Server listening on 0.0.0.0 port 22.
2026-10-14T08:58:35.068525+00:00 sshd sshd[1]: Server listening on :: port 22.
$ docker exec sshd grep -c 'sshd-session\[' /var/log/auth.log
25

It finds the two startup lines and none of the logins. Since OpenSSH 9.8, the listening sshd hands each connection to a separate program, sshd-session, and that’s the name on every authentication message. A filter on sshd[ returns a clean result that looks like “no attacks”. This is the most common way a detection fails: it doesn’t error, it just stops matching. Match on the message text, or on both program names.

On a systemd host, the same messages are in the journal as well. There you’d use journalctl -u ssh -g 'Failed password' (the unit is sshd on RHEL-family systems). This lab has no systemd, so it reads the file.

Parsing it in Python

Copy the log out of the container, so a script can read it.

docker cp sshd:/var/log/auth.log .

A first attempt at counting failures per source and user:

# count_failures.py: failed SSH passwords per source IP and user
import re
from collections import Counter

FAILED = re.compile(r"Failed password for (\S+) from (\S+)")

fails = Counter()
with open("auth.log") as log:
    for line in log:
        if m := FAILED.search(line):
            user, ip = m.groups()
            fails[(ip, user)] += 1

for (ip, user), n in fails.most_common():
    print(f"{ip:<14} {user:<8} {n}")
print("total failures:", sum(fails.values()))
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python count_failures.py
172.30.0.20    alice    3
total failures: 3

There were five failures, and it found three. For Failed password for invalid user admin from ..., the (\S+) group takes invalid, then the pattern needs from and finds user instead, so the line doesn’t match at all. The two failures it drops are the guesses at accounts that don’t exist, which are the most typical sign of automated guessing. Nothing errors; the count is just low.

The fix makes invalid user an optional, non-capturing part of the pattern. While you’re there, catch the success too, and say how many failures came before it.

# auth_summary.py: failures per IP and user, then any login that followed them
import re
from collections import Counter

FAILED = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
ACCEPTED = re.compile(r"^(\S+) .*Accepted \S+ for (\S+) from (\S+)")

fails = Counter()
with open("auth.log") as log:
    for line in log:
        if m := FAILED.search(line):
            user, ip = m.groups()
            fails[(ip, user)] += 1
        elif m := ACCEPTED.search(line):
            when, user, ip = m.groups()
            prior = sum(n for (i, _), n in fails.items() if i == ip)
            print(f"ACCEPTED {user} from {ip} at {when} after {prior} failures")

for (ip, user), n in fails.most_common():
    print(f"{ip:<14} {user:<8} {n}")
print("total failures:", sum(fails.values()))
$ docker run --rm -v "$PWD":/work -w /work python:3.12-slim python auth_summary.py
ACCEPTED alice from 172.30.0.20 at 2026-10-14T08:58:54.158436+00:00 after 5 failures
172.30.0.20    alice    3
172.30.0.20    admin    1
172.30.0.20    oracle   1
total failures: 5

All five failures are counted now. The ACCEPTED line is the one an analyst cares about: one source guessed five times, then got in. On a real server, failures alone are constant background noise, but a success from the same IP right after them is worth a look. Accepted \S+ matches password and publickey, so key logins are caught as well.

This script counts every failure since the start of the file. Real detections add a time window, and the next lesson writes the same idea as a portable Sigma rule.

Practice

  1. Count how many attempts used a username that doesn’t exist, with a single grep.
Solution
docker exec sshd grep -c 'Invalid user' /var/log/auth.log

It prints 2, one each for admin and oracle. Each unknown user gets exactly one Invalid user line per connection, so counting those lines counts the attempts.

  1. Change the summary so it only reports source IPs with three or more failures, as an ALERT line.
Solution
# threshold.py: report only source IPs with 3 or more failures
import re
from collections import Counter

FAILED = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
by_ip = Counter()
with open("auth.log") as log:
    for line in log:
        if m := FAILED.search(line):
            by_ip[m[2]] += 1

for ip, n in by_ip.items():
    if n >= 3:
        print(f"ALERT {ip}: {n} failed passwords")

It prints ALERT 172.30.0.20: 5 failed passwords. Count by IP alone here. Counting by IP and user would split the five failures into 3, 1 and 1, and the guesses at admin and oracle would never reach the threshold.

  1. “After 5 failures” counts every failure since the log began. Make it count only failures in the ten minutes before the login. datetime.fromisoformat reads the RFC 3339 timestamps directly.
Solution
# window.py: only count failures in the 10 minutes before the login
import re
from datetime import datetime, timedelta

FAILED = re.compile(r"^(\S+) .*Failed password for (?:invalid user )?(\S+) from (\S+)")
ACCEPTED = re.compile(r"^(\S+) .*Accepted \S+ for (\S+) from (\S+)")
WINDOW = timedelta(minutes=10)

failures = []  # (time, ip)
with open("auth.log") as log:
    for line in log:
        if m := FAILED.search(line):
            failures.append((datetime.fromisoformat(m[1]), m[3]))
        elif m := ACCEPTED.search(line):
            when, user, ip = datetime.fromisoformat(m[1]), m[2], m[3]
            recent = [t for t, i in failures if i == ip and when - t <= WINDOW]
            print(f"ACCEPTED {user} from {ip}: {len(recent)} failures in the last 10 min")

It prints ACCEPTED alice from 172.30.0.20: 5 failures in the last 10 min. Run logins.sh again an hour later and the old failures drop out of the window, so only the new five count.

  1. Rerun logins.sh from a second client address, --ip 172.30.0.21, copy the log out again, and run auth_summary.py. What changes in the output?
Solution

There are two ACCEPTED lines, the table gains three rows for 172.30.0.21, and the total becomes 10. The first login still reports 5 failures. The second login from 172.30.0.21 also reports 5, not 10, because prior only sums failures from the same source IP. That’s the point of grouping by IP: two clients don’t add up to one bigger attack.

Clean up

docker rm -f sshd
docker network rm ds-lab

Keep auth.log; the next lessons read it. If you remove the network, recreate it with the same --subnet when a later lesson needs it.

Frequently Asked Questions

Where are SSH login attempts logged on Linux?
sshd sends its messages to syslog with the auth facility. On Debian and Ubuntu, rsyslog writes them to /var/log/auth.log; on RHEL, Fedora and their relatives the file is /var/log/secure. On a systemd host the same messages are also in the journal, which you read with journalctl -u ssh on Debian or journalctl -u sshd on RHEL.
What does 'Failed password for invalid user' mean in auth.log?
It means someone tried to log in as an account that doesn't exist on the machine. sshd logs an 'Invalid user NAME from IP' line first, then 'Failed password for invalid user NAME'. Usernames such as admin, oracle, test or ubuntu on a host that doesn't have them are typical of automated password guessing.
Why does grep 'sshd\[' miss SSH login failures on newer systems?
OpenSSH 9.8 split the server into a listener and a per-connection program called sshd-session, so authentication messages are now tagged sshd-session[PID] rather than sshd[PID]. A filter written for older logs still finds the 'Server listening' lines but none of the logins. Match both names, or match on the message text instead of the program name.
Which SSH log line is the strongest sign of a successful brute-force attack?
An 'Accepted password' line from a source IP that has just produced a run of 'Failed password' lines. Failures alone are background noise on any internet-facing host; a success after them means a guess may have worked. Detection tools correlate the two by source IP and a time window rather than alerting on every failure.
Is it safe to practise SSH brute-force detection?
Only against machines you own, which is why this lesson builds its own SSH server in a container on an internal Docker network with no route out. A handful of wrong passwords from a second container is enough to produce every log line you need. Never point password-guessing tools at systems you aren't authorised to test.