Should you shut down a compromised server immediately?

Usually not as the first step. Shutting down destroys volatile evidence such as running processes, network connections and files that only exist as open handles of a process. Collect that evidence first, then contain the system by isolating it from the network, and shut it down or rebuild it during recovery. The exception is active damage, such as encryption in progress, where stopping it outweighs the evidence.

What is the order of volatility in digital forensics?

It is the order in which evidence disappears, and so the order to collect it: CPU and memory state first, then running processes and network connections, then temporary files, then disk contents, then logs and backups held elsewhere. RFC 3227 describes it. For a container, that means process lists and /proc before docker diff and the filesystem image.

How do you recover a deleted binary from a running Linux process?

While the process is still running, /proc/PID/exe links to its executable even after the file has been deleted, and reading it returns the full binary. Copy it out with cat /proc/PID/exe and hash it. In a container you may need extra privileges for this, such as docker exec --privileged, and once the process exits that copy is gone for good.

How do you preserve a Docker container for forensics?

While it is still running, record docker top, docker inspect and docker diff, copy anything that only exists in /proc, then docker commit the container to an image and docker save that image to a tar file. Hash every file with sha256sum and store the hashes outside the incident. Then isolate the container with docker network disconnect instead of stopping it.

What does NIST SP 800-61 Revision 3 change?

Revision 3, published in 2025, replaces the old four-phase incident response lifecycle with recommendations mapped to the NIST Cybersecurity Framework 2.0 functions: Govern, Identify, Protect, Detect, Respond and Recover. Evidence handling sits under Respond, for example the outcome that incident data is collected with its integrity and provenance preserved.

Incident Response: Preserve the Evidence, Then Contain

The triage lesson ended with an alert worth acting on. The instinct is to stop the affected machine at once. In this lesson an intruder logs in to your lab container, starts a disguised process and deletes its binary. You’ll collect the evidence in order of volatility, hash it, and only then cut the container off the network. The last section shows exactly what stopping first throws away. Tested with Docker Engine 29.8.2, Docker CLI 29.9.0 (docker:29-cli), Debian 13.7 with OpenSSH 10.0p2, and the sshd-lab and ssh-client images from the auth logs lesson.

The incident

Start a fresh copy of the SSH server, then play the intruder: log in with the guessed password, leave a note, start a copy of sleep named kworker so it looks like a kernel thread, and delete the copy while it runs.

docker run -d --name victim --hostname victim --network ds-lab --ip 172.30.0.12 sshd-lab

# intruder.sh: log in with the guessed password and leave traces behind
opts="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"
sshpass -p correct-horse ssh $opts alice@$1 '
  mkdir -p /tmp/.x && cp /bin/sleep /tmp/.x/kworker
  echo "next: look for database credentials" > /tmp/.x/notes
  setsid /tmp/.x/kworker 86400 >/dev/null 2>&1 < /dev/null &
  rm /tmp/.x/kworker
  echo "traces left on $(hostname)"'

$ docker run --rm --network ds-lab --ip 172.30.0.20 -v "$PWD":/work -w /work ssh-client sh intruder.sh 172.30.0.12
traces left on victim

The “malware” is sleep, so nothing here is harmful, but the moves are real ones. Naming a process after a kernel thread is masquerading (ATT&CK T1036), and deleting the binary after starting it means a disk search won’t find it.

First look, without changing anything

$ docker top victim -eo pid,user,args
PID                 USER                COMMAND
328577              root                sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
328600              root                rsyslogd
328674              1000                /tmp/.x/kworker 86400

A process owned by UID 1000 (alice) running from a hidden directory in /tmp is not normal for an SSH server. The PIDs are host PIDs; inside the container the same process has a small PID of its own. docker top reads the process list from the host, so it doesn’t run anything inside the container the intruder may have tampered with.

Preserve, most volatile first

Running processes disappear before files on disk, so the script collects in that order and hashes everything at the end. NIST SP 800-61r3 (2025) names this step as Respond outcome RS.AN-07: incident data is collected, with its integrity and provenance preserved.

# collect.sh: preserve a running container's evidence, most volatile first
set -e
c=$1; out=evidence-$c
mkdir -p "$out"
docker top "$c" -eo pid,user,args > "$out/processes.txt"
docker exec --privileged "$c" sh -c 'ls -l /proc/[0-9]*/exe 2>/dev/null' > "$out/exe-links.txt"
for pid in $(docker exec --privileged "$c" sh -c \
    'for p in /proc/[0-9]*; do readlink $p/exe | grep -q "(deleted)" && echo ${p#/proc/}; done'); do
  docker exec --privileged "$c" cat "/proc/$pid/exe" > "$out/deleted-exe-pid$pid.bin"
done
docker inspect "$c" > "$out/inspect.json"
docker diff "$c" > "$out/diff.txt"
docker commit -m "IR preserve" "$c" "evidence/$c:preserved" > /dev/null
docker save "evidence/$c:preserved" -o "$out/image.tar"
date -u +%Y-%m-%dT%H:%M:%SZ > "$out/COLLECTED"
cd "$out" && sha256sum * > SHA256SUMS

The collection, inspection and containment commands in this lesson, from here to docker top after the disconnect, ran in the docker:29-cli image with the Docker socket mounted, so their output is from a Linux shell: docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":/work -w /work docker:29-cli sh. On a Linux host you can run them directly. The docker run ... ssh-client lines run on the host as usual.

$ sh collect.sh victim
$ ls evidence-victim
COLLECTED
SHA256SUMS
deleted-exe-pid23.bin
diff.txt
exe-links.txt
image.tar
inspect.json
processes.txt
$ cat evidence-victim/exe-links.txt
lrwxrwxrwx 1 root  root  0 Oct 14 08:24 /proc/1/exe -> /usr/sbin/sshd
lrwxrwxrwx 1 alice alice 0 Oct 14 08:24 /proc/23/exe -> /tmp/.x/kworker (deleted)
lrwxrwxrwx 1 root  root  0 Oct 14 08:24 /proc/26/exe -> /usr/bin/dash
lrwxrwxrwx 1 root  root  0 Oct 14 08:24 /proc/8/exe -> /usr/sbin/rsyslogd

/proc/PID/exe is a link to the program a process is running. For PID 23 the file is gone, marked (deleted), but the kernel still holds it open, so cat /proc/23/exe reads the whole binary back. That is the only copy left anywhere. /proc/26 is the sh that docker exec started for the listing itself.

--privileged matters here. Root inside a container doesn’t get CAP_SYS_PTRACE by default, and without it reading another user’s /proc/PID/exe fails with Permission denied. --privileged on docker exec grants it to that one command, not to the container.

$ sha256sum evidence-victim/deleted-exe-pid23.bin
0637e6d47579929cb72efa46f361861b319d62c62fe8a9d10731fd7655eb5936  evidence-victim/deleted-exe-pid23.bin
$ docker exec victim sha256sum /bin/sleep
0637e6d47579929cb72efa46f361861b319d62c62fe8a9d10731fd7655eb5936  /bin/sleep

The recovered binary’s hash is identical to /bin/sleep, so “kworker” is a renamed sleep. In a real case you would look that hash up in your threat-intel sources or a malware database; the YARA lesson works with hashes and file contents safely.

$ cat evidence-victim/diff.txt
C /run
A /run/motd.dynamic
A /run/rsyslogd.pid
A /run/sshd.pid
C /tmp
A /tmp/.x
A /tmp/.x/notes
C /var
C /var/log
A /var/log/syslog
A /var/log/auth.log

docker diff lists every file added (A) or changed (C) since the container started from its image. /tmp/.x/notes is there. The binary isn’t, because it was deleted, which is why the /proc step came first. auth.log holds the intruder’s login; the image saved by docker commit and docker save contains it and every other file, for analysis later on a separate machine.

$ cat evidence-victim/SHA256SUMS
f4fc44d055eca5d8b6c21e769d73f5890869b1dffd991e26910ceaac4ad93011  COLLECTED
0637e6d47579929cb72efa46f361861b319d62c62fe8a9d10731fd7655eb5936  deleted-exe-pid23.bin
61284c17c74f08ec4483c538053ff7d5dbd1c18776d8f5e8f85669829bed599e  diff.txt
cc5560a1e4c9e8de53a20e659a18380de57a14e1244f8758fb8c67d4a8672a4a  exe-links.txt
9f6163e5cc6c4055062ec3efda091f4ba3f699ae3e580dbe47ff9f1905978058  image.tar
e310a99f8760132ff651e1fe73295126df9dbf995e420cdf6beea5350dbaad64  inspect.json
453fa5155e43fe3cc9368164ee36f9c34493addba835a267f2076eb6b74bb60d  processes.txt
$ cd evidence-victim && sha256sum -c SHA256SUMS
COLLECTED: OK
deleted-exe-pid23.bin: OK
diff.txt: OK
exe-links.txt: OK
image.tar: OK
inspect.json: OK
processes.txt: OK

The hashes are what make the evidence trustworthy later. Record SHA256SUMS somewhere the incident can’t reach, such as the ticket, and anyone can show the files haven’t changed since collection. COLLECTED holds the UTC collection time, and it’s hashed with the rest.

Contain without stopping

Now cut the container off. Disconnecting it from the network ends the intruder’s access and keeps every process running for further investigation.

$ docker network disconnect ds-lab victim
$ docker inspect -f '{{json .NetworkSettings.Networks}}' victim
{}
$ docker top victim -eo pid,user,args
PID                 USER                COMMAND
328577              root                sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
328600              root                rsyslogd
328674              1000                /tmp/.x/kworker 86400

The container has no networks, and the disguised process is still running, so you can keep examining it. From the intruder’s side:

# reach.sh: can the intruder still get in?
ssh -o ConnectTimeout=3 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -o BatchMode=yes alice@172.30.0.12 true
echo "exit code $?"

$ docker run --rm --network ds-lab --ip 172.30.0.20 -v "$PWD":/work -w /work ssh-client sh reach.sh
ssh: connect to host 172.30.0.12 port 22: Connection timed out
exit code 255

Contained. On a real network the same step is a firewall rule, a switch port change or a cloud security group swap. The principle is the same: isolate, don’t destroy.

What stopping first loses

To see the cost of the instinctive response, run the same intrusion against two more containers: victim2, started normally, and victim3, started with --rm the way many people run throwaway containers.

docker run -d --name victim2 --hostname victim2 --network ds-lab --ip 172.30.0.13 sshd-lab
docker run -d --rm --name victim3 --hostname victim3 --network ds-lab --ip 172.30.0.14 sshd-lab

Run intruder.sh against 172.30.0.13 and 172.30.0.14; each prints traces left on victim2 or victim3. Then stop them before collecting anything.

$ docker stop victim2
victim2
$ docker top victim2
Error response from daemon: container 25ae135fc145f3bbeae2502f3dff61bd2c62e156796d224b187f4131c20dea62 is not running
$ docker exec --privileged victim2 ls -l /proc/23/exe
Error response from daemon: container 25ae135fc145f3bbeae2502f3dff61bd2c62e156796d224b187f4131c20dea62 is not running
$ docker diff victim2
C /tmp
A /tmp/.x
A /tmp/.x/notes
C /run
A /run/motd.dynamic
A /run/rsyslogd.pid
C /var
C /var/log
A /var/log/syslog
A /var/log/auth.log

The process list is gone, and so is the deleted binary: it existed only as an open file of a process that no longer exists. Restarting the container wouldn’t bring it back. The disk evidence survives, so you’d know something left a note in /tmp/.x, but not what was running or what it was. Even the disk changed: /run/sshd.pid is missing, because sshd removed it as it shut down.

$ docker stop victim3
victim3
$ docker diff victim3
Error response from daemon: No such container: victim3

With --rm, stopping the container deletes it, so the note, the auth log with the intruder’s login and everything else went with it. The only record left is whatever was shipped off the machine before, such as logs in a SIEM. That’s a good argument for central logging, and for checking how a container was started before you touch it.

Evidence	Collected first	Stopped first	Stopped first, `--rm`
Running processes	yes	lost	lost
Deleted binary	recovered, hashed	lost	lost
File changes (`docker diff`)	yes	yes, partly changed	lost
Auth log, notes, full image	yes	yes	lost

Clean up the extra containers with docker rm -f victim2. Keep victim for the practice tasks.

The next lesson puts the evidence in order: the failed logins, the web requests and the IDS alerts on one UTC timeline.

Practice

Copy evidence-victim somewhere else, change one line of diff.txt, and run sha256sum -c SHA256SUMS in the copy. What does it report?

Solution

$ echo 'A /tmp/innocent' > diff.txt
$ sha256sum -c SHA256SUMS
COLLECTED: OK
deleted-exe-pid23.bin: OK
diff.txt: FAILED
exe-links.txt: OK
image.tar: OK
inspect.json: OK
processes.txt: OK
sha256sum: WARNING: 1 of 7 computed checksums did NOT match

Any change to any file shows up, and the exit status is non-zero, so a script can check it too. This only proves anything if the original SHA256SUMS is stored somewhere the editor of the files can’t also change, which is why it goes into the ticket at collection time.

Examine the preserved image without connecting it to anything: print the intruder’s note from evidence/victim:preserved.

Solution

$ docker run --rm --network none evidence/victim:preserved cat /tmp/.x/notes
next: look for database credentials
$ docker run --rm --network none evidence/victim:preserved ls -la /tmp/.x
total 12
drwxrwxr-x 2 alice alice 4096 Oct 14 08:23 .
drwxrwxrwt 1 root  root  4096 Oct 14 08:23 ..
-rw-rw-r-- 1 alice alice   36 Oct 14 08:23 notes

--network none gives the analysis container no network at all, and passing a command replaces the image’s CMD, so sshd never starts. The file times show when the intruder wrote the note; the timeline lesson lines that up with the login in auth.log. Work on a copy: running a container from the image doesn’t change the image, but treat image.tar itself as read-only.

docker pause freezes every process in a container. Pause victim, then try docker top and docker exec. Why is pausing a weaker containment step than disconnecting the network?

Solution

docker top victim still lists all three processes, because it reads them from the host, but docker exec victim true fails with Container victim is paused, unpause the container before exec, so you can’t collect anything from inside while it’s paused. The container also stays on the network, so the moment someone unpauses it the intruder’s processes carry on as before. Pausing is useful for a few seconds, which is what docker commit does by default while it copies the filesystem, but isolation is the containment step. Run docker unpause victim afterwards.

Your colleague’s runbook says “on alert: docker restart the container”. List what that step destroys, using this lesson’s evidence table.

Solution

A restart stops and starts the container, so every running process and the deleted binary are lost, just as with docker stop. The intruder’s files in /tmp/.x survive, because a restart keeps the container’s filesystem, and auth.log survives, but new log lines and new pid files mix in with the old ones. If the container ran with --rm, there’s nothing to restart; docker stop alone deleted it. A runbook should say “collect, then isolate”, and leave restarts and rebuilds for the recovery phase.

The incident

First look, without changing anything

Preserve, most volatile first

Contain without stopping

What stopping first loses

Practice

Frequently Asked Questions