Retesting and Verifying Fixes Across Rounds

The findings report went to the client, and they’ve fixed some of it. Now they want to know what’s actually fixed. A retest repeats the checks that proved each finding and compares the results with the earlier rounds. The answer for each finding is one of four: fixed, still open, new, or regressed (fixed once, and back again). This lesson builds that comparison in Python and starts with the matching bug that makes a renamed finding look fixed.

A retest is still testing, so it needs authorization: the scope letter usually names a retest window, and the scope checker should allow those dates before anything runs. Log the retest actions the same way as the first round. The examples need only Python 3.12 and run on Linux.

Two rounds of findings

Round 1 is the findings file from the report lesson, trimmed to the fields a retest needs. Each finding also records where it lives (host, port) and what kind of weakness it is (cwe, the CWE number). F-04, the setuid binary, is left out: it lives in the container image, not on a host and port, so it’s retested by repeating the image diff from the CVSS lesson.

[
  {"id": "F-01", "title": "Admin API answers without authentication", "severity": "Critical",
   "score": 9.8, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-306"},
  {"id": "F-02", "title": "Session cookie set without the Secure flag", "severity": "Medium",
   "score": 4.2, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-614"},
  {"id": "F-03", "title": "Error pages show stack traces", "severity": "Medium",
   "score": 5.3, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-209"},
  {"id": "F-05", "title": "Server header discloses the exact version", "severity": "Low",
   "score": 3.7, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-200"}
]

A different tester ran round 2. They numbered the findings afresh and reworded one title.

[
  {"id": "R2-01", "title": "Login form has no rate limiting", "severity": "Medium",
   "score": 5.3, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-307"},
  {"id": "R2-02", "title": "Session cookie missing the Secure flag", "severity": "Medium",
   "score": 4.2, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-614"},
  {"id": "R2-03", "title": "Server header discloses the exact version", "severity": "Low",
   "score": 3.7, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-200"}
]

Save them as round1.json and round2.json. Read side by side, the client fixed F-01 and F-03, the cookie and the Server header are still there, and the login form is new.

Matching by title

IDs changed between rounds, so the title looks like the obvious thing to match on.

# naive_retest.py: match round 1 and round 2 findings by their title
import json

before = {f["title"]: f for f in json.load(open("round1.json"))}
after = {f["title"]: f for f in json.load(open("round2.json"))}

for title in before:
    status = "STILL OPEN" if title in after else "FIXED"
    print(f"{status:<10}  {title}")
for title in after:
    if title not in before:
        print(f"{'NEW':<10}  {title}")

$ python naive_retest.py
FIXED       Admin API answers without authentication
FIXED       Session cookie set without the Secure flag
FIXED       Error pages show stack traces
STILL OPEN  Server header discloses the exact version
NEW         Login form has no rate limiting
NEW         Session cookie missing the Secure flag

The session cookie is reported twice: fixed under its old title and new under its new one. Nothing about the cookie changed except the wording. The client reads “fixed”, closes the ticket, and the cookie ships without its Secure flag. Matching by ID is worse here, because F-02 and R2-02 share nothing. Matching by list position fails the moment a finding is added or removed, since everything after it shifts by one.

A fingerprint from what the finding is

Match on facts that don’t depend on who wrote the report: the host, the port and the CWE. A hash of those three makes a short, stable fingerprint.

# retest.py: match findings across rounds by a fingerprint, not a title
import hashlib
import json


def fingerprint(f):
    key = f"{f['host']}:{f['port']}:{f['cwe']}"
    return hashlib.sha256(key.encode()).hexdigest()[:12]


before = {fingerprint(f): f for f in json.load(open("round1.json"))}
after = {fingerprint(f): f for f in json.load(open("round2.json"))}

for fp, f in before.items():
    status = "STILL OPEN" if fp in after else "FIXED"
    print(f"{status:<10}  {fp}  {f['id']:<5}  {f['title']}")
for fp, f in after.items():
    if fp not in before:
        print(f"{'NEW':<10}  {fp}  {f['id']:<5}  {f['title']}")

$ python retest.py
FIXED       00bdfce3a7de  F-01   Admin API answers without authentication
STILL OPEN  be62d8ae6e54  F-02   Session cookie set without the Secure flag
FIXED       1d56402d3a92  F-03   Error pages show stack traces
STILL OPEN  0acf297d6aed  F-05   Server header discloses the exact version
NEW         23f42d2c5692  R2-01  Login form has no rate limiting

Now the cookie is still open, once. 10.0.0.5:8080:CWE-614 hashes to be62d8ae6e54 in both rounds, whatever the title says. Every distinct problem now appears exactly once. The hash itself adds no security; it’s a compact label you can put in a report or a ticket. Twelve hex characters is plenty to keep a few hundred findings apart. Put the fingerprint in each finding’s ticket, and the client’s tracker can follow it across rounds too.

Regressions need the whole history

Round 3 comes back after another deployment. The stack-trace problem from F-03, fixed in round 2, is back under a new title.

[
  {"id": "R3-01", "title": "Stack trace returned on a malformed order ID", "severity": "Medium",
   "score": 5.3, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-209"},
  {"id": "R3-02", "title": "Login form has no rate limiting", "severity": "Medium",
   "score": 5.3, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-307"},
  {"id": "R3-03", "title": "Server header discloses the exact version", "severity": "Low",
   "score": 3.7, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-200"}
]

Comparing round 3 with round 2 alone would call R3-01 “new”. The program below takes every round in order and checks the earlier ones too, then prints the result as a Markdown section for the retest report.

# retest_report.py: classify the latest round against all earlier ones, as Markdown
import hashlib
import json
import sys

ORDER = {"regressed": 0, "new": 1, "still open": 2, "fixed": 3}
RANK = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Info": 4}


def fingerprint(f):
    key = f"{f['host']}:{f['port']}:{f['cwe']}"
    return hashlib.sha256(key.encode()).hexdigest()[:12]


rounds = [{fingerprint(f): f for f in json.load(open(p))} for p in sys.argv[1:]]
*earlier, previous, latest = rounds
seen_earlier = set().union(*earlier)
rows = [("still open" if fp in previous else "regressed" if fp in seen_earlier else "new", f)
        for fp, f in latest.items()]
rows += [("fixed", f) for fp, f in previous.items() if fp not in latest]
rows.sort(key=lambda r: (ORDER[r[0]], RANK[r[1]["severity"]], -r[1]["score"]))

print(f"## Retest: round {len(rounds)}\n")
print("| Status | ID | Severity | Title |\n| --- | --- | --- | --- |")
for status, f in rows:
    print(f"| {status} | {f['id']} | {f['severity']} | {f['title']} |")

$ python retest_report.py round1.json round2.json
## Retest: round 2

| Status | ID | Severity | Title |
| --- | --- | --- | --- |
| new | R2-01 | Medium | Login form has no rate limiting |
| still open | R2-02 | Medium | Session cookie missing the Secure flag |
| still open | R2-03 | Low | Server header discloses the exact version |
| fixed | F-01 | Critical | Admin API answers without authentication |
| fixed | F-03 | Medium | Error pages show stack traces |
$ python retest_report.py round1.json round2.json round3.json
## Retest: round 3

| Status | ID | Severity | Title |
| --- | --- | --- | --- |
| regressed | R3-01 | Medium | Stack trace returned on a malformed order ID |
| still open | R3-02 | Medium | Login form has no rate limiting |
| still open | R3-03 | Low | Server header discloses the exact version |
| fixed | R2-02 | Medium | Session cookie missing the Secure flag |

R3-01 is “regressed”: it’s in round 3, missing from round 2, and present in round 1. The *earlier, previous, latest unpacking splits the rounds so the latest is compared with the one before it, and anything older lands in earlier. With only two files, earlier is an empty list and set().union() returns an empty set, so nothing can be “regressed”. That’s correct, because a regression needs at least three rounds.

Regressions sort to the top on purpose. A new finding means the testing found something. A regression means a fix wasn’t protected, by a test or a configuration check, and the next deployment undid it. The “fixed” rows show the previous round’s ID and title, which is how the client knew it. Findings fixed in an earlier round and still absent aren’t listed: round 3 says nothing about F-01. To see them, compare against seen_earlier as well.

Practice

1. The client moves the app’s version banner to port 8443. Copy round3.json to round3-moved.json, change R3-03’s port to 8443, and run the report. What happens to the Server header finding?

$ python retest_report.py round1.json round2.json round3-moved.json
## Retest: round 3

| Status | ID | Severity | Title |
| --- | --- | --- | --- |
| regressed | R3-01 | Medium | Stack trace returned on a malformed order ID |
| new | R3-03 | Low | Server header discloses the exact version |
| still open | R3-02 | Medium | Login form has no rate limiting |
| fixed | R2-02 | Medium | Session cookie missing the Secure flag |
| fixed | R2-03 | Low | Server header discloses the exact version |

2. Round 4 finds two access-control bugs on the same host and port, both CWE-639, on different API paths. Copy round3.json to round4.json, renumber its findings R4-01 to R4-03, add the two new ones (with a path field), and run the report. Then write dupes.py to catch the problem, and fix the fingerprint.

  {"id": "R4-04", "title": "Any user can read another user's order", "severity": "Medium",
   "score": 6.5, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-639", "path": "/api/orders/{id}"},
  {"id": "R4-05", "title": "Any user can read another user's invoice", "severity": "Medium",
   "score": 6.5, "host": "10.0.0.5", "port": 8080, "cwe": "CWE-639", "path": "/api/invoices/{id}"}

$ python retest_report.py round1.json round2.json round3.json round4.json
## Retest: round 4

| Status | ID | Severity | Title |
| --- | --- | --- | --- |
| new | R4-05 | Medium | Any user can read another user's invoice |
| still open | R4-01 | Medium | Stack trace returned on a malformed order ID |
| still open | R4-02 | Medium | Login form has no rate limiting |
| still open | R4-03 | Low | Server header discloses the exact version |

# dupes.py: find findings in one round that share a fingerprint key
import json
import sys
from collections import defaultdict

ids = defaultdict(list)
for f in json.load(open(sys.argv[1])):
    ids[f"{f['host']}:{f['port']}:{f['cwe']}"].append(f["id"])

for key, found in ids.items():
    if len(found) > 1:
        print(f"DUPLICATE {key}: {', '.join(found)}")

$ python dupes.py round4.json
DUPLICATE 10.0.0.5:8080:CWE-639: R4-04, R4-05

    key = f"{f['host']}:{f['port']}:{f['cwe']}:{f.get('path', '')}"

$ python retest_report.py round1.json round2.json round3.json round4.json
## Retest: round 4

| Status | ID | Severity | Title |
| --- | --- | --- | --- |
| new | R4-04 | Medium | Any user can read another user's order |
| new | R4-05 | Medium | Any user can read another user's invoice |
| still open | R4-01 | Medium | Stack trace returned on a malformed order ID |
| still open | R4-02 | Medium | Login form has no rate limiting |
| still open | R4-03 | Low | Server header discloses the exact version |

3. Make retest_report.py exit 1 when anything has regressed, so a pipeline that builds the retest report can stop and get a human to look.

sys.exit(1 if any(status == "regressed" for status, _ in rows) else 0)

$ python retest_report.py round1.json round2.json > retest-2.md; echo "exit=$?"
exit=0
$ python retest_report.py round1.json round2.json round3.json > retest-3.md; echo "exit=$?"
exit=1