What goes into a penetration test report?

An executive summary for managers, with the scope, the dates and a count of findings by severity, and then one section per finding for the people who fix it. Each finding has a title, a severity with its CVSS score and vector, the evidence that proves it, and a concrete remediation. Reports usually end with the methodology and the scope that authorized the work.

Why not sort findings by severity name?

Because severity names sort alphabetically, not by seriousness. Critical, High, Low, Medium is the alphabetical order, so a reverse sort puts Medium first and Critical last. Map each name to a rank number and sort by the rank, then by CVSS score to order findings within a severity.

What counts as evidence in a findings report?

Something the client can reproduce or check: the request and the response that showed the bug, a header, a file listing, or a command and its output. Evidence must be specific enough that the client can confirm the fix with the same check. Redact any real credentials or personal data you saw while testing.

Who reads a penetration test report?

Two audiences. Managers and auditors read the executive summary to see how much risk there is and whether it's being handled. Engineers read the individual findings to reproduce and fix each one. Write the summary without jargon and the findings with exact detail.

Writing the Findings Report: From JSON to a Sorted Report

The report is what the client pays for. Testing finds the problems, but the report is what gets them fixed. It needs an executive summary a manager can read in a minute, then one section per finding that an engineer can act on. This lesson builds that report from a findings file with Python’s standard library. On the way, it hits the bug that puts a Critical finding at the bottom of the list.

The findings come from the earlier lessons: the scope that authorized the work, and the CVSS scores that rank them. The examples need only Python 3.12.

The findings file

Testers record each finding as they confirm it, so the file is in discovery order, not severity order. Each finding has a severity label, a CVSS score and vector, evidence and a remediation.

[
  {"id": "F-01", "title": "Admin API answers without authentication",
   "severity": "Critical", "score": 9.8, "cvss": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
   "evidence": "An anonymous GET /admin/users on 10.0.0.5:8080 returned 200 and the user list.",
   "remediation": "Require an admin session on every /admin route, and test that anonymous requests get 401."},
  {"id": "F-02", "title": "Session cookie set without the Secure flag",
   "severity": "Medium", "score": 4.2, "cvss": "CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:L/I:L/A:N",
   "evidence": "Login response header: Set-Cookie: session=...; HttpOnly; Path=/ (no Secure).",
   "remediation": "Add Secure and SameSite=Lax to the session cookie."},
  {"id": "F-03", "title": "Error pages show stack traces",
   "severity": "Medium", "score": 5.3, "cvss": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N",
   "evidence": "GET /api/orders/abc returned a Python traceback naming /srv/app/orders.py.",
   "remediation": "Turn off debug mode, return a generic error page, and log the detail server-side."},
  {"id": "F-04", "title": "Unexpected setuid binary in the app image",
   "severity": "High", "score": 7.8, "cvss": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
   "evidence": "Diff against the debian:12-slim baseline: -rwsr-xr-x root /usr/local/bin/backup-helper.",
   "remediation": "Remove the setuid bit (chmod 0755), or remove the binary if nothing needs it."},
  {"id": "F-05", "title": "Server header discloses the exact version",
   "severity": "Low", "score": 3.7, "cvss": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:N/A:N",
   "evidence": "Every response carries Server: nginx/1.24.0.",
   "remediation": "Set server_tokens off; in the nginx configuration."}
]

Each piece of evidence is something the client can repeat: a request and its response, a header, a diff. The session cookie in F-02 is shown as session=.... A report never carries a live credential or token, even one you captured during the test.

Worst first, the obvious way

The report should open with the worst finding. Sorting by the severity field, in reverse, looks like it does that:

# sort_findings.py: worst finding first?
import json

findings = json.load(open("findings.json"))
for f in sorted(findings, key=lambda f: f["severity"], reverse=True):
    print(f"{f['id']}  {f['severity']:<8}  {f['score']:>4}  {f['title']}")

$ python sort_findings.py
F-02  Medium     4.2  Session cookie set without the Secure flag
F-03  Medium     5.3  Error pages show stack traces
F-05  Low        3.7  Server header discloses the exact version
F-04  High       7.8  Unexpected setuid binary in the app image
F-01  Critical   9.8  Admin API answers without authentication

The Critical finding is last, and Medium is first. Python sorts strings alphabetically, so the order is Critical, High, Low, Medium, and reversing it gives Medium, Low, High, Critical. That’s almost exactly backwards. It’s easy to miss, because Medium findings are the most common and the list looks plausible. A client who reads the first page sees a cookie flag and stops reading before the unauthenticated admin API.

Inside the Medium pair, F-02 (4.2) also comes before F-03 (5.3). Python’s sort is stable, so items with equal keys keep their file order, and the score never entered the key.

A rank map, then the score

Give each severity a number in the order you mean, and sort by that number first and by score second:

# sort_findings.py: worst finding first, by rank and then by score
import json

RANK = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Info": 4}

findings = json.load(open("findings.json"))
for f in sorted(findings, key=lambda f: (RANK[f["severity"]], -f["score"])):
    print(f"{f['id']}  {f['severity']:<8}  {f['score']:>4}  {f['title']}")

$ python sort_findings.py
F-01  Critical   9.8  Admin API answers without authentication
F-04  High       7.8  Unexpected setuid binary in the app image
F-03  Medium     5.3  Error pages show stack traces
F-02  Medium     4.2  Session cookie set without the Secure flag
F-05  Low        3.7  Server header discloses the exact version

The key is a tuple, so Python compares ranks first and looks at the score only when two ranks are equal. Negating the score sorts higher scores first without reverse=True, which would also reverse the ranks. Now the two Medium findings are in score order as well. The rank map also documents the severity scale in one line. Info is in it even though nothing uses it yet, so an informational finding has a place when one turns up.

The report

The report puts the same sort to work, with an executive summary of counts in rank order, a table, and a section per finding.

# report.py: findings.json -> report.md, worst first
import json
from collections import Counter

RANK = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Info": 4}

findings = json.load(open("findings.json"))
findings.sort(key=lambda f: (RANK[f["severity"]], -f["score"]))
counts = Counter(f["severity"] for f in findings)
summary = ", ".join(f"{counts[s]} {s}" for s in RANK if counts[s])

lines = ["## Executive summary", "", f"{len(findings)} findings: {summary}.", "",
         "| ID | Severity | CVSS | Title |", "| --- | --- | --- | --- |"]
lines += [f"| {f['id']} | {f['severity']} | {f['score']} | {f['title']} |" for f in findings]
for f in findings:
    lines += ["", f"## {f['id']}: {f['title']}", "",
              f"**{f['severity']} ({f['score']})** `{f['cvss']}`", "",
              f"**Evidence:** {f['evidence']}", "",
              f"**Remediation:** {f['remediation']}"]

with open("report.md", "w") as out:
    out.write("\n".join(lines) + "\n")
print(f"wrote report.md: {summary}")

$ python report.py
wrote report.md: 1 Critical, 1 High, 2 Medium, 1 Low
$ head -n 19 report.md
## Executive summary

5 findings: 1 Critical, 1 High, 2 Medium, 1 Low.

| ID | Severity | CVSS | Title |
| --- | --- | --- | --- |
| F-01 | Critical | 9.8 | Admin API answers without authentication |
| F-04 | High | 7.8 | Unexpected setuid binary in the app image |
| F-03 | Medium | 5.3 | Error pages show stack traces |
| F-02 | Medium | 4.2 | Session cookie set without the Secure flag |
| F-05 | Low | 3.7 | Server header discloses the exact version |

## F-01: Admin API answers without authentication

**Critical (9.8)** `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H`

**Evidence:** An anonymous GET /admin/users on 10.0.0.5:8080 returned 200 and the user list.

**Remediation:** Require an admin session on every /admin route, and test that anonymous requests get 401.

The summary iterates over RANK, not over counts. A Counter keeps the order it first saw each key, which is the file’s discovery order. Walking the rank map gives Critical-to-Low every time, and if counts[s] skips severities with no findings. Reading a missing key from a Counter returns 0 instead of raising KeyError.

Every finding carries its vector next to its score, so a reviewer can check the score. Each remediation names a fix the client can test, such as “anonymous requests get 401”, so the retest can use the same check that found the problem. A full engagement report adds the scope, the dates and the methodology before the findings. Those come straight from scope.yaml.

Practice

Add an informational finding, F-06, with severity Info and score 0.0: the site has no /.well-known/security.txt. Where does it land in the report, and what does the summary say?

Solution

  {"id": "F-06", "title": "Security.txt file is missing", "severity": "Info", "score": 0.0,
   "cvss": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N",
   "evidence": "GET /.well-known/security.txt returned 404.",
   "remediation": "Publish a security.txt with a contact address."}

$ python report.py
wrote report.md: 1 Critical, 1 High, 2 Medium, 1 Low, 1 Info
$ head -n 12 report.md
## Executive summary

6 findings: 1 Critical, 1 High, 2 Medium, 1 Low, 1 Info.

| ID | Severity | CVSS | Title |
| --- | --- | --- | --- |
| F-01 | Critical | 9.8 | Admin API answers without authentication |
| F-04 | High | 7.8 | Unexpected setuid binary in the app image |
| F-03 | Medium | 5.3 | Error pages show stack traces |
| F-02 | Medium | 4.2 | Session cookie set without the Secure flag |
| F-05 | Low | 3.7 | Server header discloses the exact version |
| F-06 | Info | 0.0 | Security.txt file is missing |

It sorts last because Info has the highest rank number, and no code changed. CVSS itself calls 0.0 “None”. Reports usually say “Info” or “Informational” for an observation with no direct impact, which is why the rank map uses that name.

Rename F-03 to Error pages show stack traces | file paths and regenerate. Look at F-03’s table row. What will a Markdown renderer show, and how do you fix it?

Solution

$ python report.py
wrote report.md: 1 Critical, 1 High, 2 Medium, 1 Low
$ grep -n "F-03 |" report.md
9:| F-03 | Medium | 5.3 | Error pages show stack traces | file paths |

The | in the title starts a fifth cell in a four-column table. A GitHub-flavoured Markdown renderer drops the extra cell, so the title shows as “Error pages show stack traces” and “file paths” disappears. Escape the pipe in table cells:

def cell(text):
    return text.replace("|", "\\|")

Then use {cell(f['title'])} in the table row:

$ python report.py
wrote report.md: 1 Critical, 1 High, 2 Medium, 1 Low
$ grep -n "F-03" report.md
9:| F-03 | Medium | 5.3 | Error pages show stack traces \| file paths |
29:## F-03: Error pages show stack traces | file paths

The heading on line 29 doesn’t need escaping, because a pipe only means something inside a table. Evidence often contains pipes too (shell pipelines), so escape every field that goes into a table cell.

A tester labels F-03 High but leaves its score at 5.3. The report then puts a 5.3 above the remaining Medium and next to a 7.8. Write a check that flags any label that doesn’t match its CVSS band.

Solution

# check_bands.py: does each finding's label match its CVSS score?
import json


def band(score):
    if score == 0:
        return "Info"
    for limit, name in [(3.9, "Low"), (6.9, "Medium"), (8.9, "High")]:
        if score <= limit:
            return name
    return "Critical"


for f in json.load(open("findings.json")):
    if f["severity"] != band(f["score"]):
        print(f"{f['id']}: labelled {f['severity']}, but {f['score']} is {band(f['score'])}")

$ python check_bands.py
F-03: labelled High, but 5.3 is Medium

The bands are the CVSS v3.1 qualitative ratings, with “Info” for 0.0 as in this report. Sometimes a tester raises a severity on purpose, for business reasons. When that happens, record it with an environmental vector rather than a bare label, so the score and the label agree and the reason is written down.

A scanner export writes severities in lower case, so F-05 arrives as "low". What does report.py do, and how should it fail?

Solution

Unchanged, it crashes inside the sort with a traceback ending in KeyError: 'low'. That’s better than a wrong report, but it doesn’t say which finding is broken. Check every label before sorting (and add import sys):

findings = json.load(open("findings.json"))
unknown = [f"{f['id']}={f['severity']!r}" for f in findings if f["severity"] not in RANK]
if unknown:
    sys.exit(f"unknown severity: {', '.join(unknown)}; use one of {', '.join(RANK)}")

$ python report.py; echo "exit=$?"
unknown severity: F-05='low'; use one of Critical, High, Medium, Low, Info
exit=1

It’s tempting to call .capitalize() and carry on. That works for low, but it turns "informational" into "Informational", which still isn’t in the map. Refusing with the list of allowed values is clearer, and the person who exported the file can fix it there.

The findings file

Worst first, the obvious way

A rank map, then the score

The report

Practice

Frequently Asked Questions