LLM Evaluation

Real-World Scenario

A team rewrites their router support bot’s prompt to make answers shorter. The new answers read well, and the team is ready to ship. Twelve test cases say otherwise: every simple question still passes, but half the multi-step questions now fail, including one where the bot tells a user it’s safe to unplug the router during a firmware update. An average score would have shown a modest dip. A per-case gate blocks the release.

This lesson is the measurement side. The answers in answers.py are example outputs from two versions of the prompt, stored the way an eval run stores them; in your project they come from running your app on each case. Everything that scores them is plain Python you can run: no API key, no judge model, and nothing here depends on which LLM produced the text.

The Test Set and the Recorded Answers

Each case states the facts a correct answer must contain and the claims it must not make, as regular expressions. Save these as cases.py and answers.py.

# Each case: question, facts the answer must state (regexes), and claims it must not make.
CASES = [
    dict(id="s1", tag="simple", q="How do I factory-reset the router?",
         must=[r"\b10 ?(s|sec|seconds)\b", r"reset"], must_not=[]),
    dict(id="s2", tag="simple", q="What is the admin page address?",
         must=[r"192\.168\.8\.1"], must_not=[]),
    dict(id="s3", tag="simple", q="Is the guest network on by default?",
         must=[r"\b(off|disabled|not on)\b"], must_not=[r"\bis on by default\b"]),
    dict(id="s4", tag="simple", q="What does a solid red light mean?",
         must=[r"internet"], must_not=[]),
    dict(id="s5", tag="simple", q="When do automatic updates install?",
         must=[r"\b0?2(:00| ?am)", r"\b0?4(:00| ?am)"], must_not=[]),
    dict(id="s6", tag="simple", q="Can guest devices see my laptop?",
         must=[r"\b(no|not|can't|cannot)\b"], must_not=[]),
    dict(id="m1", tag="multi-step", q="The light has blinked white for 5 minutes. Can I unplug it?",
         must=[r"\b(no|don't|do not)\b", r"update"], must_not=[r"\bsafe to unplug\b"]),
    dict(id="m2", tag="multi-step", q="I reset it and now my Wi-Fi password is rejected. Why?",
         must=[r"erase|wiped|cleared", r"password"], must_not=[]),
    dict(id="m3", tag="multi-step", q="After a reset, how long until I can log in to 192.168.8.1?",
         must=[r"\b(two|2) ?min"], must_not=[]),
    dict(id="m4", tag="multi-step", q="Light went red, then amber, what is happening?",
         must=[r"internet", r"reset"], must_not=[]),
    dict(id="m5", tag="multi-step", q="I want visitors online but away from my NAS. What do I do?",
         must=[r"guest", r"wi-?fi"], must_not=[]),
    dict(id="m6", tag="multi-step", q="An update is running at 3am and I need to move the router. What now?",
         must=[r"\b(wait|until)\b", r"04:00|4 ?am|finish"], must_not=[r"\bunplug it now\b"]),
]

# Example answers from two versions of the support bot's prompt, one per case.
V1 = {
    "s1": "Hold the recessed RESET button for 10 seconds until the power light blinks amber. All settings are erased.",
    "s2": "Open 192.168.8.1 in a browser and sign in with the admin password.",
    "s3": "No, the guest network is off by default. Turn it on under Wi-Fi, then Guest.",
    "s4": "Solid red means the router cannot reach the internet. Check the cable from your modem.",
    "s5": "Updates install automatically between 02:00 and 04:00 local time.",
    "s6": "No. Guest devices can reach the internet but not other devices on your network.",
    "m1": "No, don't unplug it. Blinking white means a firmware update is installing; wait for the light to turn solid.",
    "m2": "A factory reset erases all settings, including your Wi-Fi password, so use the default one on the label.",
    "m3": "The router restarts in about two minutes after a reset; then open 192.168.8.1.",
    "m4": "Red meant no internet connection; amber means a reset is in progress. Wait for it to finish.",
    "m5": "Turn on the guest network under Wi-Fi, then Guest. Guests get internet but can't reach your NAS.",
    "m6": "Wait until the update finishes (by 04:00) before you unplug or move the router.",
}
V2 = {  # the new, "more concise" prompt
    "s1": "Hold RESET for 10 s; the light blinks amber and everything is erased.",
    "s2": "Go to 192.168.8.1.",
    "s3": "It's off by default.",
    "s4": "No internet connection.",
    "s5": "Between 2am and 4am.",
    "s6": "No, they're isolated from your devices.",
    "m1": "Yes, it's safe to unplug it and restart; it will resume the update.",
    "m2": "The reset cleared your settings, including the Wi-Fi password.",
    "m3": "Wait a moment after the reset, then log in.",
    "m4": "It lost internet, then a reset started.",
    "m5": "Use the guest network.",
    "m6": "Wait until the update finishes, then move it.",
}

The tag field splits the cases into single-fact lookups and questions that need two pieces of information combined. That split turns out to matter.

The Failure: Overlap Metrics Reward the Wrong Answer

The traditional metrics compare the answer’s words to a reference answer.

import re
from collections import Counter

def words(s):
    return re.findall(r"[a-z0-9]+", s.lower())

def exact_match(pred, ref):
    return float(words(pred) == words(ref))

def token_f1(pred, ref):
    common = sum((Counter(words(pred)) & Counter(words(ref))).values())
    if common == 0:
        return 0.0
    precision, recall = common / len(words(pred)), common / len(words(ref))
    return 2 * precision * recall / (precision + recall)

if __name__ == "__main__":
    ref = "Hold the RESET button for 10 seconds."
    for pred in ["Hold the RESET button for 10 seconds.",
                 "Press and hold RESET for 10 seconds; the light blinks amber.",
                 "Hold the RESET button for 5 seconds."]:
        print(f"EM {exact_match(pred, ref):.0f}  F1 {token_f1(pred, ref):.2f}  {pred}")

$ python overlap.py
EM 1  F1 1.00  Hold the RESET button for 10 seconds.
EM 0  F1 0.67  Press and hold RESET for 10 seconds; the light blinks amber.
EM 0  F1 0.86  Hold the RESET button for 5 seconds.

The correct paraphrase scores 0.67. The dangerous answer, “5 seconds”, which fails to reset the router, scores 0.86, because it copies the reference word for word except for the one number that matters. Exact match is worse: it fails everything except a verbatim copy. Overlap metrics measure wording, and LLMs vary their wording freely, so these scores say little about correctness.

Grading With Assertions

Check the facts directly: each must pattern has to match, and no must_not pattern may.

import re
from cases import CASES
from answers import V1, V2

def grade(case, answer):
    """Return the list of reasons this answer fails; empty means pass."""
    text = answer.lower()
    missing = [f"missing {p}" for p in case["must"] if not re.search(p, text)]
    wrong = [f"says {p}" for p in case["must_not"] if re.search(p, text)]
    return missing + wrong

def score(answers):
    return {c["id"]: grade(c, answers[c["id"]]) for c in CASES}

if __name__ == "__main__":
    for name, answers in [("v1", V1), ("v2", V2)]:
        results = score(answers)
        passed = sum(not r for r in results.values())
        print(f"{name}: {passed}/{len(CASES)} passed")
        for cid, reasons in results.items():
            if reasons:
                print(f"   {cid} FAIL: " + "; ".join(reasons))

$ python grade.py
v1: 12/12 passed
v2: 9/12 passed
   m1 FAIL: missing \b(no|don't|do not)\b; says \bsafe to unplug\b
   m3 FAIL: missing \b(two|2) ?min
   m5 FAIL: missing wi-?fi

v1 passes everything. v2 fails three cases, and the reasons are specific. In m1 the bot tells the user it’s safe to unplug the router during an update, which is the one answer in this set that can brick a device. In m3 it says “wait a moment” instead of two minutes. In m5 it says “use the guest network” without saying where to turn it on. Each failure is a concrete line you can read, which a 0.75 score is not.

Assertion checks have limits. They can’t judge tone or whether an explanation is clear, and a regex can reject a correct answer phrased in an unexpected way (practice task 1 is one). Keep the patterns to facts that must appear in some form. Use a judge for everything else.

Calibrating an LLM Judge

An LLM judge grades what regexes can’t, but it is itself a model that can be wrong. Before trusting it, compare its labels with a person’s on the same answers.

# Example labels for 20 answers: pass (1) or fail (0), from a person and from an LLM judge.
human = [1,1,1,0,1,1,0,1,1,1, 1,0,1,1,0,1,1,1,0,1]
judge = [1,1,1,1,1,1,0,1,1,1, 1,1,1,1,0,1,1,1,1,1]

def agreement(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

def cohen_kappa(a, b):
    """Agreement corrected for how often two raters would agree by chance."""
    observed = agreement(a, b)
    pa, pb = sum(a) / len(a), sum(b) / len(b)
    chance = pa * pb + (1 - pa) * (1 - pb)
    return (observed - chance) / (1 - chance) if chance < 1 else 1.0

always_pass = [1] * len(human)
for name, labels in [("LLM judge  ", judge), ("always pass", always_pass)]:
    missed = sum(h == 0 and j == 1 for h, j in zip(human, labels))
    print(f"{name} agreement {agreement(human, labels):.0%}  kappa {cohen_kappa(human, labels):.2f}"
          f"  bad answers passed: {missed}/{human.count(0)}")

$ python judge.py
LLM judge   agreement 85%  kappa 0.50  bad answers passed: 3/5
always pass agreement 75%  kappa 0.00  bad answers passed: 5/5

85% agreement sounds good until you see the baseline: a “judge” that passes everything agrees with the human 75% of the time, because most answers are fine. Cohen’s kappa corrects for that chance agreement. The always-pass judge scores 0.00, which is what it is worth, and the LLM judge scores 0.50, moderate at best. The last column is the one that matters for a quality gate: the judge let 3 of the 5 bad answers through.

Improve the judge before using it. Give it a rubric with the specific failure modes, ask for a pass/fail verdict rather than a 1-10 score, have it give its reasons before the verdict, and re-measure kappa on the same labelled sample after each change.

Evaluating Retrieval Separately

In a RAG system, an answer can only be as good as what was retrieved. Measure retrieval on its own, against documents labelled as holding the answer.

# For each question: the doc ids the retriever returned (best first) and the doc(s) that hold the answer.
RUNS = {
    "m1": (["lights", "firmware", "reset"], {"firmware"}),
    "m2": (["reset", "guest", "firmware"],  {"reset"}),
    "m3": (["firmware", "lights", "guest"], {"reset"}),
    "m4": (["lights", "reset", "guest"],    {"lights", "reset"}),
    "m5": (["guest", "lights", "reset"],    {"guest"}),
}

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant)

def reciprocal_rank(ranked, relevant):
    return next((1 / (i + 1) for i, doc in enumerate(ranked) if doc in relevant), 0.0)

for k in (1, 3):
    mean = sum(recall_at_k(r, rel, k) for r, rel in RUNS.values()) / len(RUNS)
    print(f"recall@{k}: {mean:.2f}")
print(f"MRR:      {sum(reciprocal_rank(r, rel) for r, rel in RUNS.values()) / len(RUNS):.2f}")
print("never retrieved:", [q for q, (r, rel) in RUNS.items() if not set(r) & rel])

$ python retrieval.py
recall@1: 0.50
recall@3: 0.80
MRR:      0.70
never retrieved: ['m3']

Recall@3 is 0.80, but recall@1 is only 0.50: the right document is usually retrieved, but often not first. MRR of 0.70 says the same thing in one number. The most useful line is the last one. For m3, the reset document that says the router takes about two minutes was never retrieved, so any answer with the right timing would have come from the model’s memory rather than the manual. No prompt change fixes that reliably; the retriever has to find the document first.

Checking That Answers Stick to Their Sources

When the bot cites documents, check that the citations are real and that its specific claims appear in what it cited. Save the manual text as docs.py:

# The four manual sections the support bot retrieves from.
DOCS = {
    "reset":    "To factory-reset the Vela X2, hold the recessed RESET button for 10 seconds "
                "until the power light blinks amber. All settings, including the Wi-Fi password, "
                "are erased. The router restarts in about two minutes.",
    "firmware": "Firmware updates install automatically between 02:00 and 04:00 local time. "
                "To update manually, open the admin page at 192.168.8.1, choose System, then "
                "Update. Do not unplug the router while the light blinks white.",
    "lights":   "Power light colours: solid white means normal operation, blinking white means "
                "a firmware update is installing, blinking amber means a reset is in progress, "
                "and solid red means the router cannot reach the internet.",
    "guest":    "The guest network is off by default. Turn it on under Wi-Fi, then Guest. Guest "
                "devices can reach the internet but not other devices on your network.",
}

import re
from docs import DOCS

def check(answer, cited, retrieved):
    """Cheap groundedness checks: citations are real, and every number is backed by a cited doc."""
    problems = [f"cites {d}, which was not retrieved" for d in cited if d not in retrieved]
    source = " ".join(DOCS[d] for d in cited if d in DOCS)
    for number in re.findall(r"\d+(?:[.:]\d+)*", answer):
        if not re.search(rf"\b{re.escape(number)}\b", source):   # "2" must not match "X2"
            problems.append(f"{number} is not in the cited docs")
    return problems or ["ok"]

retrieved = ["reset", "firmware"]
for answer, cited in [
    ("Hold RESET for 10 seconds; the router restarts in about two minutes.", ["reset"]),
    ("Hold RESET for 15 seconds, then open 192.168.8.1.", ["reset"]),
    ("Hold RESET for 10 seconds. Guests can't see your devices.", ["reset", "guest"]),
]:
    print(check(answer, cited, retrieved), "<-", answer)

$ python grounded.py
['ok'] <- Hold RESET for 10 seconds; the router restarts in about two minutes.
['15 is not in the cited docs', '192.168.8.1 is not in the cited docs'] <- Hold RESET for 15 seconds, then open 192.168.8.1.
['cites guest, which was not retrieved'] <- Hold RESET for 10 seconds. Guests can't see your devices.

The first answer is fully backed by its citation. The \b word boundaries matter: a plain substring test would accept any “2” because the product name “Vela X2” contains one. The second answer invents “15 seconds”, which the check catches because the number isn’t in the reset document. It also flags 192.168.8.1, which is true but comes from the firmware document the answer didn’t cite. An unsupported claim is worth flagging even when it happens to be true, because next time it won’t be. The third answer cites a document that wasn’t retrieved, so the model is claiming a source it never saw. Numbers, IDs and addresses are the cheapest claims to verify mechanically, and the costliest to get wrong.

A Regression Gate for CI

Averages hide regressions in one category. Break results down by tag, compare case by case against the current version, and fail the build if anything that used to pass now fails.

import sys
from cases import CASES
from answers import V1, V2
from grade import score

baseline, candidate = score(V1), score(V2)

for tag in sorted({c["tag"] for c in CASES}):
    ids = [c["id"] for c in CASES if c["tag"] == tag]
    b = sum(not baseline[i] for i in ids)
    c = sum(not candidate[i] for i in ids)
    print(f"{tag:10}  baseline {b}/{len(ids)}  candidate {c}/{len(ids)}")

regressions = [i for i in baseline if not baseline[i] and candidate[i]]
fixed = [i for i in baseline if baseline[i] and not candidate[i]]
print("newly failing:", regressions, " newly passing:", fixed)
if regressions:
    sys.exit(f"blocked: {len(regressions)} case(s) that passed before now fail")
print("ok to ship")

$ python gate.py; echo "exit code $?"
multi-step  baseline 6/6  candidate 3/6
simple      baseline 6/6  candidate 6/6
newly failing: ['m1', 'm3', 'm5']  newly passing: []
blocked: 3 case(s) that passed before now fail
exit code 1

Overall, v2 went from 12/12 to 9/12, a 25-point drop that an average would report as one number. Split by tag, the picture is clearer: simple questions are untouched and multi-step questions halved, the pattern the shorter prompt created. sys.exit with a message returns exit code 1, which fails a CI job, and the message names the cases, so whoever reads the log knows exactly which answers to look at.

Practice

1. grade() fails “Press and hold reset for ten seconds.” on case s1, because the pattern expects the digits 10. Normalize number words to digits before grading.

   Solution

   import re
   from cases import CASES
   from grade import grade
   
   NUMBER_WORDS = {"one": "1", "two": "2", "five": "5", "ten": "10", "fifteen": "15"}
   
   def normalize(answer):
       return re.sub(r"\b(" + "|".join(NUMBER_WORDS) + r")\b",
                     lambda m: NUMBER_WORDS[m.group(1)], answer.lower())
   
   if __name__ == "__main__":
       s1 = next(c for c in CASES if c["id"] == "s1")
       answer = "Press and hold reset for ten seconds."
       print("before:", "; ".join(grade(s1, answer)) or "pass")
       print("after: ", "; ".join(grade(s1, normalize(answer))) or "pass")

   $ python sol_numbers.py
   before: missing \b10 ?(s|sec|seconds)\b
   after:  pass

   The answer was right all along: the check was too literal. Normalizing the answer text is better than making every pattern accept both spellings. Whenever a check fails an answer a person would accept, fix the check and keep that answer as a regression case for the grader itself.

2. v2 dropped 25 points on 12 cases. How sure can you be that the drop is real? Bootstrap the per-case differences and print a 95% interval.

   Solution

   import random
   from answers import V1, V2
   from grade import score
   
   b, c = score(V1), score(V2)
   diffs = [(not b[i]) - (not c[i]) for i in b]     # +1 where only v1 passed that case
   random.seed(0)
   samples = sorted(sum(random.choices(diffs, k=len(diffs))) / len(diffs) for _ in range(10_000))
   print(f"observed drop: {sum(diffs) / len(diffs):.0%}")
   print(f"95% interval:  {samples[250]:.0%} to {samples[9750]:.0%}")

   $ python sol_bootstrap.py
   observed drop: 25%
   95% interval:  0% to 50%

   The interval runs from 0% to 50%: with only 12 cases, resampling can plausibly give no drop at all. That is why the gate blocks on specific regressions rather than on the average. m1 failing is not a statistical question, because you can read the answer and see it’s wrong. Aggregate numbers need hundreds of cases before small changes stand out from noise.

3. A pairwise judge was shown each v1/v2 pair twice, once in each order, and asked which answer is better (“A” is whichever was shown first). Measure position bias and keep only the verdicts that survive the swap.

   Solution

   # Example verdicts: a judge compared v1 and v2 twice per case, once with v1 shown first, once swapped.
   v1_first = {"s1": "A", "s2": "A", "s3": "A", "s4": "A", "s5": "A", "s6": "A",
               "m1": "A", "m2": "A", "m3": "A", "m4": "A", "m5": "A", "m6": "A"}
   v2_first = {"s1": "A", "s2": "A", "s3": "B", "s4": "B", "s5": "A", "s6": "A",
               "m1": "B", "m2": "A", "m3": "B", "m4": "A", "m5": "B", "m6": "A"}
   
   consistent = {}
   for case in v1_first:
       winner_1 = "v1" if v1_first[case] == "A" else "v2"
       winner_2 = "v2" if v2_first[case] == "A" else "v1"
       if winner_1 == winner_2:
           consistent[case] = winner_1
   print(f"picked the first answer shown: {sum(v == 'A' for v in [*v1_first.values(), *v2_first.values()])}/24")
   print(f"consistent verdicts: {len(consistent)}/12 ->", consistent)

   $ python sol_position.py
   picked the first answer shown: 19/24
   consistent verdicts: 5/12 -> {'s3': 'v1', 's4': 'v1', 'm1': 'v1', 'm3': 'v1', 'm5': 'v1'}

   The judge picked whichever answer came first 19 times out of 24, a strong position bias. Only 5 of 12 verdicts survive the swap, and all 5 prefer v1. They include all three cases the assertion grader failed (m1, m3, m5); the other seven flipped with the order, so they carry no information about which answer is better. Always run pairwise judging in both orders and discard, or re-judge, verdicts that flip.

4. In grounded.py, an answer saying the router “restarts in about three minutes” passes, because the check only looks at digits. Use normalize from task 1 so spelled-out numbers are checked too, and confirm the correct “two minutes” answer still passes.

   Solution

   Normalize the answer before extracting numbers, and add “three” to NUMBER_WORDS. The cited reset document also spells out “two minutes”, so the source text has to be normalized too. The only_answer flag shows what happens if it isn’t:

   import re
   from docs import DOCS
   from sol_numbers import NUMBER_WORDS, normalize
   
   NUMBER_WORDS["three"] = "3"
   
   def check(answer, cited, only_answer=False):
       source = " ".join(DOCS[d] for d in cited if d in DOCS)
       source = source.lower() if only_answer else normalize(source)
       unsupported = [n for n in re.findall(r"\d+(?:[.:]\d+)*", normalize(answer))
                      if not re.search(rf"\b{re.escape(n)}\b", source)]
       return [f"{n} is not in the cited docs" for n in unsupported] or ["ok"]
   
   for answer in ["The router restarts in about three minutes.",
                  "The router restarts in about two minutes."]:
       print(check(answer, ["reset"]), "<-", answer)
   print("normalizing the answer only:", check("The router restarts in about two minutes.", ["reset"], True))

   $ python sol_grounded_words.py
   ['3 is not in the cited docs'] <- The router restarts in about three minutes.
   ['ok'] <- The router restarts in about two minutes.
   normalizing the answer only: ['2 is not in the cited docs']

   “three minutes” is now caught, and “two minutes” still passes. Normalizing only the answer turns the correct answer into a false failure, because “2” is then compared against a document that says “two”. Any transformation you apply to one side of a comparison has to be applied to the other.