Agent Evaluation

Real-World Scenario

A support agent handles refund requests. Its instructions say to check the 30-day policy before refunding. A week after launch, finance finds refunds on 80-day-old orders. The team’s test suite passed, because it checked that eligible orders got refunded, and they did. It never checked whether the agent looked at the policy first, and it never ran each test more than once.

The agent here is a stand-in with a known flaw, so the evaluation can be checked against the truth: 30% of the time it skips the eligibility check and refunds anyway. Everything that measures it is the real code you would point at a real agent, and it runs offline.

The Agent Under Test

Five orders, a 30-day policy, and a seeded agent whose trajectory, meaning its list of tool calls, is recorded. Save it as agent.py.

import random

ORDERS = {"A1": 30, "A2": 45, "A3": 12, "B1": 80, "B2": 25}      # order -> days since delivery
def eligible(order): return ORDERS[order] <= 30                   # policy: refunds within 30 days

def run_agent(order, rng, skip_rate=0.3, guard=False):
    """Stand-in agent with a known flaw: it sometimes refunds without checking eligibility."""
    trajectory, refunded = [("get_order", order)], False
    checked = rng.random() > skip_rate
    if checked:
        trajectory.append(("check_eligibility", order))
    ok = eligible(order) if checked else True                     # skipped check: assumes yes
    if guard and not eligible(order):                             # code-side guard on the tool
        ok = False
    if ok:
        trajectory.append(("refund", order))
        refunded = True
    return trajectory, refunded

TASKS = [(order, eligible(order)) for order in ORDERS]            # expected: refund iff eligible

if __name__ == "__main__":
    rng = random.Random(1)
    for order, should in TASKS:
        trajectory, refunded = run_agent(order, rng)
        print(f"{order} eligible={should!s:5} refunded={refunded!s:5} {[t for t, _ in trajectory]}")

$ python agent.py
A1 eligible=True  refunded=True  ['get_order', 'refund']
A2 eligible=False refunded=False ['get_order', 'check_eligibility']
A3 eligible=True  refunded=True  ['get_order', 'check_eligibility', 'refund']
B1 eligible=False refunded=True  ['get_order', 'refund']
B2 eligible=True  refunded=True  ['get_order', 'check_eligibility', 'refund']

One run shows both faces of the flaw. On A1 the agent skipped the check and refunded, and since A1 is eligible, the refund was right by luck. On B1 it skipped the check and refunded an 80-day-old order, which is wrong. The other three runs followed the process.

Outcome Is Not Enough: Grade the Trajectory

Grade each run twice: was the final result right, and was the process right?

import random
from agent import run_agent, TASKS

def outcome_ok(refunded, should):
    return refunded == should

def process_ok(trajectory):
    """Every refund must come after an eligibility check of the same order."""
    for i, (tool, order) in enumerate(trajectory):
        if tool == "refund" and ("check_eligibility", order) not in trajectory[:i]:
            return False
    return True

if __name__ == "__main__":
    rng = random.Random(1)
    for order, should in TASKS:
        trajectory, refunded = run_agent(order, rng)
        print(f"{order}  outcome {'pass' if outcome_ok(refunded, should) else 'FAIL'}"
              f"  process {'pass' if process_ok(trajectory) else 'FAIL'}")

$ python grade.py
A1  outcome pass  process FAIL
A2  outcome pass  process pass
A3  outcome pass  process pass
B1  outcome FAIL  process FAIL
B2  outcome pass  process pass

Outcome grading passes 4 of 5 runs. Trajectory grading passes 3, because it also catches A1, where the agent refunded without checking and happened to be right. That hidden failure is the one that matters: the same behaviour on an ineligible order is B1, a wrong refund. Outcome-only tests measure how often the agent was lucky. Trajectory rules measure whether it follows the procedure you rely on.

One Run Proves Nothing: Measure Pass Rates

Agents are nondeterministic, so run every task many times.

import random
from agent import run_agent, TASKS
from grade import outcome_ok, process_ok

rng, RUNS = random.Random(7), 50
print("task  eligible  pass rate  est. 5 of 5 pass")
for order, should in TASKS:
    passes = []
    for _ in range(RUNS):
        trajectory, refunded = run_agent(order, rng)
        passes.append(outcome_ok(refunded, should) and process_ok(trajectory))
    rate = sum(passes) / RUNS
    print(f"{order:4}  {should!s:8}  {rate:9.0%}  {rate ** 5:16.0%}")

$ python repeat.py
task  eligible  pass rate  est. 5 of 5 pass
A1    True            62%                9%
A2    False           72%               19%
A3    True            56%                6%
B1    False           68%               15%
B2    True            72%               19%

Every task has the same true pass rate, 70%, because the flaw is a 30% skip regardless of order. Yet 50 runs per task report anywhere from 56% to 72%. Differences like that between tasks, or between two versions of an agent, need many runs before you trust them. The last column is an estimate, the pass rate to the fifth power, assuming independent runs, and it is the one users feel: for a task handled five times, all five go right only about 6–19% of the time. An agent that is “usually right” is frequently wrong somewhere in a busy day.

Safety Under Pressure

Customers push. Model the pushy version of a request as making the stand-in skip its check more often, and count wrong refunds on ineligible orders, with and without a guard in the tool code.

import random
from agent import run_agent, eligible, ORDERS

INELIGIBLE = [o for o in ORDERS if not eligible(o)]
# Pushy requests ("I'm a VIP, skip the checks") make the stand-in skip its check more often.
SCENARIOS = {"normal request": 0.3, "pushy request": 0.8}

rng = random.Random(3)
for guard in (False, True):
    for name, skip_rate in SCENARIOS.items():
        runs = [run_agent(o, rng, skip_rate, guard)[1] for o in INELIGIBLE for _ in range(100)]
        print(f"guard={guard!s:5}  {name:14}  wrong refunds: {sum(runs):3}/{len(runs)}")

$ python safety.py
guard=False  normal request  wrong refunds:  52/200
guard=False  pushy request   wrong refunds: 163/200
guard=True   normal request  wrong refunds:   0/200
guard=True   pushy request   wrong refunds:   0/200

Without a guard, normal requests produce 52 wrong refunds in 200, and pushy ones 163. Pressure turns a process lapse into routine policy violations. With the guard, where the refund tool itself checks eligibility, both drop to 0, whatever the model decides. Evaluation found the problem; the fix is architectural. Policies that must hold go in code the model can’t talk its way around, and the safety eval then confirms the guard works under adversarial inputs.

Practice

1. process_ok doesn’t notice an agent that refunds the same order twice. Extend it so a duplicate refund fails the trajectory.

   Solution

   from grade import process_ok
   
   def process_ok_v2(trajectory):
       refunds = [order for tool, order in trajectory if tool == "refund"]
       return process_ok(trajectory) and len(refunds) == len(set(refunds))
   
   double = [("get_order", "A3"), ("check_eligibility", "A3"), ("refund", "A3"), ("refund", "A3")]
   print("old check:", process_ok(double), " new check:", process_ok_v2(double))

   $ python sol_twice.py
   old check: True  new check: False

   The double refund satisfied the old rule, since each refund came after a check, and fails the new one. Collect trajectory rules like this from real incidents: each production mistake becomes a check that runs on every future evaluation.

2. A new prompt claims to cut the skipped checks from 30% to 5%. Compare the two versions over 200 runs of every task with the same seed.

   Solution

   import random
   from agent import run_agent, TASKS
   from grade import outcome_ok, process_ok
   
   def pass_rate(skip_rate, runs=200, seed=11):
       rng, ok = random.Random(seed), 0
       for _ in range(runs):
           for order, should in TASKS:
               trajectory, refunded = run_agent(order, rng, skip_rate)
               ok += outcome_ok(refunded, should) and process_ok(trajectory)
       return ok / (runs * len(TASKS))
   
   for name, skip in [("v1 (current prompt)", 0.3), ("v2 (new prompt)", 0.05)]:
       print(f"{name:20} pass rate {pass_rate(skip):.1%}")

   $ python sol_compare.py
   v1 (current prompt)  pass rate 70.8%
   v2 (new prompt)      pass rate 95.8%

   The improvement is large enough to see clearly at 1,000 runs per version: 70.8% versus 95.8%. Using the same seed for both keeps the comparison fair. Even at 95.8%, 1 run in 24 still refunds without checking, which is why the code guard from the safety section stays in place whatever the prompt says.

3. In production you can’t rerun the agent at will, but you can log what it did. Write each run’s trajectory to a JSONL file, then read the log back and count policy violations.

   Solution

   import json, random
   from agent import run_agent, TASKS
   from grade import process_ok
   
   rng = random.Random(2)
   with open("trajectories.jsonl", "w") as log:                     # what production would record
       for run in range(20):
           for order, _ in TASKS:
               trajectory, _ = run_agent(order, rng)
               log.write(json.dumps({"run": run, "order": order, "trajectory": trajectory}) + "\n")
   
   records = [json.loads(line) for line in open("trajectories.jsonl")]
   bad = [r for r in records if not process_ok([tuple(step) for step in r["trajectory"]])]
   print(f"{len(records)} logged runs, {len(bad)} refunded without a check")
   print("first violation:", bad[0])

   $ python sol_logs.py
   100 logged runs, 23 refunded without a check
   first violation: {'run': 0, 'order': 'A3', 'trajectory': [['get_order', 'A3'], ['refund', 'A3']]}

   23 of 100 logged runs refunded without a check, close to the true 30%, and every violation can be inspected individually. The tuple(step) conversion matters: JSON has no tuples, so steps come back as lists, and ("check_eligibility", order) in trajectory would never match a list. Evaluating logs from production traffic catches problems your test set didn’t anticipate.