DailyRNG
Login
Master

Evaluating agents

An agent is only as good as the goals it actually reaches. Measure that with a real test set, run it repeatedly, and watch the pass rate over time.

What you will see

Measuring reliability over repeated runs Runs: 6 Passes: 3 Reliability: 0.50 A single pass might be luck; repeated runs measure how reliably it performs.

About 2 min of reading

Try It

Pick a metric, run a real test set, and track pass rate

1

Build a small test set of real cases with golden answers.

print() print("Evaluating agents") print("Pick a metric before you build, or you won't know if it improved.") test_cases = [ {"in": "refund my order 123", "want": "start_refund(order=123)"}, {"in": "I'm angry", "want": "handoff_to_human"}, ] print("Real test cases: %d" % len(test_cases)) for c in test_cases: print(" in: %s -> want %s" % (c["in"], c["want"]))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Evaluating agents Pick a metric before you build, or you won't know if it improved. Real test cases: 2 in: refund my order 123 -> want start_refund(order=123) in: I'm angry -> want handoff_to_human
A small, real test set beats a huge fake one.
2

Score each agent output against the golden answer.

print() print("Scoring each case against the golden answer") agent_outputs = { "refund my order 123": "start_refund(order=123)", "I'm angry": "offhand comment", } for c in test_cases: got = agent_outputs.get(c["in"]) ok = got == c["want"] print(" %s -> %s (%s)" % (c["in"], got, "pass" if ok else "fail"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Scoring each case against the golden answer refund my order 123 -> start_refund(order=123) (pass) I'm angry -> offhand comment (fail)
3

Run cases repeatedly to measure reliability, not one lucky pass.

print() print("Measuring reliability over repeated runs") results = [] for c in test_cases: for _ in range(3): results.append(agent_outputs.get(c["in"]) == c["want"]) pass_rate = sum(results) / len(results) print("Runs: %d" % len(results)) print("Passes: %d" % sum(results)) print("Reliability: %.2f" % pass_rate) print("A single pass might be luck; repeated runs measure how reliably it performs.")

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Measuring reliability over repeated runs Runs: 6 Passes: 3 Reliability: 0.50 A single pass might be luck; repeated runs measure how reliably it performs.

Data Flow

Define a success metric, run a real test set, and track the score to catch regressions.
1 Choose a success metric
↓
2 Build a real test set
↓
3 Run and score every case
↓
4 Track over time and review samples
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

print() Evaluating agents
print("Evaluating agents") Pick a metric before you build, or you won't know if it improved.
print("Pick a metric before you build, or you won't know if it improved.") Real test cases: 2
test_cases = [ sets nothing on screen
{"in": "refund my order 123", "want": "start_refund(order=123)"}, sets nothing on screen
{"in": "I'm angry", "want": "handoff_to_human"}, sets nothing on screen
] sets nothing on screen
print("Real test cases: %d" % len(test_cases)) in: refund my order 123 -> want start_refund(order=123)
for c in test_cases: sets nothing on screen
print(" in: %s -> want %s" % (c["in"], c["want"])) in: I'm angry -> want handoff_to_human
Why this works
Automated eval pipelines

Evaluate agents in CI so regressions catch early:

New prompt/deploy -> run test cases ->
  pass rate < 95%? -> block deploy

Track pass rate over time. A slow decline usually means a model update made things worse.

Human-in-the-loop review

Automated metrics miss nuance. Add periodic human review:

Every week, read 20 random replies.
Rate quality. Feed findings back into tests.

Numbers tell you what broke. Humans tell you why.

Combine both for a trustworthy agent.

Examples

Defining success

An agent is only as good as its results

# Did it finish the task?
score = task.finished and task.satisfied_user
# Or compare to a golden answer
score = similarity(agent_out, reference)

Pick a metric before you build, or you won't know if it improved.

Why it works: a clear metric turns "seems fine" into data.

Building a test set

Real examples, not random ones

test_cases = [
    {"in": "refund my order 123",
     "want": "start_refund(order=123)"},
    {"in": "I'm angry", "want": "handoff_to_human"},
]

Run every case and count how many pass. Regression tests keep new features from breaking old ones.

A small, real test set beats a huge fake one.

Measuring reliability

Measure reliability, not one lucky pass

# Run each case a few times, count passes
results = [run_case(c) for c in test_cases for _ in range(3)]
pass_rate = sum(results) / len(results)
print('Reliability: ' + str(round(pass_rate, 2)))

A single pass might be luck. Running cases repeatedly measures how reliably the agent performs.

Why it works: reliability matters more than one lucky success.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Reasoning & planning
Previous Lesson
Reasoning & planning
Next Lesson
Agent security & prompt injection
Agent security & prompt injection →