DailyRNG
Login
Master

Evaluating your chatbot

You cannot improve what you do not measure. Learn to build an automated test suite of prompts and answers so every change to your bot stays correct over time.

What you will see

Running 3 test cases What are your hours? -> score 1.00 [PASS] How do I reset my password? -> score 1.00 [PASS] What is your refund policy? -> score 1.00 [PASS] Passed 3/3

About 2 min of reading

Try It

Automated test suite for a chatbot

1

Write a grader that checks a reply for the expected keywords.

# Automated test harness: a set of prompt/answer cases, scored automatically. def score_answer(answer, expected_keywords): low = answer.lower() hits = sum(1 for k in expected_keywords if k in low) return round(hits / len(expected_keywords), 2) cases = [ {"prompt": "What are your hours?", "answer": "We are open 9 to 5 every day."}, {"prompt": "How do I reset my password?", "answer": "Go to settings and choose reset password."}, {"prompt": "What is your refund policy?", "answer": "Our refund policy is 30 days."}, ] keywords = [["open", "9", "5"], ["reset", "password"], ["refund", "30"]] print("Evaluating your chatbot")

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Evaluating your chatbot
Use exact keywords so the grader is deterministic.
2

Score each case and report pass/fail, then print the total.

print("Running %d test cases" % len(cases)) passed = 0 for i, c in enumerate(cases): s = score_answer(c["answer"], keywords[i]) verdict = "PASS" if s >= 1.0 else "FAIL" print("%s -> score %.2f [%s]" % (c["prompt"], s, verdict)) if s >= 1.0: passed += 1 print("Passed %d/%d" % (passed, len(cases)))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Running 3 test cases What are your hours? -> score 1.00 [PASS] How do I reset my password? -> score 1.00 [PASS] What is your refund policy? -> score 1.00 [PASS] Passed 3/3

Data Flow

A curated set of questions and rubrics runs automatically; each answer is scored, and the results track correctness and quality across changes.
1 Author a set of test questions
↓
2 Provide expected answers or rubrics
↓
3 Send each prompt to the bot
↓
4 Score every answer against the rubric
↓
5 Record pass and fail results
↓
6 Run the suite after every change
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

# Automated test harness: a set of prompt/answer cases, scored automatically. sets nothing on screen
def score_answer(answer, expected_keywords): sets nothing on screen
low = answer.lower() sets nothing on screen
hits = sum(1 for k in expected_keywords if k in low) sets nothing on screen
return round(hits / len(expected_keywords), 2) sets nothing on screen
cases = [ sets nothing on screen
{"prompt": "What are your hours?", "answer": "We are open 9 to 5 every day."}, sets nothing on screen
{"prompt": "How do I reset my password?", "answer": "Go to settings and choose reset password."}, sets nothing on screen
{"prompt": "What is your refund policy?", "answer": "Our refund policy is 30 days."}, sets nothing on screen
] sets nothing on screen
keywords = [["open", "9", "5"], ["reset", "password"], ["refund", "30"]] sets nothing on screen
print("Evaluating your chatbot") Evaluating your chatbot
Why this works
What to measure

A chatbot can fail in several distinct ways. Track each one:

  • **Correctness*: does the answer have the right facts?
  • **Helpfulness*: does it actually solve the user's question?
  • **Safety*: does it avoid harmful or off-topic replies?
  • **Latency*: how long until the first word appears?

Decide which metrics matter for your product, then write tests that score them. Run the suite on every code change. Over time the suite becomes the proof that your improvements did not break anything else.

Examples

Building a test harness

A chatbot evaluation is a set of prompt-and-answer pairs you run automatically. Write a small tester:

const cases = [
  { prompt: 'What are your hours?', expected: 'open' },
  { prompt: 'How do I reset my password?', expected: 'reset-flow' }
];

async function runEval() {
  for (const c of cases) {
    const answer = await ask(c.prompt);
    console.log(c.prompt, '=>', score(c.prompt, answer));
  }
}

Each case checks whether the answer is correct or wrong. The score function can compare keywords, or ask a model to grade the reply. Run runEval() whenever you change the prompt to catch regressions.

Grading with a model

For nuanced answers, use a model to judge another model. Send the prompt, the answer, and the rubric together:

async function grade(question, answer, rubric) {
  const reply = await ask([
    { role: 'system', content: 'You are a strict grader. Score 1-5.' },
    { role: 'user', content: `Question: ${question}\nAnswer: ${answer}\nRubric: ${rubric}` }
  ]);
  return Number(reply);
}

This lets you measure quality beyond simple keyword matching. It costs a little, but it catches the problems hand-written tests miss. A good test suite is the foundation of a chatbot you can trust in production.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Giving the bot knowledge
Previous Lesson
Giving the bot knowledge
Next Lesson
Cost and rate limits
Cost and rate limits →