Master

Evaluating a retrieval system

Quality is a number you can compute, and computing it is what makes an improvement real. The measures are small: how often the right passage appears at all, where it appears, and whether the answer used it.

What you will see

The indexes that missed - indexes 2 and 3. Those two questions are the ones to read, and they are the fix list.

About 1 min of reading

Try It

Compute the measures.

The ranks

ranks = [1, 2, 12, None, 3] k = 4 hits = sum(1 for r in ranks if r is not None and r <= k) print(hits, len(ranks), round(hits / len(ranks), 2))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

3 5 0.6 - recall at 4 is 60 percent. The None is a question whose passage was not retrieved at all.

The misses

print([i for i, r in enumerate(ranks) if r is None or r > k])

Before you run it, say out loud what it will print. Then open the answer.

What it prints

The indexes that missed - indexes 2 and 3. Those two questions are the ones to read, and they are the fix list.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Questions and expected passages give numbers.

1 Run every question through retrieval

↓

2 Find the rank of the expected passage

↓

3 Compute recall at k and mean rank

↓

4 Check the answer cited a passage

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

ranks = [1, 2, 12, None, 3] sets nothing on screen

k = 4 sets nothing on screen

hits = sum(1 for r in ranks if r is not None and r <= k) sets nothing on screen

print(hits, len(ranks), round(hits / len(ranks), 2)) 3 5 0.6 - recall at 4 is 60 percent. The None is a question whose passage was not retrieved at all.

Why this works

Build the set from real questions

Questions from users, from the issue tracker, from what the team asks in chat. A set you invented tests your imagination, and the questions that break a system are the ones you did not imagine.

Track the number over time

A change to chunk size, model, or k moves the number. Recording it before and after is what turns a guess into a decision, and it is the number a reviewer can ask for.

Examples

Recall at k

The fraction of questions where a correct passage is in the top k. It answers one question: could the right text have been found at all, which is the retrieval half of the failure.

Where it ranked

A correct passage at rank nine is a miss at k=4 and a hit at k=20. Mean rank shows a system that finds the answer but buries it, which is fixed by reranking rather than by more context.

Groundedness is the other half

Did the answer use only the retrieved passages? A high recall with a low groundedness means retrieval is fine and the prompt is letting the model add its own facts.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← A retrieval system you can run

Previous Lesson

A retrieval system you can run

Next Lesson

Failure modes and guardrails

Failure modes and guardrails →