Basic

Keyword search beside it

Embeddings find meaning and miss exact words; keyword search finds exact words and misses meaning. Real retrieval usually runs both and merges the results, because the questions that break one are the ones the other answers.

What you will see

The exact document. An embedding search may rank it lower than three other pages, which is why the keyword result is merged in.

About 1 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

See the exact token miss.

1

The question

q = 'Error E4021 meaning'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A question whose answer is keyed on an exact code.
2

The keyword hit

grep 'E4021' docs/*.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The exact document. An embedding search may rank it lower than three other pages, which is why the keyword result is merged in.

Data Flow

Two searches merge into one list.
1 Run the embedding search
↓
2 Run the keyword search
↓
3 Union the candidates
↓
4 Rank the union
Why this works
Exact tokens need exact search

If your questions contain codes, IDs, or names, keyword search is not optional. An embedding alone will quietly drop the exact match, and that drop is the failure mode to check for.

Measure the misses

Run a list of real questions and record which found nothing. That miss rate is the number that decides whether to add the other search, and guessing it is how a system ships broken.

Examples

Where each one fails

A part number, an error code, or a name is an exact token an embedding may rank low. A question phrased differently from the document is the one keyword search misses entirely.

Merge, do not choose

Run both, then combine the lists so a chunk found by either is a candidate. That union is what keeps a part number searchable without losing the paraphrased questions.

Rank the merged list

A chunk found by both searches is stronger than one found by either. Combining the two scores into one ranking is the standard way to merge without hand tuned weights.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Ranking the chunks
Previous Lesson
Ranking the chunks
Next Lesson
Citing what was used
Citing what was used →