Embeddings find meaning and miss exact words; keyword search finds exact words and misses meaning. Real retrieval usually runs both and merges the results, because the questions that break one are the ones the other answers.
What you will see
The exact document. An embedding search may rank it lower than three other pages, which is why the keyword result is merged in.
About 1 min of reading
Quick check
Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.
Try It
See the exact token miss.
1
The question
q = 'Error E4021 meaning'
Before you run it, say out loud what it will print. Then open the answer.
What it prints
A question whose answer is keyed on an exact code.
2
The keyword hit
grep 'E4021' docs/*.md
Before you run it, say out loud what it will print. Then open the answer.
What it prints
The exact document. An embedding search may rank it lower than three other pages, which is why the keyword result is merged in.
Data Flow
Two searches merge into one list.
1Run the embedding search
↓
2Run the keyword search
↓
3Union the candidates
↓
4Rank the union
Why this worksExact tokens need exact search
If your questions contain codes, IDs, or names, keyword search is not optional. An embedding alone will quietly drop the exact match, and that drop is the failure mode to check for.
Measure the misses
Run a list of real questions and record which found nothing. That miss rate is the number that decides whether to add the other search, and guessing it is how a system ships broken.
Examples
Where each one fails
A part number, an error code, or a name is an exact token an embedding may rank low. A question phrased differently from the document is the one keyword search misses entirely.
Merge, do not choose
Run both, then combine the lists so a chunk found by either is a candidate. That union is what keeps a part number searchable without losing the paraphrased questions.
Rank the merged list
A chunk found by both searches is stronger than one found by either. Combining the two scores into one ranking is the standard way to merge without hand tuned weights.
Words this lesson uses
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.