A retrieval system is a program with a quality you can measure. Take real questions, write the passage that should answer each, and run the search. The list of questions that missed is the honest state of the system.
About 1 min of reading
Score a tiny system.
The test set
Before you run it, say out loud what it will print. Then open the answer.
The check
Before you run it, say out loud what it will print. Then open the answer.
Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.
A test needs a right answer. Naming the passage that should be retrieved turns a vague worry into a check, and it is the step that finds a missing document before a user does.
Chunk size, model, and k all change the score. Re-running the test set after each change is what keeps an improvement real rather than a shuffle that happened to look better.
Twenty questions from real users, each with the passage that answers it. Run the search and check the right passage is in the top few. That is a test suite, and it is the only way quality is measured.
How often the right passage is in the top k. That number, not a feeling about the answers, is what decides whether the system is ready, and it is the number a team can argue about.
Read the questions that missed and find what they have in common: a code, a phrasing, a missing document. Each pattern is a fix, and fixing hits changes nothing.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.