Advanced

Adding knowledge with RAG

RAG lets a local model answer from your own files. It searches your documents, hands the relevant ones to the model, and the model answers from them.

What you will see

Top retrieved chunks for the question: similarity=0.9663 learning models improve with data. Vecto similarity=0.9657 Machine learning models improve with dat similarity=0.9637 models improve with data. Vector databa Context window only loads relevant

The full result is shown at the end of the exercise below.

About 1 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Retrieval-Augmented Generation with a vector database

Split documents into chunks and store their embeddings.

def chunk_text(text, chunk_size=100): return [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)] def embed(text): vec = [0.0] * 8 for i, ch in enumerate(text): vec[i % 8] += ord(ch) return vec library = chunk_text("Machine learning models improve with data. Vector databases store embeddings for retrieval. " * 6) print() print("Adding knowledge with RAG") print("Documents chunked into %d pieces" % len(library)) vector_db = [(chunk, embed(chunk)) for chunk in library]

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Adding knowledge with RAG Documents chunked into 6 pieces

Retrieve the closest chunks for a question by cosine similarity.

def cosine_similarity(a, b): dot = sum(x * y for x, y in zip(a, b)) norm_a = sum(x * x for x in a) ** 0.5 norm_b = sum(y * y for y in b) ** 0.5 return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0 question = "How do models improve?" q_vec = embed(question) print("Top retrieved chunks for the question:") scored = sorted(((cosine_similarity(q_vec, emb), chunk) for chunk, emb in vector_db), reverse=True) for score, chunk in scored[:3]: print(" similarity=%.4f %s" % (score, chunk[:40])) print("Context window only loads relevant chunks: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Data Flow

Your files are indexed, relevant chunks are searched at question time, and the model answers using only those chunks.

1 Index documents

↓

2 Store as vectors

↓

3 Search at runtime

↓

4 Answer from context

Why this works

Index your files

Turn your documents into searchable chunks and store them in a vector database. When you ask a question, the closest chunks are pulled out first.

Context window limits

A local model can only read so much at once. RAG sends only the most relevant chunks, not your whole library, so it fits inside the context window.

Examples

Fresh facts, not memory

You: "What does my contract say about refunds?"
1. Search your PDFs
2. Find the refund clause
3. Model answers from that text

The model does not need to be retrained. It just reads what you give it.

Embeddings make files searchable

Document -> split into chunks
Chunk -> turned into numbers (embedding)
Question -> same numbers
Find closest chunks

Embeddings are vectors that capture meaning. Similar ideas land close together in that space, so searching finds the chunks most like your question.

Chunk size matters

Chunk too big  -> too much noise
Chunk too small -> loses meaning
Chunk just right -> clear context

A typical chunk is a few hundred words with some overlap. Right-sized chunks help the model find the exact passage it needs without wasting context window space.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Quantization: smaller models, less memory

Previous Lesson

Quantization: smaller models, less memory

Next Lesson

Finetuning a local model

Finetuning a local model →