Advanced

Finetuning a local model

Finetuning teaches a base model your specific style or facts. It changes the model's weights so it behaves the way you want, instead of only changing your prompts.

What you will see

Finetuning a local model With 2 examples: memorized With 25 examples: learned patterns With 300 examples: generalized Sweet spot is a few hundred examples: True

About 1 min of reading

Try It

Finetune a small model on a good set of examples

See how the outcome changes with different numbers of examples.

def finetune(examples): if len(examples) < 5: return "memorized" if len(examples) < 50: return "learned patterns" return "generalized" print() print("Finetuning a local model") few_examples = ["A", "B"] print("With %d examples: %s" % (len(few_examples), finetune(few_examples))) medium_examples = ["ex%d" % i for i in range(25)] print("With %d examples: %s" % (len(medium_examples), finetune(medium_examples))) good_examples = ["ex%d" % i for i in range(300)] print("With %d examples: %s" % (len(good_examples), finetune(good_examples))) print("Sweet spot is a few hundred examples: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Finetuning a local model With 2 examples: memorized With 25 examples: learned patterns With 300 examples: generalized Sweet spot is a few hundred examples: True

Quality beats quantity; too few examples make the model memorize instead of learn.

Data Flow

Examples are gathered, the model is finetuned on them, and the result is checked to ensure it learned the pattern, not just memorized.

1 Gather examples

↓

2 Finetune the model

↓

3 Check overfitting

↓

4 Test new inputs

Why this works

Enough data needed

Finetuning needs a good set of examples, usually dozens to hundreds. Too few examples make it memorize instead of learn. Quality beats quantity here.

It costs compute

Finetuning takes a strong GPU and time. For most people, a small model finetuned on a few hundred examples is the sweet spot.

Examples

Teaching a new habit

Base model:  knows general English
Finetuned:   writes in your company's tone

You feed it examples of the output you want, and it adapts. After finetuning it sounds like your brand.

LoRA is lightweight

Full finetune  -> rewrite all weights, needs GPU
LoRA           -> small adapter, runs on laptop

LoRA (Low-Rank Adaptation) trains only a tiny part of the model. It is much cheaper and faster, so most people use LoRA for a local model instead of rewriting everything.

Watching the loss

Epoch 1: loss 2.5
Epoch 5: loss 1.1
Epoch 10: loss 1.1 (stuck)

Loss measures how wrong the model is. It should drop as training continues, then level off. If it keeps dropping on training data but fails on tests, the model is overfitting and needs more data.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Adding knowledge with RAG

Previous Lesson

Adding knowledge with RAG

Next Lesson

Choosing hardware for local models

Choosing hardware for local models →