Master

Optimizing prompts

A prompt is a variable you can measure. Run it against a fixed test set, score the output, and keep whatever version wins. That is how you optimize instead of guessing.

What you will see

Cost small model tuned prompt: 60 Cost big model vague prompt: 1200 Cheapest model hitting quality bar: True

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Optimize prompts with a repeatable test and score cycle

Run the write/test/score/edit cycle over several versions and track scores.

print() print("Optimizing prompts") cases = [ {"input": "summarize A", "ok": True}, {"input": "summarize B", "ok": True}, {"input": "summarize C", "ok": False}, {"input": "summarize D", "ok": True}, ] versions = [] for i in range(4): ok = sum(1 for c in cases if c["ok"]) + (1 if i == 0 else 0) score = round(ok / len(cases), 2) tokens = max(40, 60 - i * 5) versions.append((i, score, tokens)) print("v%d: score=%.2f tokens=%d" % (i, score, tokens)) best = max(versions, key=lambda v: v[1]) print("Best version: v%s with score %.2f" % (best[0], best[1]))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Optimizing prompts v0: score=1.00 tokens=60 v1: score=0.75 tokens=55 v2: score=0.75 tokens=50 v3: score=0.75 tokens=45 Best version: v0 with score 1.00

Write -> test -> score -> edit -> repeat. Track each version's score to see progress.

Weigh cost against model size to pick the cheapest model that still passes.

def cost(tokens, model_size): return tokens * model_size small_model = cost(60, 1) big_model = cost(120, 10) print("Cost small model tuned prompt: %d" % small_model) print("Cost big model vague prompt: %d" % big_model) print("Cheapest model hitting quality bar: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Cost small model tuned prompt: 60 Cost big model vague prompt: 1200 Cheapest model hitting quality bar: True

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Prompts are tested against a fixed set, scored, and edited in a loop until quality and cost are balanced.

1 Write a prompt variant

↓

2 Run it on the test set

↓

3 Score and compare

↓

4 Keep the best, iterate

Why this works

The optimization loop

Optimizing is a repeatable cycle:

Write -> test on cases -> score -> edit -> repeat

Track each version's score so you can see progress. Over time you learn which phrasings actually move the needle.

Balancing cost and quality

Bigger models cost more. A well-tuned prompt on a small model often beats a vague one on a big model.

Cost = tokens x model size
Quality = prompt quality x model size

Aim for the cheapest model that still hits your quality bar. That is the sweet spot of prompt engineering.

Examples

A/B testing prompts

variants = [prompt_a, prompt_b]
scores = [score(prompt, test_cases) for prompt in variants]
best = max(scores, key=lambda x: x[1])

Run both prompts against the same test set and keep the winner. This is prompt optimization.

Why it works: gut feeling is wrong; data tells the truth.

Trimming tokens

Before: "Could you please try to write a summary of the document that I provided?"
After:  "Summarize the document."

Long prompts cost more and are slower. Trim until quality holds.

If it works with fewer words, use fewer words.

A scoring rubric

def rubric(out):
    return score(out, accuracy) + score(out, tone)

A fixed rubric turns a vague feeling into numbers you can compare.

Why it works: you cannot optimize what you cannot measure.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Constrained output

Previous Lesson

Constrained output

Next Lesson

Reasoning techniques

Reasoning techniques →