Master

Deciding whether the review is actually worth keeping

A review system only improves if you watch it. Left unmeasured, it quietly gets worse -- catching fewer real bugs, then nobody trusting it, then nobody reading it. Measuring is what separates a system that stays useful from one that's just there for appearances.

What you will see

Missed: 'missing input validation' -> added to checklist Missed: 'unencrypted secret stored' -> added to checklist Total checklist items: 2 Every bug traced back to a pattern: True

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Trace missed bugs back into a living checklist

Record the checklist of patterns the review should catch.

checklist = [] def check_pattern(item): for rule in checklist: if rule in item: return True return False print() print("Measuring the review") print("Checklist items: %d" % len(checklist))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Measuring the review Checklist items: 0

For each production bug the review missed, add it to the checklist.

missed_bugs = [ "missing input validation", "unencrypted secret stored", ] for bug in missed_bugs: caught = check_pattern(bug) if not caught: checklist.append(bug) print("Missed: '%s' -> added to checklist" % bug) else: print("Caught: '%s'" % bug) print("Total checklist items: %d" % len(checklist)) print("Every bug traced back to a pattern: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Missed: 'missing input validation' -> added to checklist Missed: 'unencrypted secret stored' -> added to checklist Total checklist items: 2 Every bug traced back to a pattern: True

Data Flow

Track bugs that slip past, add patterns to the checklist, and watch quality improve over time.

1 Track misses

↓

2 Add a pattern

↓

3 Re-review

↓

4 Compare quality

Why this works

Count the missed bugs

Every bug that reaches production should be traced back. Did the AI miss it? If so, teach it to catch that specific pattern.

Turn misses into a checklist

Add each miss as a checklist item. Over time the AI review gets sharper and more useful, and the checklist itself becomes a living document that reflects your codebase.

Examples

Counting the bugs that slipped past

Track: after launch, did a bug get through the AI review?
If yes, add that pattern to your review checklist.

Month 1: AI caught 60% of bugs.
Month 3: AI caught 85% after tuning the prompts."

Every production bug is a learning opportunity. Trace it back to the review: was it caught, or was it missed? If it was missed, teach the system to spot that pattern next time.

Measuring the time you actually saved

Track: how long does a manual review take versus an AI review?
Before: 20 minutes per PR.
After: 5 minutes to check the AI's notes."

# Simple timer around the review step
start = time.time()
review = ai.review(pr)
elapsed = time.time() - start

Cutting review time lets teams ship faster, but only if you're honest about the number. Share the real figure with the team -- it's what makes the case for keeping the system running when budgets get tight.

Watching the false positives

Track: how often does the AI flag something that isn't actually a bug?
If the false positive rate is high, your prompt needs tuning."

Month 1: 20% false positives.
Month 3: 8% after tightening the prompts."

Fewer false positives means developers trust the tool, and trust is what makes an automated review stick. Once people stop reading the output because it's always wrong, the system is dead.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Making review happen on every push

Previous Lesson

Making review happen on every push