DailyRNG
Login
Basic

Scoring a whole run

A run is a set of cases through one model. The score is the pass rate, and it is meaningless unless the denominator is printed next to it. Twenty passes out of twenty is not the same as twenty out of two hundred.

What you will see

Line two. That line number is the case the next fix targets.

About 1 min of reading

Try It

Print a score with its denominator

1

Count passes

printf 'pass\nfail\npass\npass\n' > /tmp/grades.txt; grep -c '^pass' /tmp/grades.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Three.
2

Count the total

wc -l < /tmp/grades.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Four. The rate is three of four, and both numbers must appear.
3

Print the failing case

grep -n '^fail' /tmp/grades.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Line two. That line number is the case the next fix targets.

Data Flow

Cases to a printed run
1 Every case runs through the model
↓
2 Each result is graded
↓
3 Passes are counted against the total
↓
4 The rate and the failing list are printed
Why this works
Why the failing list is the deliverable

The list names the next fix. The rate only says a fix is needed somewhere.

Why the set size is chosen on purpose

The set is sized to the cost of one run. A set that costs more than the change it measures is too large to be useful.

Examples

The denominator always printed

Pass rate and count print together. A rate alone lets a small set look like a big win.

What a score does not say

The score does not say which cases failed. That list is printed separately, and it is the list people actually read.

A score that is too high

A ninety eight percent pass rate on a set of ten cases is a set that is too easy, not a model that is nearly perfect.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Grading with a rubric instead of a string
Previous Lesson
Grading with a rubric instead of a string
Next Lesson
Comparing two runs honestly
Comparing two runs honestly →