Basic

Grading with a rubric instead of a string

Some answers have no keyword to look for: quality, tone, and honesty. A rubric is a short written standard the grader is asked to apply, so the judgement has a definition rather than a memory.

What you will see

Three labels. Unsure is what makes a bad rubric visible instead of silent.

About 1 min of reading

Try It

Write a rubric that fits in four lines

1

Write the standard

printf 'Answer names the concept.\nAnswer gives one example.\nAnswer does not invent a number.\nOtherwise fail.\n' > /tmp/rubric.txt; wc -l < /tmp/rubric.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Four lines. Each line is checkable against one output.
2

Check every line is testable

grep -c 'should feel\|nice\|good' /tmp/rubric.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Zero. Words like nice or good are not checkable and must be rewritten.
3

Add the unsure label

printf 'pass\nfail\nunsure\n' > /tmp/labels.txt; cat /tmp/labels.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Three labels. Unsure is what makes a bad rubric visible instead of silent.

Data Flow

Rubric to a defensible grade
1 The rubric is written once
↓
2 The output and rubric are sent together
↓
3 The judge returns a label and a reason
↓
4 The reason is kept with the grade
Why this works
Where a rubric is too long

Past a few lines the rubric starts grading itself. Each line must be checkable on the output alone.

Why the grade scale is tiny

Pass, fail, unsure. A ten point scale invites disagreement that carries no more information than three labels.

Examples

A rubric in four lines

State the standard in the same words every time. The grader is not remembering a standard, it is reading one.

Why the rubric asks for a reason

A grade with a reason can be argued with. A grade alone is a coin flip nobody can correct.

The judge is a model too

A judge model has its own failures. That is why the rubric is short, and why the judge is never the only grade.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← One assertion against one output
Previous Lesson
One assertion against one output
Next Lesson
Scoring a whole run
Scoring a whole run →