Master

Testing the rule, not the table

I listed expected outputs for a sort and the list was always incomplete. A property test states a rule that must hold for any input and generates the inputs for you, so it covers cases you never wrote down.

What you will see

============================== 1 passed in 0.10s ===============================

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Write a property test and read the shrunk counterexample it finds.

A buggy sort that drops the last element. State the property that length is preserved.

from hypothesis import given from hypothesis import strategies as st def my_sort(xs): return sorted(xs)[:-1] @given(st.lists(st.integers())) def test_length_preserved(xs): assert len(my_sort(xs)) == len(xs)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

>>>

Run it. The library finds a failing input and shrinks it to the smallest one, the empty list.

pytest test_prop.py

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Falsifying example: test_length_preserved( xs=[], ) assert len(my_sort([])) == len([]) 1 failed

The shrunk example is empty, which is the real bug: the sort drops an element. Property testing found the edge case without you writing it. That is the payoff over a table of examples.

Fix the sort and the property holds across the generated inputs.

def my_sort(xs): return sorted(xs) pytest test_prop.py

Before you run it, say out loud what it will print. Then open the answer.

What it prints

============================== 1 passed in 0.10s ===============================

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

You state a rule, the library generates many inputs, checks the rule on each, and on failure shrinks to the smallest counterexample.

1 State a rule for all inputs

↓

2 Generate many inputs

↓

3 Check the rule each time

↓

4 Shrink any failure to smallest

Why this works

Shrinking is the gift

A random failing input is useless to read. Shrinking finds the smallest input that still fails, which is usually the real bug in miniature. This is why property testing finds edge cases you did not think to write: it searches, then distils.

When you have a choice

Property versus example

An example test checks one input against one answer. A property test checks a rule that must hold for all inputs, and generates inputs to hunt for a counterexample. The property covers the space; the example covers the points you chose. For pure functions with an inverse or an invariant, a property is far stronger.

Examples

State the rule

@given(st.lists(st.integers()))
def test_sort_is_ordered(xs):
    ys = sorted(xs)
    assert all(ys[i] <= ys[i + 1] for i in range(len(ys) - 1))

You assert a property (the result is ordered), not an exact answer. The library feeds many random lists and checks the rule on each.

It shrinks the failure

Falsifying example: [9, 3, 5, 1]

When a random input breaks the rule, the library shrinks it to the smallest case that still fails. You get a tiny reproducible counterexample, not a huge random one.

Round-trip property

@given(st.text())
def test_encode_decode(s):
    assert decode(encode(s)) == s

A round trip must return the original. That single property tests encode and decode together across thousands of inputs you never listed.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Reading a stack trace

Previous Lesson

Reading a stack trace

Next Lesson

Testing the seam, not the world

Testing the seam, not the world →