DailyRNG
Login
Master

Building a large site in chunks

A site that does not fit in one pass is a site that must be cut. The cut is honest only if the restart point is recorded, and the record is the claim.

What you will see

c3. The rerun starts after the record, which is not a redo.

About 1 min of reading

Try It

Record a restart point

1

Write the chunks

printf 'c1 400\nc2 400\nc3 400\n' > /tmp/c.txt; wc -l < /tmp/c.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
3. Three named sets is the cut.
2

Record the point

printf 'done c1\ndone c2\n' > /tmp/d.txt; tail -1 /tmp/d.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
done c2. The point is written, which is the claim.
3

Start after the point

printf 'c1 c2 c3\n' | tr ' ' '\n' | awk 'NR>2'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
c3. The rerun starts after the record, which is not a redo.

Data Flow

A large site to chunks and a record
1 The pages are cut into named sets
↓
2 Each set is built and reported
↓
3 The finished chunk is recorded
↓
4 A rerun reads the record and starts after it
Why this works
Why the cut is by page and not by byte

A page is the unit the report counts. Cutting at a unit the report counts is cutting without a second count.

The failure at this step

A rerun that starts at the beginning. The record is the answer, and the record names the chunk.

Examples

The chunk is a named set

The set is written before it is built. A written set is a set that a restart can be compared against.

The restart point is recorded

The last chunk that finished is written. A written point is a point that a rerun can start from.

A chunk that fails stops the run

The run stops and the point stays. The point staying is what makes a rerun not a redo.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Deploying gated by the report
Previous Lesson
Deploying gated by the report
Next Lesson
A wrong page as ordered prints
A wrong page as ordered prints →