The record of a build
Chunks, bytes, and the time. The record is what a size claim and a speed claim are checked against.
Building an index that fits
The library is larger than the memory. The index is built in pieces, and the pieces are the reason a large library is possible at all.
What you will see
One line. A failed build restarts at chunk 2, not at the start.
About 1 min of reading
Try It
Chunk to a limit and record
1
Write the memory limit
printf 'chunk_bytes 8000000\n' > /tmp/m.txt; cat /tmp/m.txt
Before you run it, say out loud what it will print. Then open the answer.
What it prints
One line. Bytes is the thing counted, which is memory.
2
Write the chunks
printf 'chunk1 8000000\nchunk2 8000000\nchunk3 1200000\n' > /tmp/c.txt; awk '{s+=$2} END{print s}' /tmp/c.txt
Before you run it, say out loud what it will print. Then open the answer.
What it prints
17200000. The sum is the index size, which is the claim.
3
Record the restart point
printf 'last_chunk=2 date=2026-10-10\n' > /tmp/r.txt; cat /tmp/r.txt
Before you run it, say out loud what it will print. Then open the answer.
What it prints
One line. A failed build restarts at chunk 2, not at the start.
Data Flow
A library to an index that fits
1
A chunk is built to the byte limit
↓
2
The chunk is written and released
↓
3
The chunks are merged by sorted key in one walk
↓
4
The last chunk is recorded for a restart
Why this works
When you have a choice
Why the limit is bytes rather than documents
Documents differ in size. A byte limit is a limit on the thing that is actually counted, which is memory.
Examples
The chunk is a memory limit
A chunk is built until the limit, then written and released. Released is the word that makes the next chunk possible.
The chunks are merged by key
Two chunks with the same key become one posting. One walk over sorted keys is the merge, and the walk is the cost.
The build is resumable
The last chunk written is recorded. Recorded means a failed build restarts at the chunk rather than at the start.
Words this lesson uses
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.
← Adding documents without a rebuild
Previous Lesson
Adding documents without a rebuild