Basic

Reading documents honestly

The documents are files, and files lie. The lying is about encoding and size, and both are answered before any indexing happens.

What you will see

Two lines with numbers. The number is what the index is allowed to hold.

About 1 min of reading

Try It

Refuse a large file

Write the size limit

printf 'max_bytes 200000\n' > /tmp/m.txt; cat /tmp/m.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

One line. The limit is checked before the file is read.

Attempt a file over it

printf 'bytes 900000\n' | awk '{print ($1>200000) ? "refuse before read" : "read"}'

Before you run it, say out loud what it will print. Then open the answer.

What it prints

refuse before read. The refusal came before memory was asked for.

Assign a number id

printf 'aardvark.txt 0\nzebra.txt 1\n' > /tmp/i.txt; cat /tmp/i.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Two lines with numbers. The number is what the index is allowed to hold.

Data Flow

Files to ids and a record

1 The size is checked before reading

↓

2 The encoding is decoded once

↓

3 A number is assigned as the id

↓

4 Refusals are written with a reason

Why this works

Why the id is assigned at read

The index holds ids only. An id assigned later is an id that changes the index, and changing the index is a rebuild.

The record of a read

Count, refused, and the reason for each refusal. The record is the claim that the library was read completely.

Examples

The size is checked first

A file that is too large is refused. Refusing at the door is refusing before memory is asked for.

The encoding is decided once

One encoding for the whole library. One encoding is what makes two words compare equal later.

A document has an id

The id is a number, not the path. A number is stable, and stable is what an index is allowed to hold.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← What a search engine is for

Previous Lesson

What a search engine is for

Next Lesson

Normalizing text

Normalizing text →