Why the position is kept at all
A word list answers whether a document has the words. A phrase needs the order, and the order costs one number per token.
The count is the record
Tokens per document. The count is what a size claim is checked against.
Splitting into tokens
A token is a word with a place. The place is the position, and the position is what makes a phrase answer possible later.
What you will see
refuse doc named. The refusal names the document, which is the record.
About 1 min of reading
Try It
Tokenize with positions
1
Write the break set
printf 'break space comma dot\n' > /tmp/b.txt; wc -w < /tmp/b.txt
Before you run it, say out loud what it will print. Then open the answer.
What it prints
4. The set is small, which is what makes the rule readable.
2
Split with positions
printf 'red fish blue fish\n' | awk '{for(i=1;i<=NF;i++) print $i, i-1}'
Before you run it, say out loud what it will print. Then open the answer.
What it prints
Four lines with numbers 0 to 3. The numbers are the positions.
3
Refuse a long token
printf 'token antidisestablishmentarianism\n' | awk '{print (length($2)>20) ? "refuse doc named" : "ok"}'
Before you run it, say out loud what it will print. Then open the answer.
What it prints
refuse doc named. The refusal names the document, which is the record.
Data Flow
Text to tokens with positions
1
The break set is written
↓
2
The text is split at breaks
↓
3
Each token keeps its position
↓
4
A token over the limit names the document
Why this works
Examples
The token carries a position
The position is a number. A number is a claim about order, and order is what a phrase needs.
The split rule is a set
What is not a letter or a number is a break. A set is small, and small is what makes the rule readable.
A long word is a claim
A token longer than the limit is refused. The refusal names the document, which is the record.
Words this lesson uses
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.
← Normalizing text
Previous Lesson
Normalizing text