Basic

Normalizing text

Two words that mean the same thing must compare equal. Comparing equal is a decision about which differences are thrown away, and the decision is written.

What you will see

One line with a reason. The claim names the loss, which is the honest answer.

About 1 min of reading

Try It

Fold and prove equality

Write the two words

printf 'Cafe\ncafe\n' > /tmp/w.txt; tr 'A-Z' 'a-z' < /tmp/w.txt | sort -u | wc -l

Before you run it, say out loud what it will print. Then open the answer.

What it prints

1. One distinct word is the equality, which is the claim.

Write the rule order

printf '1 fold-case\n2 fold-accents\n3 split\n' > /tmp/o.txt; cat /tmp/o.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Three numbered lines. The order is written, which is what decides the answer.

Name the loss

printf 'loss=case reason=both-answer\n' > /tmp/l.txt; cat /tmp/l.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

One line with a reason. The claim names the loss, which is the honest answer.

Data Flow

Text to comparable words

1 The rules are written in order

↓

2 Case is folded by rule

↓

3 Accents are folded by table

↓

4 The boundaries are kept

Why this works

Why the rules are ordered

A fold before a split is not a split before a fold. The order changes the answer, so the order is written.

The one difference never thrown away

Word boundaries. A boundary is what makes two words two words, and two words is the smallest claim.

Examples

Case is thrown away

The throw is a rule that says so. A rule that says so is a rule that can be reversed in a search.

Accents are folded

The fold maps a letter to a base. A map is a table, and a table is a file that can be read.

The throw is not free

Two words that differ only in case now answer the same. The sameness is a claim, and the claim names the loss.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Reading documents honestly

Previous Lesson

Reading documents honestly

Next Lesson

Splitting into tokens

Splitting into tokens →