DailyRNG
Login
Basic

Tokens as a language with a shape

The first decision is which language the text is written in. The language is a list of names, and the list is short enough to be written down once.

What you will see

One line with a place and a reason. That line is the whole answer to a bad character.

About 1 min of reading

Try It

Write the language list

1

Write the names

printf 'number\nplus\nname\nlparen\nrparen\n' > /tmp/L.txt; wc -l < /tmp/L.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
5. Five names is the language, and the language is the list.
2

Match the longest

printf '<=' | grep -c -f -e '<|<=' /dev/stdin; printf 'le\n' >> /tmp/w.txt; cat /tmp/w.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
le. The longer name won, which is the rule that consumed the text.
3

Name a failure place

printf 'pos=3 char=@ reason=no-name-starts-with-@\n' > /tmp/f.txt; cat /tmp/f.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line with a place and a reason. That line is the whole answer to a bad character.

Data Flow

Text to a list of names
1 The next characters are matched against the list
↓
2 The longest match wins
↓
3 The name and place are kept together
↓
4 A character that matches nothing names a place
Why this works
Why the longest match wins

Two names can start the same way. The longest is the one that consumed the text, and consumed is the word that decides.

The failure at this step

A character that starts no name. The failure names a place, and the place is the whole answer.

Examples

A token is a name and a place

The name says which rule matched. The place says where, and the place is what makes an error answer possible.

The list is the language

A name that is not on the list is not in the language. The list is the whole definition of what is allowed here.

Whitespace is a rule, not noise

It is dropped by a rule that says so. Dropping by rule is dropping visibly, which is dropping that can be reversed.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← What a compiler is for
Previous Lesson
What a compiler is for
Next Lesson
A grammar as rules that decide shape
A grammar as rules that decide shape →