Basic

Text stored as numbers

Every letter is a number, and a file stores those numbers as bytes. The mapping from number to letter is a table, and the rule for reading it is the encoding.

What you will see

4 5

About 2 min of reading

Try It

Inspect how text becomes numbers.

1

Print the number for A.

print(ord("A"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
65
2

Print the letter for 65.

print(chr(65))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A
3

Show a word as numbers.

print([ord(c) for c in "owl"])

Before you run it, say out loud what it will print. Then open the answer.

What it prints
[111, 119, 108]
4

Compare character count and byte count.

s = "caf\u00e9" print(len(s), len(s.encode("utf-8")))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
4 5
The accent is one character but two bytes.

Data Flow

Characters are numbers by a shared table, and an encoding rule decides how many bytes each number takes.
1 letter is a code point
↓
2 code point stored as bytes
↓
3 utf-8 is the rule
↓
4 character count and byte count differ
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

print(ord("A")) 65
Why this works
The table is shared

Every program that says utf-8 uses the same table, so text written on one machine reads on another. The agreement is the whole point of the standard.

Count characters, size bytes

A string length counts characters. A file size counts bytes. They are equal only for plain letters, which is the common source of off-by-one bugs.

Examples

Letter to number

print(ord("A"))
print(ord("a"))

A is 65 and a is 97. Letters are numbers by a table, and the table is what an encoding defines.

Number to letter

print(chr(65))

The number 65 prints as A. The mapping goes both ways, which is how text is stored and read.

A word as numbers

print([ord(c) for c in "owl"])

Three letters are three numbers. A file holds the bytes of those numbers, nothing else.

Beyond one byte

print(ord("é"))

An accented letter is 233, which needs more than seven bits. Plain letters fit in one byte, most other characters do not.

The code point

print(hex(ord("é")))

The number for a character is its code point. The number is fixed worldwide, the bytes that store it depend on the rule.

Encoding is the rule

print("é".encode("utf-8"))
print("é".encode("utf-16"))

The same character becomes two different byte rows under two rules. The rule used has to be the rule read.

Why plain letters are safe

print("owl".encode("utf-8").decode("latin-1"))

Plain letters are the same bytes in the common rules, so they survive a mismatch. Everything else can change.

Text has a count and a size

s = "caf\u00e9"
print(len(s), len(s.encode("utf-8")))

Four characters take five bytes. Character count and byte count differ once accents appear.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Hexadecimal
Previous Lesson
Hexadecimal
Next Lesson
Negative numbers in bits
Negative numbers in bits →