Advanced

Encodings and why they differ

A file stores bytes, and an encoding is the rule that maps those bytes back to characters. Two tools that use different rules read the same bytes as different text.

What you will see

ok

About 1 min of reading

Try It

Encode and decode text.

1

Encode a word to utf-8 bytes.

print("owl".encode("utf-8"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
b'owl'
2

Decode two bytes that mean an accent.

print(b"\xc3\xa9".decode("utf-8"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
é
3

Encode text with a combining accent.

print("café".encode("utf-8"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
b'caf\xc3\xa9'
The accent takes two bytes, so the byte count is more than the character count.
4

Round-trip with the same rule.

print("ok".encode("utf-8").decode("utf-8"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
ok

Data Flow

An encoding maps bytes to characters by a rule, and a mismatch between the rule used and the rule read garbles text.
1 text encoded to bytes
↓
2 bytes need a rule
↓
3 a different rule reads differently
↓
4 same rule round-trips
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

print("owl".encode("utf-8")) b'owl'
Why this works
utf-8 is the default

utf-8 stores plain letters in one byte and everything else in more. It is the rule almost every modern file and web page uses, so it is the safe default.

Where the bug lives

Garbled characters are two tools disagreeing about the rule. The bytes are fine; the reading is wrong. Finding the rule that was used is the fix.

Examples

Encode and decode

print("owl".encode("utf-8"))

encode turns text into bytes using a rule. The bytes are what a file holds, and decode turns them back.

The same bytes, two rules

print(b"\xc3\xa9".decode("utf-8"))

Those two bytes mean one accented letter under utf-8. Read under another rule they mean two different characters.

A mismatch shows up

print("café".encode("utf-8"))

Text with an accent needs more than one byte. A tool that assumes one byte per character reads it wrong.

Pick one rule

print("ok".encode("utf-8").decode("utf-8"))

Encoding and decoding with the same rule returns the original. The whole discipline is using one rule everywhere.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← How a decimal is stored
Previous Lesson
How a decimal is stored
Next Lesson
Byte order
Byte order →