Basic

Encoding and bytes

A file is bytes. Text is numbers mapped to symbols. The encoding is the map, and picking the wrong one turns your text into the wrong shapes.

What you will see

4 2

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Count bytes against letters.

Bytes of two letters.

data = "hi".encode("utf-8") print(len(data), data[0])

Before you run it, say out loud what it will print. Then open the answer.

What it prints

2 104

Bytes of one accented letter.

data = "é".encode("utf-8") print(len(data))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Round trip.

data = "hi".encode("utf-8") print(data.decode("utf-8"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Bytes of two accented letters.

data = "éé".encode("utf-8") print(len(data), 2)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

4 2

Data Flow

Files hold bytes. An encoding maps bytes to symbols, and the same bytes read with the wrong map give wrong symbols.

1 hold the bytes

↓

2 pick an encoding

↓

3 decode once

↓

4 name it on open

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

data = "hi".encode("utf-8") sets nothing on screen

print(len(data), data[0]) 2 104

Why this works

Bytes first, text second

Every file is bytes. The encoding is a choice about how to read them, and the same bytes can be read many ways.

utf-8 is the safe default

Name utf-8 on every open. The platform default differs between machines, which is how files break when they travel.

Examples

Bytes are numbers

data = "hi".encode("utf-8")
print(len(data), data[0])

Two letters became two bytes. The first byte is 104, the code for h.

One letter, many bytes

data = "é".encode("utf-8")
print(len(data))

An accented letter is two bytes in utf-8. Length in bytes is not length in letters.

Decode with the same map

data = "hi".encode("utf-8")
back = data.decode("utf-8")
print(back)

Encode and decode with the same name and you get the original back.

The wrong map breaks

data = "é".encode("utf-8")
print(len(data))

Reading those two bytes as a single byte map gives two wrong symbols instead of one.

Name the encoding

with open("u.txt", encoding="utf-8") as f:
    pass
print("named")

Passing the encoding makes the file portable. Leaving it out uses a platform default.

A byte count is not a letter count

data = "éé".encode("utf-8")
print(len(data), 2)

Four bytes for two letters. Anything that counts bytes sees double.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Writing a file

Previous Lesson

Writing a file

Next Lesson

Line endings

Line endings →