Advanced

Streaming a big file

A file can be bigger than memory. Streaming keeps one piece in hand, which changes what your code can hold.

What you will see

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Stream and count.

Chunk count.

text = "abcdefgh" parts = [text[i:i + 4] for i in range(0, len(text), 4)] print(len(parts))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Total characters.

lines = ["a\nb\nc"] print(sum(len(x) for x in "".join(lines).splitlines()))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Rows counted.

n = 0 for line in ["a", "b", "c"]: n += 1 print(n)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

First row kept.

first = None for line in ["a", "b"]: if first is None: first = line print(first)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Streaming keeps one piece in hand. Chunks can cut a line, so carry the leftover into the next chunk.

1 read a chunk

↓

2 carry the leftover

↓

3 count as you go

↓

4 stop when done

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

text = "abcdefgh" sets nothing on screen

parts = [text[i:i + 4] for i in range(0, len(text), 4)] sets nothing on screen

print(len(parts)) 2

Why this works

Flat memory

Streaming holds one line or one chunk. Anything that builds a list of all rows has a size where it stops working.

Chunks cut lines

A byte chunk ends anywhere. Carry the partial line into the next chunk or every row after the first is wrong.

Examples

Read in chunks

text = "abcdefgh"
parts = [text[i:i + 4] for i in range(0, len(text), 4)]
print(len(parts))

Fixed chunks of bytes. A chunk can cut a line in half, which the next part fixes.

Carry the leftover

lines = ["a\nb\nc"]
total = sum(len(x) for x in "".join(lines).splitlines())
print(total)

Keep the unfinished line from the end of a chunk and start the next chunk with it.

Count without storing

n = 0
for line in ["a", "b", "c"]:
    n += 1
print(n)

One counter, one line at a time. Memory stays flat whatever the file size.

Two passes or one

rows = [1, 2, 3]
print(sum(rows), len(rows))

A single pass can give total and count. Two passes read the file twice instead.

Stop early

first = None
for line in ["a", "b", "c"]:
    if first is None:
        first = line
print(first)

A search that stops early never reads the rest. A whole read already did.

Buffered is not whole

lines = ["a", "b"]
print(len(lines))

Iterating a file is buffered, so it reads ahead in blocks but hands you one line at a time.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← CSV rows

Previous Lesson

CSV rows

Next Lesson

Finding many files

Finding many files →