Master

The bit layout of a real format

Real formats state their layout in a header: a signature, sizes, then data. Reading that header is how any tool recognises a file, and it is the same idea at every scale.

What you will see

abc

About 2 min of reading

Try It

Parse a tiny binary record.

Show a file signature as hex.

print(b"\x89PNG".hex())

Before you run it, say out loud what it will print. Then open the answer.

What it prints

89504e47

Read a big-endian width.

print(int.from_bytes(b"\x00\x00\x02\xd0", "big"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

720

Read a two-byte length.

print(int.from_bytes(b"\x00\x05", "big"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Read length then payload.

import struct rec = b"\x00\x03abc" n = struct.unpack(">H", rec[:2])[0] print(rec[2:2 + n].decode("ascii"))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

abc

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

A format opens with a signature and sizes, and a reader that follows that layout parses the data after it.

1 signature names the file

↓

2 header stores sizes

↓

3 length marks a record end

↓

4 fields parsed in order

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

print(b"\x89PNG".hex()) 89504e47

Why this works

The header is the contract

A format is a header plus fields, written down once. Every reader implements that contract, and every writer follows it.

Why formats version

A version byte in the header lets a reader support several layouts. Without it, a changed format silently reads wrong.

This is every file

Images, archives, saves, and network packets are all headers plus fields. Learning one layout teaches the pattern for all of them.

Examples

A signature first

print(b"\x89PNG".hex())

A file starts with bytes that name it. That signature is how a tool tells a PNG from any other file without a filename.

Sizes in the header

print(int.from_bytes(b"\x00\x00\x02\xd0", "big"))

A header stores the width as four bytes big-endian. Reading it with the right order gives the number.

A record with a length

print(int.from_bytes(b"\x00\x05", "big"))

A length field tells where a record ends. Formats that skip it cannot be skipped over, which is why it is there.

Parse a tiny file

import struct
rec = b"\x00\x03abc"
n = struct.unpack(">H", rec[:2])[0]
print(rec[2:2 + n].decode("ascii"))

Length then payload, read in order. That is the whole loop of a binary parser.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Checksums and hashes

Previous Lesson

Checksums and hashes