DailyRNG
Login
Advanced

Pages, not the whole list

I asked for everything and the server refused. Big lists come back in pages, and you follow a pointer to the next one. The API is a chain of responses, not one giant one.

What you will see

{"data": [], "next": "/users?cursor=end"}

About 2 min of reading

Try It

Walk a paginated endpoint and stop correctly.

1

Request the first page. It carries a next pointer, so there is more.

curl -s 'https://api.example/users?limit=2'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
{"data": [1, 2], "next": "/users?cursor=abc"}
2

Follow the pointer. This page has no next, so it is the last.

curl -s 'https://api.example/users?cursor=abc'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
{"data": [3]}
3

A trailing empty page must end the loop, not spin. Stop on empty items.

curl -s 'https://api.example/users?cursor=end'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
{"data": [], "next": "/users?cursor=end"}
This page has a next but no items. If you loop only on the presence of next, you spin forever on this empty page. End the loop when a page has no items.

Data Flow

A big list arrives as pages; each response carries a pointer to the next, and you loop until a page is empty, holding only one page at a time.
1 Request the first page
↓
2 Response carries a next pointer
↓
3 Loop while pages have items
↓
4 Hold one page at a time
Why this works
Cursor beats offset

Some APIs page by offset (?skip=100), which breaks when items are added in the middle. A cursor (an opaque token pointing at the last item seen) does not shift. Prefer the cursor when the list changes while you read it.

Stop on empty, not on next

A page can have a next that returns empty, or return items with no next. End the loop when a page has no items, not only when next is missing, so a trailing empty page does not spin forever.

Examples

A page and a pointer

{"data": [1, 2, 3], "next": "/users?cursor=abc"}

The response carries a next field. If it is present, there is more. You request that URL and get the following page.

Walk the pages

url = '/users'
while url:
    page = get(url)
    yield from page['data']
    url = page.get('next')

The loop is the whole idea: keep going while there is a next. You never hold the full list, only one page.

Page size is a knob

/users?limit=100

A bigger page means fewer round trips but a bigger body. Pick the size by what you do with the data, not by what the server allows.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Proving who you are
Previous Lesson
Proving who you are
Next Lesson
When the server says slow down
When the server says slow down →