DailyRNG
Login
Basic

Loading a dataset

Every analysis starts the same way: get the data into memory as rows you can loop over. CSV and JSON cover most of what you will meet. The mistake is analysing the text instead of the values. A CSV read gives you strings; a number you cannot add is not a number.

What you will see

22, an int.

About 2 min of reading

Try It

Read a small CSV and print one row as a dictionary.

1

Create a tiny CSV file

name,count apple,3 pear,12 fig,7

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Three lines plus a header.
2

Read it with DictReader

import csv with open('fruit.csv') as f: rows = list(csv.DictReader(f)) print(rows[1])

Before you run it, say out loud what it will print. Then open the answer.

What it prints
{'name': 'pear', 'count': '12'} - note the quotes: still a string.
If you see '12' with quotes, that is the trap this lesson is about.
3

Convert and add

total = sum(int(r['count']) for r in rows) print(total)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
22, an int.

Data Flow

File bytes become records you can loop over.
1 Open the file and read its bytes
↓
2 Parse the format into rows
↓
3 Convert each string to its real type
↓
4 Hold the rows in a list of records
Why this works
Read the header once

Column names are data, not fixed code, so write the analysis against the names you read rather than against positions. If a column is renamed, a position-based script silently computes the wrong thing; a name-based one fails loudly, which is what you want.

When you have a choice

Pick the shape first

Decide what one row means before you write any analysis. A row should be one observation of one thing. If your file has dates across the top or several tables glued together, fix the shape first; every later step is easier on tidy data.

Examples

CSV into rows

In Python, csv.DictReader turns a file into one dictionary per row, keyed by the header. In JavaScript, you parse the text yourself or use a library. Either way, the shape you want is a list of records.

Strings are not numbers

csv.DictReader gives strings for every column. int(row['count']) or float(...) converts. If you skip the conversion, sorted() on a column of numbers sorts '10' before '9' and your averages come out as concatenations.

JSON is already typed

json.load() or JSON.parse() keeps numbers as numbers, which is why JSON round-trips are so convenient. The trade-off is that JSON needs the whole document in memory and has no streaming for huge files.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← The first look at a new dataset
Previous Lesson
The first look at a new dataset
Next Lesson
Selecting and filtering rows
Selecting and filtering rows →