Advanced

Speeding up pipelines

Slow pipelines waste time. The biggest delay is usually installing dependencies. Learn how caching makes pipelines fast.

What you will see

Cache hit -> pip skipped (3s) After change: key=pip-62d36509 -> miss -> pip install (2m0s)

About 1 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Installing dependencies is the slowest part of a pipeline, so it is cached by hashing the lock file. If the hash is unchanged the install is skipped in seconds; only when the file changes does the slow step run again.

I hash the requirements file to build a cache key and simulate a first run that misses the cache and does the full slow install.

import hashlib def hashfile(name, contents): return hashlib.sha256(contents.encode()).hexdigest()[:8] requirements = "flask==2.0\nrequests==2.28\n" key = "pip-" + hashfile("requirements.txt", requirements) cache = {} print("Speeding up pipelines") print("Cache key: %s" % key) if key not in cache: print("Cache miss -> pip install (2m0s)") install_time = 120 cache[key] = "built"

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Speeding up pipelines Cache key: pip-01e2b615 Cache miss -> pip install (2m0s)

The key must include a hash of the exact dependency file. Change the requirements and the key changes, which is what forces a rebuild and keeps the cache honest.

I simulate a second run on the same key that hits the cache and skips the install, then a run after a change that produces a new key and misses.

requirements = "flask==2.0\nrequests==2.28\n" def hashfile(name, contents): return hashlib.sha256(contents.encode()).hexdigest()[:8] key = "pip-" + hashfile("requirements.txt", requirements) print("Cache hit -> pip skipped (3s)") new_requirements = requirements + "pytest==7.0\n" new_key = "pip-" + hashfile("requirements.txt", new_requirements) print("After change: key=%s -> miss -> pip install (2m0s)" % new_key)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Cache hit -> pip skipped (3s) After change: key=pip-62d36509 -> miss -> pip install (2m0s)

An exact key reuse skips everything, but a changed lock file creates a new key and rebuilds. A cache that never refreshes ships stale packages.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Check the cache first, reuse it if fresh, otherwise build and store it.

1 Compute cache key

↓

2 Cache hit? Reuse

↓

3 Cache miss: build

↓

4 Store for next time

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

import hashlib sets nothing on screen

def hashfile(name, contents): sets nothing on screen

return hashlib.sha256(contents.encode()).hexdigest()[:8] sets nothing on screen

requirements = "flask==2.0\nrequests==2.28\n" sets nothing on screen

key = "pip-" + hashfile("requirements.txt", requirements) sets nothing on screen

cache = {} sets nothing on screen

print("Speeding up pipelines") Speeding up pipelines

print("Cache key: %s" % key) Cache key: pip-01e2b615

if key not in cache: sets nothing on screen

print("Cache miss -> pip install (2m0s)") Cache miss -> pip install (2m0s)

install_time = 120 sets nothing on screen

cache[key] = "built" sets nothing on screen

Why this works

What to cache

Cache dependencies, build output, and test data. Anything that takes time to create but rarely changes is a good cache.

Caching keys

A cache key decides when to reuse. Hash the requirements file so the cache updates only when dependencies change.

Examples

Caching dependencies

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('requirements.txt') }}

This caches installed packages. If your requirements file does not change, the cache is reused. The install step jumps from minutes to seconds.

Cache miss vs cache hit

First run:   key=abc123 -> miss -> pip install (2m)
Second run:  key=abc123 -> hit -> pip skipped (3s)
After change: key=def456 -> miss -> pip install (2m)

The first run builds the cache. Later runs reuse it in seconds. Only when the key changes does the slow install happen again.

Restoring and saving

- uses: actions/cache@v4
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

restore-keys gives a fallback. If an exact match is missing, GitHub returns the newest similar cache. The install becomes a quick update, not a fresh download.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← GitHub Actions basics

Previous Lesson

GitHub Actions basics

Next Lesson

Parallel and matrix builds

Parallel and matrix builds →