DailyRNG
Login
Master

Debugging a container that will not start

A box that fails to start has a cause in one of four places: the recipe, the environment, the data, or the command. Read them in that order, because the earlier ones are cheaper to check and the later ones need the earlier ones to be true.

What you will see

A version number. If it is behind what the release expects, the migration is the fault, not the code.

About 1 min of reading

Try It

Bisect a broken box by layer

1

Read the exit code

docker run --rm app:v1.4.2; echo "exit=$?"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A number. 126 or 127 means the command itself could not be started, which is a recipe fault.
2

Clear the recipe

docker run --rm -it app:v1.4.2 sh -c 'echo ok'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
ok. If this fails, the base image or a layer is broken and the app code is not involved.
3

Clear the environment

docker run --rm app:v1.4.2 sh -c 'env | sort | head -20'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The variables the box actually has. Compare against what the app requires; a missing one is the whole fault.
4

Clear the data

docker run --rm app:v1.4.2 python -c "import sqlite3;print(sqlite3.connect('app.db').execute('SELECT version FROM _meta').fetchone())"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A version number. If it is behind what the release expects, the migration is the fault, not the code.
Do not add a missing environment variable by editing the running box. Put it in the recipe or the compose file, or the fix disappears with the box.

Data Flow

Failure to cause
1 Read the exit code to name the layer
↓
2 Try a shell to clear the recipe
↓
3 Print the environment to clear it
↓
4 Read the data version to clear the data
Why this works
The four-cause order

Recipe, environment, data, command. Each check is one command. Do not start reading application code until the first three are cleared, because application code is the expensive one.

What to fix once found

Fix the artifact, not the box. A running box is disposable; the recipe, the pin, or the migration is the durable thing. A fix that only works in one box has not fixed the release.

Examples

Get a shell, not a stack trace

docker run --rm -it app:v1.4.2 sh starts the box without running the app. If the shell works, the recipe and the base image are fine and the fault is in the command or the data.

Read the exit code first

Exit 126 is exec-failed, 127 is command-not-found, 128 plus N is a signal. Those numbers name the layer that failed before any log line does.

Compare to a box that works

Build the previous release's tag and run the same command. If it works, the diff between the two recipes is the fault. Bisect on the recipe lines, not on the app code.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← A release plan for a containerized product
Previous Lesson
A release plan for a containerized product
Next Lesson
Rolling back a release that is already live
Rolling back a release that is already live →