Basic

Testing that links go where they should

A link test is a fetch test in a loop. Collect the links the page offers, request each one, and assert the status. The loop turns a list of hrefs into a proof that the site is connected, which is the bug users hit most often.

What you will see

It prints BROKEN and exits 1. If it exited 0, the check is not checking.

About 1 min of reading

Try It

Crawl one page's links

Extract the hrefs

grep -oE 'href="/[a-z0-9/-]+' /tmp/p.html | sed 's|href="/||' | sort -u > /tmp/links.txt; wc -l < /tmp/links.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

The distinct same-origin paths. External links are excluded by the leading slash.

Request each one

while read -r l; do printf '%s %s ' "$l" "$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:3000$l")"; done < /tmp/links.txt | tee /tmp/status.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

One line per link with its status.

Fail on any non-200

awk '$2 != 200 { print "BROKEN", $1; bad=1 } END { exit bad }' /tmp/status.txt; echo "exit=$?"

Before you run it, say out loud what it will print. Then open the answer.

What it prints

exit=0 when every link is 200. A broken link prints its own path.

Check a known-bad case

printf '/nope-not-a-page 404 ' | awk '$2 != 200 { print "BROKEN", $1; bad=1 } END { exit bad }'; echo "exit=$?"

Before you run it, say out loud what it will print. Then open the answer.

What it prints

It prints BROKEN and exits 1. If it exited 0, the check is not checking.

Do not follow redirects in the loop. A 300-range answer to a stated link should fail, not be papered over.

Data Flow

Page to link verdicts

1 Extract same-origin hrefs

↓

2 Request each href

↓

3 Record the status per href

↓

4 Fail on any non-200

Why this works

Scope of the crawl

Same-origin links only. An external link is someone else's server, and its status belongs in a link-checking task, not a page test.

Where the loop is too slow

A page with hundreds of links makes the loop slow. Run the crawl on the pages users reach, and keep the deep crawl as a separate slower job.

Examples

Collect then request

Extract the hrefs from one page, then request each. The set is the page's promise; the statuses are whether it kept it.

Assert 200, not not-404

A redirect chain or a 300-range answer to a link the page stated directly is a smell. Assert the exact status the page promised.

Fail with the link named

Print the href on the failing line. A crawl that says only FAIL gives a list someone has to reproduce by hand.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Selecting from the served document

Previous Lesson

Selecting from the served document

Next Lesson

Testing a form without a browser

Testing a form without a browser →