Basic

Stopwords as a written decision

Some words carry no information. Throwing them away is a decision, and the decision is a file rather than a rule remembered in code.

What you will see

refuse whole-query fallback. No query is no answer, which is the catch.

About 1 min of reading

Try It

Write the list and count the throw

Write the list

printf 'the\na\nand\n' > /tmp/s.txt; wc -l < /tmp/s.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints

3. Three words is the list, and the list is a file.

Count what is kept

printf 'red fish\n' | grep -v -f -x /tmp/s.txt /dev/stdin | wc -l

Before you run it, say out loud what it will print. Then open the answer.

What it prints

2. Two kept words is the count, which is the claim.

Refuse an empty query

printf 'kept 0\n' | awk '{print ($1==0) ? "refuse whole-query fallback" : "answer"}'

Before you run it, say out loud what it will print. Then open the answer.

What it prints

refuse whole-query fallback. No query is no answer, which is the catch.

Data Flow

A document to kept words

1 The list is read at build

↓

2 A kept word becomes a token

↓

3 A listed word is counted as thrown

↓

4 A query of only listed words answers from the whole

Why this works

Why the list is not hardcoded

A list in code is a list that a language change must rebuild. A file is a list that is read at build.

The one word never thrown away

A word that is the only word of a query. Throwing it leaves no query, and no query is no answer.

Examples

The list is a file

The words are written in a file. A file is a list that a reader can change without touching code.

The throw is measured

The words thrown are counted. A count is a claim, and a claim is what a size claim is checked against.

The throw can be reversed

A query word that is on the list is answered from the whole. Answering from the whole is the reversal.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← The inverted index

Previous Lesson

The inverted index

Next Lesson

Answering a query

Answering a query →