DailyRNG
Login
Advanced

Word weight from the library

A common word carries no information. The information is measured across the library, and a measurement across the library is a number written at build time.

What you will see

Two lines. The weight lives with the key, which costs nothing per answer.

About 1 min of reading

Try It

Count documents per word

1

Write three documents

printf '0 the cat\n1 the dog\n2 a cat\n' > /tmp/d.txt; awk '{for(i=2;i<=NF;i++)print $i}' /tmp/d.txt | sort | uniq -c

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Counts: a 1, cat 2, dog 1, the 2. Documents holding the word, which is the claim.
2

Give the common word no weight

printf 'docs 3 word_docs 3\n' | awk '{print ($2==$3) ? "weight 0" : "weight positive"}'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
weight 0. Zero is the answer for a word in every document.
3

Store the weight with the key

printf 'cat weight 0.41\nthe weight 0.00\n' > /tmp/w.txt; cat /tmp/w.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Two lines. The weight lives with the key, which costs nothing per answer.

Data Flow

A library to a weight per key
1 The documents holding each word are counted
↓
2 The count becomes a weight
↓
3 The weight is stored with the key
↓
4 A word in every document gets zero
Why this works
Why the weight is stored with the key

The key is looked up once. A weight stored with the key is a weight that costs nothing at answer time.

The one claim that changes with the library

Every weight. A weight is a claim about the whole library, so a library change is a weight change.

Examples

The count is documents, not tokens

How many documents have the word. Documents is the claim, and tokens is a different claim.

The weight is written at build

Written means the answer reads a number. A number read is cheaper than a count made per answer.

A word in every document has no weight

Zero weight is an answer. The answer is honest because the count was made at build.

← Ranking as a score with a reason
Previous Lesson
Ranking as a score with a reason
Next Lesson
Phrase queries from positions
Phrase queries from positions →