Sedgewick & Wayne · §4.2 Sorting · Frequency counts
Zipf's Law: The Most Common Words
Sort the words of a book, count each distinct word, and sort the counts. The k-th most common word then occurs about 1/k times as often as the most common one: rank × count stays roughly constant. Try it on four novels or on your own text.
1 · From words to counts
The frequency count in two sorts and one pass. Step through it on the opening of A Tale of Two Cities: sort the words so that equal words are neighbors, count each run of equal words, then sort the counts, largest first.
2 · Rank and count
The most common words of a whole novel. Zipf is the count that 1/k predicts from the top word: count of rank 1, divided by k. Watch the last column, k × count.
Text
Show
3 · The log-log plot
Every distinct word as one point: rank on the x-axis, count on the y-axis, both on a logarithmic scale. A power law count ∝ 1/ks is a straight line with slope −s. Move the slider to fit the dashed line; hover or touch the plot to read a word.
Exponent s of the dashed line
1.00
the words of the textcount = C / ks
Hover over the plot.
4 · Predict a count
You know only the count of the most common word. Predict how often the word at a given rank occurs, then check it against 1/k and against the book.
Hint: count of rank k ≈ count of rank 1 / k.
5 · Your own text
Paste a text or open a .txt file; it stays in your browser. Words are lower-cased runs of letters (an apostrophe inside a word is kept). Then pick Your text in cards 2 to 4. A few hundred words show the start of the pattern; a whole book shows it clearly.
Texts: Project Gutenberg editions as distributed with the textbook's data files, license header and footer removed. Words are counted case-insensitively; "Mr." counts as "mr".