How to Let gzip Find the Signal in a Pile of Documents

Jean-Luc Martel — Mon, 03 Aug 2026 23:19:53 +0000

Suppose you have a directory full of text documents.

Most are repetitive, padded with boilerplate, or otherwise low-signal. A few contain the useful material. You could read every file manually, feed them all into an embedding pipeline, or ask an LLM to rank them.

Or you could ask gzip.

The basic idea is simple:

Repetitive text compresses well. Varied text usually does not.

That makes compression ratio a crude but surprisingly useful proxy for redundancy.

It will not tell you which document is best. But it can help you identify which documents contain less repetition and deserve a closer look.

The heuristic

For each document:

Measure its original size.
Compress it individually with gzip.
Measure the compressed size.
Calculate:

compressed size / original size

A lower ratio means the document compressed well, which usually indicates more repetition.

A higher ratio means the document was harder to compress, which may indicate more varied or information-dense content.

In other words:

lower ratio: more redundant
higher ratio: less redundant

The Bash command

Here is a small Bash pipeline that ranks .txt files by compression ratio:

find ./documents -type f -name '*.txt' -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      printf "%.3f\t%8d\t%8d\t%s\n", gz/raw, raw, gz, file
    }
  '
done | sort -nr

Example output:

0.642      18432      11834  ./documents/research-notes.txt
0.417      30211      12600  ./documents/project-summary.txt
0.091      27102       2467  ./documents/standard-contract.txt

The columns are:

ratio    original bytes    compressed bytes    filename

Because the output is sorted in descending order, the least compressible files appear first.

Those are the files I would inspect first when looking for the possible “gems.”

To find the most repetitive documents instead, reverse the sort:

sort -n

Why `gzip -n`?

The -n flag prevents gzip from storing the original filename and timestamp in its output.

That makes the compressed sizes more comparable across files and across runs.

Without it, a small amount of unrelated metadata can leak into the measurement.

What this is actually measuring

This technique does not measure truth, relevance, writing quality, or semantic importance.

It measures compressibility.

Those things sometimes correlate, but they are not the same.

A document full of repeated boilerplate will usually compress extremely well. A document with more distinct vocabulary, sentence structure, numbers, and ideas may compress less efficiently.

That makes the ratio useful as a first-pass ranking signal.

It is closer to a metal detector than a treasure map.

Important caveats

Small files produce noisy ratios

gzip adds headers and other fixed overhead. For tiny files, that overhead can dominate the result.

You may want to ignore documents below a minimum size:

find ./documents -type f -name '*.txt' -size +1k -print0

Already-compressed formats will mislead you

Running this directly against PDF, DOCX, ZIP, JPG, or other compressed formats mostly measures the compression characteristics of the container format.

Extract the text first.

For example, with PDFs:

pdftotext input.pdf output.txt

Incompressible does not mean valuable

Encrypted data, random identifiers, hashes, minified code, and corrupted text are all difficult to compress.

They may score highly while containing little useful information.

Repetition is not always fluff

Contracts, API documentation, technical specifications, and scientific papers may repeat terminology because precision requires it.

A lower ratio can indicate redundancy, but it can also indicate consistency.

Language and formatting matter

Compression ratios can be affected by:

document length
whitespace
markup
tables
repeated headings
source language
character encoding
templated metadata

For a fairer comparison, normalize the documents first.

For example:

tr -s '[:space:]' ' ' < input.txt

You could also strip HTML, remove headers and footers, or convert everything to lowercase before compression.

Just remember that normalization changes what you are measuring.

A slightly more useful version

For larger collections, I would filter out tiny files and print the percentage saved:

find ./documents -type f -name '*.txt' -size +1k -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      ratio = gz / raw
      saved = 100 * (1 - ratio)

      printf "%6.2f%% saved\t%8d bytes\t%s\n",
             saved, raw, file
    }
  '
done | sort -n

This sorts the files with the lowest percentage saved first, meaning the least compressible documents rise to the top.

Where this could be useful

This trick can be handy for quickly triaging:

scraped web pages
exported support tickets
meeting transcripts
research notes
log samples
generated reports
document archives
large sets of Markdown files

It is especially useful when you want a fast local heuristic without setting up a database, embedding model, or external API.

Compression as a feature

The broader idea is more interesting than the Bash command.

Compression ratio can be treated as a lightweight feature in a ranking system.

You could combine it with:

document length
vocabulary diversity
duplicate paragraph counts
keyword density
entropy
embedding similarity
recency
source reputation

Compression alone is crude.

Compression plus a few other signals could become a genuinely useful document-triage tool.

Final thought

There are sophisticated ways to rank a pile of documents.

Sometimes, though, a 40-year-old compression algorithm is enough to tell you which files keep repeating themselves.

And that is often a very good place to start.

DEV Community: Jean-Luc Martel