I Saved 65.9% on the Data I Trained On, 53.5% on Unseen Data

Compressing small records one at a time does not work. A 900-byte JSON object gives the compressor no history to learn from; the file ends before it has even memorised the field names. zstd's trained dictionary feature exists precisely to fill that hole: it extracts common patterns from your corpus and hands the compressor a pre-loaded history that says "you already know these".

Tonight I put that promise to the test against my own blog's feed.json output. The first result looked wonderful: the dictionary shrank per-record output by 65.9%. Then I tried the same dictionary on records it had never seen during training. The gain fell to 53.5%. Those 12.4 points were not the dictionary's doing — they were my measurement error. And that same error sits in most articles written about zstd dictionaries, because everyone measures on the corpus they trained on.

What follows is that whole evening: the commands, the raw output, and a concrete threshold for where to stop growing the dictionary. One machine, one zstd binary, a corpus anyone can download — repeat it yourself if you like.

The corpus: my own blog feed

I did not want to invent synthetic records for this experiment. I wanted real production data that anyone could also fetch. The blog's JSON Feed output fits exactly: 1,346 posts when I pulled it on the evening of 17 October 2026, each one a separate record. The feed is live and growing, so if you repeat this a few weeks from now the record count will be higher and the absolute figures will not match — the ratios will.

curl -s https://mustafaerbay.com.tr/feed.json -o feed.json
python3 - <<'EOF'
import json, os
d = json.load(open('feed.json'))
items = sorted(d['items'], key=lambda x: (x['date_published'], x['id']))
for name, arr in (('train', items[:1000]), ('test', items[1000:])):
    os.makedirs(name, exist_ok=True)
    for i, it in enumerate(arr):
        blob = json.dumps(it, ensure_ascii=False, sort_keys=True).encode()
        open(f'{name}/{i:04d}.json', 'wb').write(blob)
print(len(items))
EOF

I split the data chronologically, not randomly: train holds 1,000 records from 2024-05-07 to 2026-07-15, and test holds the 346 records after 2026-07-16. In production you train a dictionary on the past and compress the future, so the measurement should work the same way.

Raw sizes: the training set is 909,086 bytes (909 bytes per record on average), the test set 314,337 bytes (908 bytes on average). The totals differ, but the record profiles match — 909 against 908 bytes — so a per-record comparison is fair.

One warning: if you check with du -sh you will see 5.3 MB. That number is filesystem block allocation, not data. A thousand 900-byte files each occupy a 4 KiB block on disk. Every figure in this article is a real byte total read with stat -f '%z'.

Step 1: The baseline without a dictionary

First, how far do we get with no dictionary at all? Each record is compressed separately — that is the whole point, individually addressable records.

mkdir -p out-nodict      # --output-dir-flat does NOT create the directory; without this it errors out
zstd -19 -q -f test/*.json --output-dir-flat out-nodict
find out-nodict -name '*.zst' -exec stat -f '%z' {} + | awk '{t+=$1} END{print t}'

141018

314,337 bytes in, 141,018 bytes out. A ratio of 2.23×. Dropping the level from 19 to 3 gave 143,855 bytes — a 2% difference. That tells you a lot: raising the compression level on a 900-byte file buys almost nothing, because the problem is not how hard the algorithm works but that it has no history to work with.

Step 2: Training the dictionary and fooling myself

Training is a single command and finishes in the blink of an eye:

zstd --train train/*.json -o d-default.dict

Trying 5 different sets of parameters
WARNING: The maximum dictionary size 112640 is too large compared to the source size 680764!
size(source)/size(dictionary) = 6.043714, but it should be >= 10! This may lead to a subpar
dictionary! We recommend training on sources at least 10x, and preferably 100x the size of
the dictionary!
k=1024
d=8
f=20
steps=4
split=75
accel=1
Save dictionary of size 112640 into file d-default.dict

Park that warning for now; we come back to it halfway through. zstd told me at training time that my corpus could not carry this dictionary. On first reading I took "subpar" to mean the compression ratio would come out a little lower. Measuring showed it meant something else.

Training on 1,000 records took 0.13 seconds. The resulting dictionary is 112,640 bytes, which is zstd's --maxdict default; the CLI manual says so in as many words: "Limit dictionary to specified size (default: 112640 bytes)." The d=8, f=20, steps=4, split=75 lines are the default trainer's parameters — the manual notes that --train is equivalent to --train-fastcover=d=8,steps=4.

Now for the appetising part. Let us try the dictionary on the records it was trained on:

mkdir -p out-train-dict
zstd -19 -q -f -D d-default.dict train/*.json --output-dir-flat out-train-dict

Without a dictionary the training set compressed to 401,550 bytes; with the dictionary it dropped to 136,965. A 65.9% gain. If I stopped here and wrote "the zstd dictionary cut output to a third", the number would be correct and the article would be wrong.

Step 3: Data it has never seen

The same dictionary, applied to the 346 records it never saw in training:

mkdir -p out-test-dict
zstd -19 -q -f -D d-default.dict test/*.json --output-dir-flat out-test-dict

65602

141,018 → 65,602 bytes. A 53.5% gain, a ratio of 4.79×. Still an excellent result — more than twice as good as the 2.23× without a dictionary. But it sits 12.4 points below the 65.9% the training set reported.

That gap is not a fault; it is the definition. The dictionary has absorbed the frequently occurring fragments of the training data, so when it compresses that same data it is looking at its own copy. On a new record, only the patterns that genuinely generalise help: field names, the date format, the URL prefix, frequently used tags.

But is the gap really memorisation, or were the two sets simply different in difficulty to begin with? The control is already in hand: without a dictionary, the compression ratio is 2.264× on the training set and 2.229× on the test set — 1.6% apart. The two sets are near-identical work for the compressor. The 12.4-point distance that opens up once the dictionary joins therefore comes from the dictionary's memory, not from a difference in the data.

The gap follows a rule: the corpus-to-dictionary ratio

Seeing the gap at a single dictionary size could have been coincidence. So I repeated the measurement at five sizes — every dictionary trained on the same 1,000 records, then applied to both the training and the test set.

Dictionary	Corpus/dictionary	Gain on training set	On unseen data	Gap
1,024 B	888×	41.0%	39.1%	1.9 points
4,096 B	222×	48.7%	44.7%	4.0 points
16,384 B	55×	55.1%	48.7%	6.4 points
65,536 B	14×	62.1%	52.1%	10.1 points
112,640 B	8×	65.9%	53.5%	12.4 points

The gap widens monotonically and predictably as the dictionary grows relative to the corpus. When the dictionary is one-888th of the corpus I fool myself by 1.9 points; at one-eighth, by 12.4.

Now back to that warning in the training output. zstd had printed "size(source)/size(dictionary) = 6.043714, but it should be >= 10" on my screen and recommended 100×. The same ratio sits in the library header; inside lib/zdict.h, the note for ZDICT_trainFromBuffer reads: "It's recommended that total size of all samples be about ~x100 times the target size of dictionary."

Notice what the warning did not say. It did not say compression would get worse — it said "subpar dictionary". The table shows what that word actually buys you. As the ratio drops I do not compress unseen data worse; quite the opposite, the big dictionary is still the best performer there. What degrades is not the compression but the trustworthiness of the measurement. The ratio governs how badly the number you see on the training set overstates reality.

Let us put a figure on it for my corpus. The dictionary was trained on the training set alone: 909,086 bytes. A 100× ratio points at a dictionary of about 9 KB. That lands between the 4 KiB row (gap 4.0 points) and the 16 KiB row (gap 6.4 points) — so obeying the rule caps my self-deception at four to six points, where the default takes it to 12.4.

Read that advice not as a growth recipe but as an honesty margin: when you hit the ratio, the number you see on the training set is close to the truth. The further below it you go, the more you owe your own measurement.

Step 4: Where to stop growing the dictionary

So does a bigger dictionary not compress better after all? On unseen data it does, but the returns shrink fast. Look at the fourth column on its own:

• The first 1 KiB buys 39.1 points over the no-dictionary baseline.
• The remaining 109 KiB adds only 14.3 points more.

In other words, the dictionary's first kilobyte delivers roughly 73% of the total benefit. Choosing 16 KiB over 110 KiB costs just 4.7 points on unseen data.

I wanted to continue with "so that is a factor of seven in memory", because 112,640 divided by 16,384 is seven. Then the numbers came in, and my own thesis caught me. Peak RSS from /usr/bin/time -l while compressing a single record: 2,016 KiB with no dictionary, 1,616 KiB with the 16 KiB dictionary, and 1,616 KiB with the 110 KiB one. No difference between the two dictionary sizes, and both below the no-dictionary run. At level 19 the match-table allocations dwarf the dictionary's raw bytes; at CLI scale, dictionary size simply does not show up in RSS.

The lesson: I had invented the memory justification for shrinking the dictionary. The real justification is not memory but carriage — the dictionary is a file that has to reach every reader, be backed up, and be versioned. Working through the library API the table may look different; measure it in your own stack then.

I also tested the amount of training data as a separate variable, pinning the dictionary size at 16 KiB and varying the record count:

Training records	Output on unseen data
25	78,671 B
100	73,971 B
300	75,437 B
1,000	72,293 B

Two things stand out. First, even the dictionary trained on 25 records captures 91% of the gain that 1,000 records produce — and that dictionary is only 11,081 bytes, because 25 samples cannot fill 16 KiB. Second, the curve is not monotonic: going from 100 records to 300 made the result worse (73,971 → 75,437). I will not present that as a law — it is one measurement on one corpus — but it does show there is no guarantee behind the intuition that "more samples always mean a better dictionary". When you grow the training set, measure the gain instead of assuming it.

Step 5: The alternative nobody measures — batching

This is the point to step back. A dictionary solves the "no history" problem. But that same problem has a far cheaper solution: compress records in batches rather than one at a time. Then the history comes from the earlier records in the batch instead of from a dictionary.

Same 346 records, same level, the only variable being batch size:

Batch	Without dictionary	With dictionary
1 record	141,018 B	65,602 B
10 records	71,962 B	52,754 B
50 records	56,988 B	48,886 B
100 records	53,560 B	47,742 B
Single stream	49,339 B	46,882 B

Read that table once more. Batches of 50 with no dictionary (56,988 B) beat per-record compression with a 110 KiB trained dictionary (65,602 B). No dictionary maintenance, no versioning, no distribution — just grouping the records.

This does not make dictionaries worthless; it clarifies what they are for. A dictionary is the answer when you cannot batch. If you need per-record random access — reading a row from a KV store, pulling a single message off a queue — there is no such thing as a batch of 50, because reading one record would mean decompressing fifty. That is exactly when a dictionary turns 2.23× into 4.79× while keeping every record independently decodable.

The two do combine, and the dictionary keeps paying even in batched layouts: on the single stream it takes 49,339 down to 46,882, another 5%. But note what that 5% costs you — dictionary distribution, versioning and DictID tracking. Adding a dictionary when you could batch instead trades a marginal gain for operational weight; when per-record access is mandatory, the dictionary is the lever that turns 2.23× into 4.79×.

Step 6: What happens if you lose the dictionary

Now the part that will hurt you in production. Data compressed with a dictionary cannot be decompressed without it. This is not a performance question, it is an availability question.

zstd -l -v single-record.zst

DictID: 1312853601
Window Size: 939 B (939 B)
Compressed Size: 148 B (148 B)
Decompressed Size: 939 B (939 B)
Check: XXH64 55ceeb5a

The frame header carries a DictID. RFC 8878 defines the field as "the ID of the dictionary required to properly decode the frame", and the zstd manual says a 4-byte random number ID is created by default. Try to decompress without it, or with the wrong dictionary:

single-record.zst : Decoding error (36) : Dictionary mismatch

Both attempts exit with status 1 and write no output file. That is good news: zstd errors out rather than silently producing wrong data.

Using a raw file as a dictionary

-D does not insist on a trained dictionary. Per RFC 8878, zstd also accepts "raw content" dictionaries — free of any format restriction beyond a minimum of 8 bytes. So you can concatenate a few sample records and hand that over as a dictionary. I tried it with five separate consecutive bundles of five records each from the training set (0–4, 100–104, 200–204, 300–304, 400–404; all between 4.2 and 4.8 KB). Their outputs on the test set spread from 92,080 to 95,614 bytes. The trained 4 KiB dictionary gives 77,948 — 15% smaller than even the best raw bundle. I am not quoting a single ratio from one bundle, because which records you happen to pick moves the result by 3,500 bytes; what training really adds is that it removes that lottery.

The raw dictionary's second difference matters more:

DictID: 0

No identity. When I tried to decompress with the wrong raw dictionary I still got an error — but a different one:

Decoding error (36) : Restored data doesn't match checksum

What caught it was not the DictID but the content checksum. RFC 8878 defines it as an XXH64 digest of the decoded data, and the zstd manual notes the check is enabled by default. So what happens if you turn it off?

zstd -19 --no-check -D rawA.dict record.json -o nc.zst
zstd -d -D rawB.dict nc.zst -o nc.out    # wrong dictionary
echo $?

0

Exit status zero. No error. And nc.out was written — 939 bytes, the same size as the original. Here is what is inside it:

{ified": "2026-04-06T00:00:00.000Z", "date_published": "2026-04-06T00:00:00.0
gelistiriciler-icin-dijital-guvitle"-5-kolayle"-maliyet-dengesie_modified": ...

It is not valid JSON, but it looks like JSON: real dates, real slug fragments, familiar field names. In a logging pipeline or a bulk restore, this has a very good chance of slipping through unnoticed.

You will not get this outcome with every wrong dictionary: the offsets the decoder reads out of the dictionary have to land in bounds, and with a sufficiently smaller wrong dictionary you get "Corrupted block" instead. My two raw dictionaries were close in size (4,201 and 4,427 bytes), which opened the silent-corruption path. Three conditions must hold at once: a raw dictionary (DictID 0), plus --no-check, plus the wrong dictionary. Each one can be chosen for a perfectly reasonable-sounding motive — "we know which dictionary it is", "the checksum burns CPU", "we refreshed the dictionary file during deploy". With a trained dictionary the DictID stops you at the first hurdle; with a raw dictionary your only defence is the checksum, so leave it on.

Do dictionaries go stale?

One question remains: does a dictionary trained on old data decay over time? On the first attempt I trained two dictionaries — the oldest 300 records and the newest 300 — both 16 KiB, both applied to the same 47 records from October 2026 (42,162 bytes). The stale one gave 11,485 bytes and the fresh one 10,842, and I was about to write "a two-and-a-half-year-old dictionary still delivers 93% of the fresh one's benefit".

Something nagged before I wrote it: how much does which 300 I picked move the answer? So I split the pool in half and trained four dictionaries, taking 300 records from the start and the end of each half.

Dictionary	Training range	Output
none	—	19,619 B
start of the old half	2024-05 → 2026-04	11,485 B
end of the old half	2026-04 → 2026-05	11,366 B
start of the new half	2026-05 → 2026-07	11,423 B
end of the new half	2026-07 → 2026-09	10,842 B

The first three rows sit within 119 bytes of one another. Two years separate them and the difference is almost nothing. Only the last row breaks away: the three months immediately before the target do 524 bytes better.

So "does the dictionary go stale" was the wrong question. Choosing between 2024 data and mid-2026 data moves the target by 62 bytes — noise. The only measurable effect is adjacency: the period right next to the target does slightly better than any distant one. My first attempt's "93%" was arithmetically right but overstated the effect, because I had picked the two extremes.

For my data this makes sense: what the dictionary captures is largely schema — field names, date format, URL prefix. As long as the schema holds, calendar age carries no weight. If your schema or vocabulary genuinely shifts, the result will differ; measure rather than assume — and when you measure, do not repeat my first-round mistake, try several slices instead of one.

The practical consequence: retraining the dictionary on a cron job creates more problems than it solves, because every new dictionary means a new DictID, and you must keep the old ones around to read the old data.

What about speed?

I repeated each measurement 10 times and took the minimum and median (Apple M4 Pro, macOS 26.6.2, zstd 1.5.7; 346 records in a single zstd invocation):

Measurement	Minimum	Median
Compress -19, no dictionary	111 ms	114 ms
Compress -19, with dictionary	127 ms	138 ms
Compress -3, with dictionary	65 ms	71 ms
Decompress, no dictionary	57 ms	67 ms
Decompress, with dictionary	58 ms	68 ms

The dictionary slows compression down by 14% at the minimum. On decompression the 1 ms difference between the two rows is smaller than the run-to-run variation within a single measurement — so the dictionary's decompression cost is unmeasurable on this data.

Do not carry that row straight into production. The measurement decompresses 346 records in a single zstd invocation: the dictionary is read and digested once, then amortised across 346 records. The scenario this article recommends dictionaries for is the opposite — per-record random access, one record per read. On the library side, a naive implementation that builds a fresh ZSTD_DDict from 112,640 bytes on every read does far more work than decoding the 939-byte record itself. The right shape is to build it once with ZSTD_createDDict and reuse it through ZSTD_DCtx_refDDict; the dictionary object is read-only and shareable across threads. My CLI figure is what that correct setup looks like, and it says nothing about code that rebuilds the dictionary on every read.

Picking the level is the more effective lever: -3 with the dictionary gives 73,202 bytes in 65 ms, while -19 gives 65,602 bytes in 127 ms. Twelve per cent more output for half the time. On a write-heavy path, -3 is almost always the right call.

A decision framework

As a checklist:

1. Try batching first. Batches of 50 without a dictionary beat a 110 KiB dictionary. If you do not need random access, the job ends here.
2. Size the dictionary against the corpus. Around 1% of the training corpus is a good starting point for --maxdict; that is zstd's own advice, and it prints a warning when you ignore it.
3. Measure on unseen data. The number you get on the training set can mislead your capacity plan by up to 12 points.
4. Start small. Even a dictionary trained on 25 records delivers 91% of the benefit; do not assume the returns of a large training set, measure them.
5. Store the dictionary as durably as the data. Losing the dictionary means losing the data. Record the DictID; include the dictionary in your backup plan.
6. Stay away from --no-check, especially with a raw dictionary. The CPU you save is not worth silently corrupted data.
7. Justify retraining. With a stable schema, calendar age cost me almost nothing; every new dictionary is a new DictID you have to carry.

Three topics this article does not cover will meet you on the way to production; at least know their names. First, dictionary distribution and version coexistence: DictID catches the wrong dictionary but it does not deliver the right one — where you put it (sidecar object, id-to-blob table, baked into the binary) and how long you retain old versions is your design. Second, the library API: this article measures the CLI, while production uses ZSTD_createCDict/createDDict, and the cost model changes there. Third, and least discussed: a shared dictionary is a side channel. In a multi-tenant store where attacker-influenced data and secrets share one dictionary, the compression ratio leaks information — the same logic as CRIME and BREACH. Use a dictionary per tenant, or keep secrets out of the dictionary path.

If you are thinking about storage cost in layers, the piece where I worked through the same logic on the logging side — Tiered Log Retention with Grafana Loki — is a good companion: there too, the gain came not from a single setting but from what question you were asking of the data.

Closing thought

The real lesson from this experiment is not about compression. The zstd dictionary genuinely does a good job — going from 2.23× to 4.79× is nothing to sniff at, and it does so while preserving per-record random access.

The lesson is this: if you measure the benefit of an optimisation on the very data you derived it from, what you are measuring is not benefit but memorisation. On my corpus the difference was 12.4 points, and the gap closed steadily as I shrank the dictionary relative to the corpus. zstd had even printed it on my screen at training time; I read "subpar dictionary" as a remark about the compression ratio and moved on. What the warning was really describing was how misleading my measurement would be.

Build a capacity plan on the wrong side of that gap and you will spend six months from now wondering why the disk is 30% fuller than you expected. Separating the two costs nothing: split the data in half, train on one half, measure on the other.

Official Sources

• zstd CLI manual (zstd.1.md) — --train, the --maxdict default (112640 bytes), -D, --[no-]check and --dictID behaviour.
• lib/zdict.h — the zstd dictionary builder header — sample count guidance for ZDICT_trainFromBuffer and the "~x100 times the target size of dictionary" recommendation.
• RFC 8878 — Zstandard Compression and the application/zstd Media Type — the Dictionary_ID frame header field, the "raw content" dictionary definition, and the XXH64-based Content_Checksum.

Comments

[Akhouri Anmol Kumar]

bro my mind when I try to read your post