Every backend engineer has had this conversation at least once: retention gets cut from 90 days to 30, or verbosity gets dialed down in production, because the storage bill for logs quietly became a line item someone in finance started asking about. It usually gets framed as a tooling problem — "we need a cheaper log platform" — but it's actually an encoding problem, and most teams never get far enough to notice, because gzip and zstd are "good enough" and switching compressors feels like a solved problem not worth revisiting.
It isn't solved. It's just under-examined. Here's the actual gap, and why domain-specific encoding closes it in a way generic compression fundamentally can't.
The concrete case study: Uber's HDFS bill
In 2022, Uber's engineering team published the numbers behind a problem a lot of platform teams will recognize immediately. Their Spark platform was generating up to 200TB of logs a day across roughly 250,000 jobs, and log retention on HDFS was capped at three days — not because three days was enough, but because it was what the storage budget could absorb. Engineers kept asking for a month of retention. Doing that naively, at existing compression, would have taken the HDFS storage bill for these logs from $180K a year to $1.8M.
Instead of buying more storage or negotiating a better rate, Uber's team adopted CLP — the Compressed Log Processor, originally developed as academic research at the University of Toronto — and restructured how the logs were encoded before they ever hit disk. The result: a 169:1 compression ratio, HDFS costs down to roughly $10K a year, and retention extended to a full month. Not a smaller version of the same tradeoff. An order-of-magnitude-plus improvement, achieved entirely at the encoding layer, with no data thrown away.
That last part matters and gets glossed over constantly: this wasn't sampling, filtering, or dropping DEBUG-level noise. It was lossless — every byte of the original log, recoverable exactly, just represented far more efficiently on disk.
Why gzip and zstd leave this much on the table
gzip and zstd are excellent general-purpose compressors. That's exactly the problem. They're built to compress any byte stream reasonably well — source code, binaries, images, log files, JSON blobs — without knowing anything about the structure of what they're compressing. A generic compressor sees a log line as an undifferentiated string and finds repetition using a sliding window (LZ77-family matching) plus entropy coding on top. That works, and it's why zstd is a sane default almost everywhere. But it's leaving structure-specific redundancy on the table, because it isn't looking for it.
A log line has a shape a generic compressor doesn't get to exploit directly:
2026-08-14T03:12:07.441Z ERROR [payment-service] user_id=48213 order_id=990214 failed to charge card: insufficient_funds
To a byte-stream compressor, this is just bytes. But structurally, it's a small number of variable fields (the timestamp, the user_id, the order_id, the specific error) sitting inside a large amount of constant scaffolding that repeats, nearly verbatim, across millions of other lines from the same log statement. CLP's actual approach — and the approach any schema-aware system worth using takes — is to explicitly separate a log message into:
The static template — everything about the message that comes from the log statement itself, shared across every occurrence of that call site (ERROR [payment-service] user_id=... order_id=... failed to charge card: ...)
The variable values — the specific timestamp, IDs, and dynamic values that differ per occurrence
Dictionaries built per-field-type — because order_id values compress very differently than free-text error messages, and treating them as one undifferentiated blob wastes the specific redundancy each field type has on its own
Once you've split a log stream this way, you're no longer compressing "text." You're compressing a small number of unique templates plus a column of variables per field — and each of those columns is enormously more repetitive, on its own, than the interleaved original ever was. This is essentially the same intuition behind columnar formats like Parquet applied to log semantics instead of tabular data: group like with like, then compress each group with an encoding suited to it, rather than compressing everything with one generic pass.
CLP's own published benchmarks back this up directly: even before the final columnar archiving step, its intermediate representation format outperforms general-purpose compressors like Zstandard, and a second compression pass over that intermediate representation roughly doubles the ratio again — which is exactly the aggregate multi-hundred-x result Uber saw in production.
The part that matters more than the ratio: searchability
Here's where a lot of "just compress it more" thinking falls apart in practice. A compression ratio is worthless operationally if getting your data back means decompressing gigabytes to grep through them. That's the actual reason most teams don't push compression harder already — they've internalized, correctly, that better compression usually means worse query latency, because you're trading disk space for CPU time on every read.
Schema-aware log compression sidesteps this because of what got separated out in the first place. If your variable fields are stored in structured, typed columns rather than buried inside opaque compressed text, you can push a query — "give me every ERROR from payment-service where order_id=990214" — down to the structural level: filter by template category and scan the relevant variable column, without ever fully decompressing the surrounding message text you don't care about. CLP's design explicitly supports search over the compressed representation without full decompression, which is the difference between "compression as an archival tradeoff" and "compression as a strict upgrade."
Where lossless schema-aware compression should actually be applied
This approach isn't equally valuable everywhere, and it's worth being honest about where the win comes from:
High cardinality, high repetition data — application logs, structured audit events, access logs — is the sweet spot. Lots of near-identical templates, lots of low-entropy repeated scaffolding.
Metrics and traces benefit from a related but distinct approach, since they're already more structured; the gains come more from columnar encoding and delta-encoding of sequential values (timestamps, counters) than from template extraction.
Genuinely high-entropy data — already-compressed binaries, encrypted payloads, random IDs with no shared structure — won't benefit much from any of this, because there's no redundancy to expose in the first place. No compression scheme, schema-aware or not, manufactures redundancy that isn't there.
Regulated retention specifically rewards the "lossless" half of this harder than most workloads, because the entire point of a compliance-driven retention requirement — SOX, PCI DSS, HIPAA — is that the original record has to be recoverable, not a statistically-representative approximation of it. A compression scheme that hits a great ratio by discarding rare fields or coarsening timestamps doesn't satisfy that requirement no matter how good the number looks in a benchmark.
The engineering takeaway
If your team is choosing between "ship less data" and "compress harder," it's usually worth checking whether you're actually compressing as hard as the structure of your data allows before deciding you have to throw data away. Uber didn't extend retention 10x by negotiating a storage discount or accepting lossier logging. They changed the unit of compression from "bytes" to "log template + typed variables," and let the compressor do dramatically less redundant work per byte stored.
That's the same principle we've built our own ingestion pipeline around at Sasquatch Labs — schema-aware, lossless compression tuned per telemetry type, verified byte-for-byte against the original on every event, so "more retention" and "smaller bill" stop being a tradeoff you have to negotiate between finance and whoever owns the audit.
References
Uber Engineering, Reducing Logging Cost by Two Orders of Magnitude Using CLP
Uber Engineering, Modernizing Logging at Uber with CLP (Part II)
Y-Scope, CLP: Compressed Log Processor (GitHub)
Luo, J.Y., CLP: Efficient and Scalable Search on Compressed Text Logs, University of Toronto
InfoQ, Uber Reduces Logging Costs by 169x Using Compressed Log Processor (CLP)
Top comments (0)