DEV Community: Premdeep Singh

How Lossless Log Compression Actually Works: Schema-Aware Encoding vs. gzip/zstd

Premdeep Singh — Sat, 05 Sep 2026 07:53:35 +0000

Every backend engineer has had this conversation at least once: retention gets cut from 90 days to 30, or verbosity gets dialed down in production, because the storage bill for logs quietly became a line item someone in finance started asking about. It usually gets framed as a tooling problem — "we need a cheaper log platform" — but it's actually an encoding problem, and most teams never get far enough to notice, because gzip and zstd are "good enough" and switching compressors feels like a solved problem not worth revisiting.

It isn't solved. It's just under-examined. Here's the actual gap, and why domain-specific encoding closes it in a way generic compression fundamentally can't.

The concrete case study: Uber's HDFS bill

In 2022, Uber's engineering team published the numbers behind a problem a lot of platform teams will recognize immediately. Their Spark platform was generating up to 200TB of logs a day across roughly 250,000 jobs, and log retention on HDFS was capped at three days — not because three days was enough, but because it was what the storage budget could absorb. Engineers kept asking for a month of retention. Doing that naively, at existing compression, would have taken the HDFS storage bill for these logs from $180K a year to $1.8M.

Instead of buying more storage or negotiating a better rate, Uber's team adopted CLP — the Compressed Log Processor, originally developed as academic research at the University of Toronto — and restructured how the logs were encoded before they ever hit disk. The result: a 169:1 compression ratio, HDFS costs down to roughly $10K a year, and retention extended to a full month. Not a smaller version of the same tradeoff. An order-of-magnitude-plus improvement, achieved entirely at the encoding layer, with no data thrown away.

That last part matters and gets glossed over constantly: this wasn't sampling, filtering, or dropping DEBUG-level noise. It was lossless — every byte of the original log, recoverable exactly, just represented far more efficiently on disk.

Why gzip and zstd leave this much on the table

gzip and zstd are excellent general-purpose compressors. That's exactly the problem. They're built to compress any byte stream reasonably well — source code, binaries, images, log files, JSON blobs — without knowing anything about the structure of what they're compressing. A generic compressor sees a log line as an undifferentiated string and finds repetition using a sliding window (LZ77-family matching) plus entropy coding on top. That works, and it's why zstd is a sane default almost everywhere. But it's leaving structure-specific redundancy on the table, because it isn't looking for it.

A log line has a shape a generic compressor doesn't get to exploit directly:

2026-08-14T03:12:07.441Z ERROR [payment-service] user_id=48213 order_id=990214 failed to charge card: insufficient_funds

To a byte-stream compressor, this is just bytes. But structurally, it's a small number of variable fields (the timestamp, the user_id, the order_id, the specific error) sitting inside a large amount of constant scaffolding that repeats, nearly verbatim, across millions of other lines from the same log statement. CLP's actual approach — and the approach any schema-aware system worth using takes — is to explicitly separate a log message into:

The static template — everything about the message that comes from the log statement itself, shared across every occurrence of that call site (ERROR [payment-service] user_id=... order_id=... failed to charge card: ...)
The variable values — the specific timestamp, IDs, and dynamic values that differ per occurrence
Dictionaries built per-field-type — because order_id values compress very differently than free-text error messages, and treating them as one undifferentiated blob wastes the specific redundancy each field type has on its own

Once you've split a log stream this way, you're no longer compressing "text." You're compressing a small number of unique templates plus a column of variables per field — and each of those columns is enormously more repetitive, on its own, than the interleaved original ever was. This is essentially the same intuition behind columnar formats like Parquet applied to log semantics instead of tabular data: group like with like, then compress each group with an encoding suited to it, rather than compressing everything with one generic pass.

CLP's own published benchmarks back this up directly: even before the final columnar archiving step, its intermediate representation format outperforms general-purpose compressors like Zstandard, and a second compression pass over that intermediate representation roughly doubles the ratio again — which is exactly the aggregate multi-hundred-x result Uber saw in production.

The part that matters more than the ratio: searchability

Here's where a lot of "just compress it more" thinking falls apart in practice. A compression ratio is worthless operationally if getting your data back means decompressing gigabytes to grep through them. That's the actual reason most teams don't push compression harder already — they've internalized, correctly, that better compression usually means worse query latency, because you're trading disk space for CPU time on every read.

Schema-aware log compression sidesteps this because of what got separated out in the first place. If your variable fields are stored in structured, typed columns rather than buried inside opaque compressed text, you can push a query — "give me every ERROR from payment-service where order_id=990214" — down to the structural level: filter by template category and scan the relevant variable column, without ever fully decompressing the surrounding message text you don't care about. CLP's design explicitly supports search over the compressed representation without full decompression, which is the difference between "compression as an archival tradeoff" and "compression as a strict upgrade."

Where lossless schema-aware compression should actually be applied

This approach isn't equally valuable everywhere, and it's worth being honest about where the win comes from:

High cardinality, high repetition data — application logs, structured audit events, access logs — is the sweet spot. Lots of near-identical templates, lots of low-entropy repeated scaffolding.
Metrics and traces benefit from a related but distinct approach, since they're already more structured; the gains come more from columnar encoding and delta-encoding of sequential values (timestamps, counters) than from template extraction.
Genuinely high-entropy data — already-compressed binaries, encrypted payloads, random IDs with no shared structure — won't benefit much from any of this, because there's no redundancy to expose in the first place. No compression scheme, schema-aware or not, manufactures redundancy that isn't there.
Regulated retention specifically rewards the "lossless" half of this harder than most workloads, because the entire point of a compliance-driven retention requirement — SOX, PCI DSS, HIPAA — is that the original record has to be recoverable, not a statistically-representative approximation of it. A compression scheme that hits a great ratio by discarding rare fields or coarsening timestamps doesn't satisfy that requirement no matter how good the number looks in a benchmark.
The engineering takeaway

If your team is choosing between "ship less data" and "compress harder," it's usually worth checking whether you're actually compressing as hard as the structure of your data allows before deciding you have to throw data away. Uber didn't extend retention 10x by negotiating a storage discount or accepting lossier logging. They changed the unit of compression from "bytes" to "log template + typed variables," and let the compressor do dramatically less redundant work per byte stored.

That's the same principle we've built our own ingestion pipeline around at Sasquatch Labs — schema-aware, lossless compression tuned per telemetry type, verified byte-for-byte against the original on every event, so "more retention" and "smaller bill" stop being a tradeoff you have to negotiate between finance and whoever owns the audit.

References

Uber Engineering, Reducing Logging Cost by Two Orders of Magnitude Using CLP
Uber Engineering, Modernizing Logging at Uber with CLP (Part II)
Y-Scope, CLP: Compressed Log Processor (GitHub)
Luo, J.Y., CLP: Efficient and Scalable Search on Compressed Text Logs, University of Toronto
InfoQ, Uber Reduces Logging Costs by 169x Using Compressed Log Processor (CLP)

I Used to Use Splunk. Here’s Why My Team Switched

Premdeep Singh — Wed, 02 Sep 2026 08:29:22 +0000

(And Why Our Security Budget Loves Us Now)

Let’s start with a scene you might recognize.

My last team managed infrastructure for a mid-sized fintech. We ran Kubernetes in AWS. We used Datadog for observability. And we poured all our security logs into Splunk. Our security team loved us for it. The compliance team loved us for it. And then, one morning, our CFO did not.

He printed out our cloud spend report, walked over to the CISO, and pointed at a single line item.

“Splunk: $650K this year. Growing.”

Then he asked the question nobody in the room had an answer for.

“What exactly are we paying for? And why would we make this trade twice?”

Cut to this year. Different company, different team. Same problem — we generate hundreds of gigabytes of logs a day. Same regulatory requirements — we handle PII, PCI-DSS, CCPA, and SOC2 audits. Same need for real-time threat detection and historical forensics.

Only this time, our Splunk equivalent costs us $28K a year. Not $650K.

The bill shows up on AWS, not on a vendor invoice. We own the encryption keys. We control the data retention policies. We can run a compliance export in minutes, without filing a support ticket.

Here’s what changed.

We Stopped Paying for Data Just to Move It
The first thing I learned in my old job: your Splunk bill mostly pays for data transfer.

Every authentication event, every NGINX log, every Kubernetes pod log, every database query – has to travel from your infrastructure to Splunk’s platform. Splunk charges you to ingest it. The cloud provider charges you to egress it.

When we looked at our AWS bill, we realized we were paying $45K a year in egress fees just to move logs to Splunk. That expense lived on a different spreadsheet. It was invisible to our security team.

We were paying twice: once to move the data, once to store it.

In the new world, we don’t move logs. We compress them and drop them into an S3 bucket in the same region as our workload. Egress cost: $0.

We Started Compressing Differently
Everyone compresses logs. But not like this.

Standard compression (gzip, zstd) treats logs as a blob of text. It’s okay — you get 3× to 5× reduction.

Property-aware compression understands what’s inside the log:

json

{
"timestamp": "2024-09-02T14:30:00Z",
"service": "auth-service",
"level": "INFO",
"message": "User logged in",
"user_id": "u-12345",
"ip": "10.0.0.1"
}
It knows:

timestamp is a predictable sequence
service repeats with low entropy
level repeats even more
user_id and ip fields are structured
Instead of compressing the JSON as text, it compresses each field separately, using the optimal algorithm for that data type.

The result is 50× to 100× compression versus raw storage. Not 3×. Not 5×. 50× or more.

Suddenly, storing a year’s worth of security logs costs pennies instead of six figures.

We Asked a Different Question
When I was costing out Splunk, I asked:

“What’s the cheapest way to get what we need?”

The conversation was always about cutting — turning off log sources, filtering events before they get sent, sampling high-volume streams.

“Do we really need DEBUG logs? Do we need all audit events?”

That question made everyone nervous. Security didn’t want gaps in coverage. Compliance couldn’t risk audit failures. But finance was pushing hard.

This time, I asked:

“What if we could keep 100% of our logs, but store them for 90% less?”

Suddenly, the conversation changed.

The Switch Was Less Scary Than We Thought
So here’s what moving off Splunk actually looked like, timeline included:

Day 1–7: Parallel deployment. We set up a new BYOC SIEM platform in our own AWS account. Redirected logs to both Splunk and the new platform at the same time.

Day 8–14: Validation. Ran identical queries in both systems, compared results row by row. Built dashboards to alert us if logs diverged.

Day 15–28: Feature parity. Rebuilt our critical alerts ('failed login spike', 'unusual data export', 'privilege escalation attempt') in the new system. Recreated compliance dashboards for SOC2 and PCI-DSS.

Day 29–35: Cutover. Shut off new logs to Splunk. Left the old data to expire naturally (90-day retention). Started routing everything to BYOC.

Day 36 onward: We have everything Splunk gave us: log search, alerting, dashboards, correlation, compliance exports – at <10% the cost.

The Math That Convinced Our CFO
Here’s the three-year comparison that ended the finance vs. security tension for us.

Cost Category Splunk + AWS Fees BYOC + AWS Fees
Splunk licensing (100 GB/day) $560K – $750K/year $0
AWS egress fees $45K/year $0
AWS compute/storage Included in Splunk infra $45K/year
Professional services (config/tuning) $30K/year $5K/year (one-time)
Annual Total $635K – $825K $50K – $55K
3-Year Total $1.9M – $2.47M $150K – $165K
Don’t take my word for it. Here’s how to check it yourself:

Grab your Splunk contact’s per-GB/day rate. Multiply by your daily GB ingested.
Look for “AWS Data Transfer” on your cloud bill – filter by Splunk IP ranges.
Add your Splunk professional services spend (config tuning, updates, new data sources).
Multiply by 3 years.
If you’re like 95% of regulated teams, you will find a 6-figure bill – and a conscious choice to either accept it, or start cutting logs.

What About Our Existing Skills?
Our security analysts knew Splunk’s SPL (Search Processing Language). Splunk’s query language looks like this:

index=firewall src_ip=10.0.0.1 | stats count by dest_port

Modern BYOC platforms intentionally use familiar SQL-like syntax:

SELECT dest_port, COUNT(*) FROM firewall WHERE src_ip = '10.0.0.1' GROUP BY dest_port

The training curve was measured in hours, not weeks. Our analysts adjusted.

We also kept our dashboards. Here’s one we rebuilt for AWS CloudTrail monitoring:

sql

SELECT
userIdentity.arn,
COUNT(*) AS event_count
FROM cloudtrail_logs
WHERE
eventTime >= NOW() - INTERVAL '1' HOUR
AND errorCode IS NOT NULL
GROUP BY userIdentity.arn
ORDER BY event_count DESC
Same alerting logic, same visualizations, same RCA workflow – just fewer zeroes on the invoice.

The Compliance Layer Nobody Talks About
Running a regulated workload changes the SIEM conversation.

Our Splunk environment:

Used Splunk Cloud’s FedRAMP-authorized tenant
Charged us 40% more for the compliance tier
Limited our administrator access
Required Splunk employees (with Splunk credentials) to perform maintenance
Our BYOC environment:

Runs in AWS GovCloud (already FedRAMP authorized)
Costs the same as commercial AWS
Gives us full admin control
Keeps all access within our identity provider (Okta)
Allows us to encrypt logs with our own KMS keys, and control who can decrypt
When the auditor asks “who can read these logs?” we point to an IAM role and an Okta group. Not a Splunk support FAQ.

The Uncomfortable Truth Nobody Wants to Say
Spending $650K on Splunk is not an engineering failure. It’s a historical artifact.

Splunk was invented when “big data” meant gigabytes, not terabytes. Its pricing model made perfect sense – charge per GB, because storage and compute were expensive, and indexing needed dedicated infrastructure.

But in 2025:

S3 storage costs $0.023/GB/month
Serverless query engines (Trino, Presto) scan terabytes in seconds
Compression algorithms can achieve 100× ratios on structured logs
Kubernetes makes deploying distributed platforms a Terraform apply away
We’re not paying for Splunk’s technology anymore. We’re paying for Splunk’s 2003 pricing model. We’re paying for Splunk’s 40% FedRAMP premiums. We’re paying for Splunk’s shareholder returns.

That’s a choice. It’s not a law.

Questions Our Team Asked Before Switching
Q: What if we lose logs during migration? A: We ran both platforms side-by-side for four weeks. If logs diverged, alarms fired. They didn’t.

Q: What about historical Splunk data? A: We left it to expire naturally (we had 90-day hot retention). Could have exported it, but the migration cost wasn’t worth it for old logs.

Q: How do we handle incidents without Splunk’s security apps? A: We rebuilt our critical use cases natively. Turns out we didn’t need 90% of the apps – we built the 10% that mattered.

Q: What if this new platform can’t scale? A: It runs in our AWS account. We scale the underlying infrastructure with our workload – horizontal scaling is built into the architecture.

Q: Is this secure enough for our threat model? A: More secure. Logs never leave our cloud boundary. They’re encrypted with our KMS keys. We control all access policies.

The Bottom Line
We use a BYOC SIEM now because:

It costs 90% less
We own the data, the keys, the access controls
We can prove to auditors exactly who can and cannot read logs
We keep 100% of our logs – no sampling, no filtering, no gaps
We don’t use it because Splunk is “bad.” We use it because the economics of SIEM changed, and we changed with them.

If your Splunk bill is starting to look like a mortgage payment, the conversation is worth having. Start with a simple question at your next team sync:

“If we were building our security monitoring from scratch today, with everything we know now about cost, compliance, and threats – what would we build?”

Would you choose Splunk again?

Our team didn’t.

Further Reading
Splunk's Ingestion Pricing Model
Lossless vs. Gzip for Logs
AWS Egress Costs

Let’s Talk
Drop your Splunk story in the comments. How big is your bill this year? Are you cutting logs to save money? I’m reading every one.