Don't Brotli Your JSON into Postgres (Unless It's an Archive)

TL;DR One question decides it: is this data part of runtime analysis, or is it an archive? Runtime data belongs in jsonb. Archives can be compressed blobs. Get the split wrong and you lose every native query, pay a decompression tax on every read, and (with Brotli q11) watch your write throughput drop 25x.

Our pipeline enriches social posts and stores a per-post snapshot in a jsonb column. Tens of thousands of snapshots a week, most a few KB, the worst comment threads pushing 100KB. One day the storage graph looked scary enough that someone proposed the obvious fix: compress each snapshot with Brotli in the app, store bytea, cut the bill.

It sounds like a free win. Brotli is what makes the web fast, JSON is famously repetitive, and Postgres will happily store bytes. We almost did it.

We didn't, and the deciding question is one you can apply to any table: is this data part of runtime analysis, or is it an archive?

If the answer is runtime, meaning dashboards, filters, joins, ad-hoc debugging queries, anything that reads fields out of the document while a user or a job waits, then compressed blobs are the wrong storage format, full stop. If the answer is archive, written once and read whole-or-never, compression is a perfectly good idea. Most of this article is about what happens when you get that split wrong, with numbers.

🚫 Cost #1: native queries stop working

This is the cost that matters most and shows up last, usually during an incident.

With jsonb, the one-off questions that make up a debugging session are just SQL. "Which snapshots from yesterday have zero comments?" is one query:

SELECT id
FROM snapshots
WHERE doc->>'platform' = 'reddit'
  AND (doc->'metrics'->>'comments')::int = 0
  AND created_at >= now() - interval '1 day';

You can put a GIN index on the column and get containment queries, build an expression index on the one hot field, or promote it to a generated column. You can run an ad-hoc psql query at 2am, point a BI tool at the table, backfill with a single UPDATE.

Once the column is a compressed bytea blob, the database cannot see inside it. Every one of those operations becomes "write a script, pull every row over the network, decompress, parse, filter in app code". Your database has been demoted to a key-value store, except one you're paying Postgres prices for. That one-line query above? Now it's a batch job.

That was the argument that killed the proposal at work: the snapshots get queried by fields we didn't anticipate when we designed the table. Every debugging session since has validated the decision.

🐌 Cost #2: every runtime read pays a decompression tax

Even for the access pattern that still works, fetch-by-id, the blob makes your code slower and heavier. The application can't use compressed bytes, so every read on the runtime path looks like this:

row = db.fetch_one("SELECT doc FROM snapshots WHERE id = %s", post_id)
snapshot = json.loads(brotli.decompress(row["doc"]))
platform = snapshot["platform"]

Ship the whole blob over the wire, decompress it, parse the JSON, then take the one field you wanted. There is no partial read. Need one attribute out of a 200KB document? You decompress 200KB.

Postgres with jsonb does the opposite: it navigates the document server-side and sends you the 30 bytes you asked for.

SELECT doc->>'platform' FROM snapshots WHERE id = 42;

📦 "But compression saves space." Yes. Postgres already does it.

Two facts people mix up here.

First, jsonb itself is not a compressed format. It's a parsed binary representation, built for fast field access, and it's routinely as large as the raw JSON text or larger.

Second, Postgres compresses large values anyway, through a mechanism called TOAST. Any variable-length value over roughly 2KB gets compressed and, if still large, sliced into chunks in a side table. This is automatic, transparent, and it's why you've never had to think about it. The default codec is pglz; since Postgres 14 you can switch a column to the faster lz4:

ALTER TABLE snapshots ALTER COLUMN doc SET COMPRESSION lz4;

TOAST's codecs are weak by modern standards, and Brotli will beat them on ratio. So the honest question is not "does app-side compression save disk?" It does. The question is how much, and what the runtime paths pay for it. I benchmarked it.

📊 The benchmark

Postgres 16 in Docker on an Apple M4 Pro, single connection over localhost, Python client. Synthetic social-post snapshots (nested author object, metrics, a growing comments array) at three sizes: 2KB, 20KB, 200KB. Five ways to store them:

• jsonb with default pglz TOAST
• jsonb with lz4 TOAST
• bytea + app-side Brotli, quality 5
• bytea + app-side Brotli, quality 11 (the "maximum compression" everyone reaches for)
• bytea + app-side zstd, level 3

The bytea columns are set to STORAGE EXTERNAL so Postgres doesn't waste time re-compressing already-compressed bytes. Insert throughput includes the app-side compression cost, because it's the same CPU budget. Reads are median latency over 200 random primary-key lookups, and for bytea they include decompressing and parsing in the app, because you can't use the JSON without doing that.

2KB docs, 3,000 rows:

Variant	Disk/row	Inserts/s	Read one field	Read whole doc
jsonb + pglz	1.4 KB	2,663	0.34 ms	0.36 ms
jsonb + lz4	1.5 KB	2,656	0.33 ms	0.36 ms
bytea + brotli q5	0.8 KB	2,206	0.37 ms	0.37 ms
bytea + brotli q11	0.8 KB	341	0.39 ms	0.33 ms
bytea + zstd 3	0.9 KB	2,759	0.42 ms	0.36 ms

20KB docs, 1,000 rows:

Variant	Disk/row	Inserts/s	Read one field	Read whole doc
jsonb + pglz	8.3 KB	1,369	0.35 ms	0.58 ms
jsonb + lz4	8.3 KB	1,632	0.34 ms	0.57 ms
bytea + brotli q5	5.6 KB	1,067	0.66 ms	0.80 ms
bytea + brotli q11	4.2 KB	65	0.43 ms	0.42 ms
bytea + zstd 3	5.6 KB	2,308	0.65 ms	0.72 ms

200KB docs, 150 rows:

Variant	Disk/row	Inserts/s	Read one field	Read whole doc
jsonb + pglz	57.3 KB	230	0.58 ms	3.01 ms
jsonb + lz4	69.4 KB	342	0.46 ms	2.86 ms
bytea + brotli q5	41.2 KB	238	2.02 ms	1.54 ms
bytea + brotli q11	33.1 KB	6	2.31 ms	2.89 ms
bytea + zstd 3	41.9 KB	433	2.89 ms	3.34 ms

Localhost round trips dominate the small-doc read numbers, so treat sub-millisecond differences there as noise. Three things in these tables are not noise.

💾 Pre-compression wins on disk, as promised. Brotli q11 stores the 20KB docs in half the space TOAST manages. Concede this honestly: it's the whole reason the idea is tempting.

🔥 Brotli q11 detonates your write path. 341 inserts/s at 2KB. 65 at 20KB. Six per second at 200KB. Not sixty, six. Brotli at high quality is a compress-once, serve-a-million-times codec, built for CDN assets with a dictionary tuned for HTML and JavaScript. On a pipeline hot path it's a self-inflicted outage: one worker that used to push 1,600 snapshots a second through jsonb+lz4 now does 65. You'd need 25 workers to stand still, and you paid extra for the privilege.

🐌 The decompression tax is visible even in the query that favors blobs. Reading one field from a 200KB document: 2.3 ms through the decompress-and-parse path against 0.5 ms for jsonb, plus roughly a thousand times more bytes on the wire. Multiply by every dashboard query that touches one attribute. Meanwhile, whole-doc reads on compressed blobs are genuinely fine (brotli q5 was the fastest in that column), which is exactly why compression suits archives: archives are read whole or not at all.

One more thing the table says quietly: zstd level 3 matches Brotli q5's ratio while inserting faster than jsonb does. If your use case truly calls for pre-compression, Brotli was never the right codec. It's a web-delivery format that wandered into a database.

⚖️ Where compressed JSON belongs

The split from the top of the article, applied:

✅ Archive: compress away. Rows written once, read whole-or-never, kept for compliance or replay. No field access, no ad-hoc queries, ratio is the only metric that matters, and the benchmark shows compressed blobs read back whole just fine. Use zstd, not Brotli q11, if the archive is written continuously.

✅ Storage-bound with known, fixed access patterns. Billions of rows where the 40% delta over TOAST is real money, and you are certain nothing will ever filter on a field. Measure on your actual data first: SELECT pg_total_relation_size(...), not hopes.

✅ Truly opaque payloads. Encrypted envelopes, rendered artifacts, raw scraped HTML. The database was never going to look inside.

But look at that last case again. If Postgres never looks inside the value, why is it in Postgres? A compressed object in S3 with a pointer row (id, URL, a few extracted columns for the fields you filter on) is cheaper per GB by an order of magnitude, keeps your tables small and your vacuums fast, and scales without touching the database. The strongest argument for compressing JSON into bytea is usually an argument for not storing it in the database at all.

✅ The checklist

Before you compress JSON into a Postgres column:

1. Will this data feed runtime analysis, now or plausibly later? Filters, dashboards, debugging, joins on a field. If yes, use jsonb. Stop here. This is most cases.
2. On Postgres 14+, did you try ALTER COLUMN ... SET COMPRESSION lz4 first? One line, keeps every query working.
3. Is the write path hot? Then high-effort codecs are out regardless. Brotli q11 on 20KB docs is 65 rows/s per core.
4. Genuinely an archive and still compressing? Use zstd. Brotli is for CDNs.
5. Is the blob large and truly opaque? Consider S3 plus a pointer row instead of a blob column.

Postgres spent thirty years making sure you don't have to think about how big your values are. TOAST compresses them, indexes reach inside them, and the planner sends you only what you ask for. Compressing runtime data in the app throws all of that away for a better ratio on a metric that was never your bottleneck. Compress the archive. Query the rest.

---

I'm a full-stack product engineer taking on freelance and consulting work: payment system design, analytics engines, high-performance SaaS applications, and data pipelines like the one in this post. See what I ship at github.com/ashwaniarya, or reach me at [e-mail].