📦 My Redis Payload Was Too Large, So I Just Gzipped It

#node #performance #redis #webdev

A while ago I started getting this error in production: "payload too large". The culprit was a Redis value that kept growing, a big JSON of vehicle data that we cache and read all the time.

My first thought was to bump the limit. My second thought was: why am I storing this much text in the first place? JSON is super repetitive. Same keys, over and over, for thousands of rows. That's basically a gift for compression.

So I tried gzip. It turned out to be one of the simplest fixes I've done, and it gave me more benefits than I expected.

🧩 The Idea

Nothing fancy:

If the JSON is small, store it as usual.
If it's bigger than a threshold, gzip it, encode it as base64, and add a gz: prefix.
When reading, check the prefix. Has gz:? Decompress. No prefix? Parse as usual.

That prefix is the trick that made this safe to ship. A valid JSON string never starts with gz:, so old entries already sitting in Redis still work. No migration, no flushing the cache, no "deploy at 2 AM and pray".

💻 The Code

const zlib = require("zlib");
const { promisify } = require("util");

const gzip = promisify(zlib.gzip);
const gunzip = promisify(zlib.gunzip);

// Big payloads can get rejected by Redis/proxy
// ("payload too large"). Values above the threshold are gzipped + base64
// with a "gz:" prefix. Valid JSON never starts with that, so old
// entries stay readable.
const COMPRESS_MIN_BYTES =
  Number(process.env.REDIS_COMPRESS_MIN_BYTES) || 64 * 1024;
const GZIP_MARK = "gz:";

async function encodePayload(value) {
  const json = JSON.stringify(value);
  if (Buffer.byteLength(json) < COMPRESS_MIN_BYTES) return json;
  const compressed = await gzip(Buffer.from(json));
  return GZIP_MARK + compressed.toString("base64");
}

async function decodePayload(raw) {
  if (!raw) return null;
  try {
    if (typeof raw === "string" && raw.startsWith(GZIP_MARK)) {
      const json = await gunzip(
        Buffer.from(raw.slice(GZIP_MARK.length), "base64")
      );
      return JSON.parse(json);
    }
    return JSON.parse(raw);
  } catch {
    return null; // corrupt/truncated entry
  }
}

module.exports = { encodePayload, decodePayload, COMPRESS_MIN_BYTES, GZIP_MARK };

Usage is just swapping JSON.stringify and JSON.parse around your Redis calls:

await redis.set("vehicles", await encodePayload(vehicles));

const vehicles = await decodePayload(await redis.get("vehicles"));

🔍 A Few Details Worth Explaining

Why a threshold? Compressing a 2 KB payload is a waste of CPU for almost no gain. I only compress above 64 KB, and it's configurable through REDIS_COMPRESS_MIN_BYTES, so I can tune it without redeploying code.

Why async gzip? zlib.gzip through promisify runs on the libuv thread pool, so it doesn't block the event loop. The sync version on a big payload would freeze every other request while it works.

Why return null on error? If an entry is corrupt or truncated, I treat it as a cache miss. The caller falls back to the database and writes a fresh value. For a cache, that's better than crashing the request.

📉 What I Got

The error is gone. Payloads that used to get rejected now go through.
Smaller values in Redis. Less memory used for the same data.
Less data over the network between the app and Redis, which matters a lot if there's a proxy in the middle.
No migration needed, thanks to the gz: prefix.

⚠️ Trade-offs

It's not free, so be honest about what you pay:

CPU time. Compressing and decompressing costs something. For my case it's worth it, but measure yours.
base64 adds roughly 33% overhead. JSON compresses so well that you still come out far ahead, but if you can store raw bytes with your client, you'd skip that step. In my setup I stayed with strings.
You can't read it in redis-cli anymore. A compressed key looks like random text. Debugging means decoding it first.
Hot keys on hot paths. If a key is read thousands of times per second, decompressing every time adds up. Consider an in-memory layer on top.

🎯 Final Thoughts

Before this, I was ready to touch Redis config or split the payload into chunks. In the end about 30 lines of code solved it.

If your Redis values are big JSON blobs, try compressing them before reaching for more complicated fixes. Just measure first, so you know it actually helps.

Have you hit a similar limit with Redis? How did you handle it? Would love to hear your approach!

DEV Community