How redis-packhash Helps You Leverage Listpack Hashes in Redis

In my earlier post, I showed that storing a million small values as Redis keys is a waste of memory. It's because every key has its own overhead. To fix that, we need to group the keys into hashes and keep them small enough that Redis will keep them in the compact listpack encoding. In my benchmark with 1M keys before, a 53.8 MB memory usage went down to 6.9 MB. I call it an 87% discount.

The catch is you have to watch a lot of things. You decide how many hashes to split the keys across, and you need to spread the keys evenly across them. Those are some parts of the bookkeeping that you need to watch in order to get listpack savings.

So I built redis-packhash, a client agnostic, no dependency npm package. You tell it how many keys you expect to store and that's it. It will decide how many hashes to use and spread every key across them evenly, behind a simple get/set.

Install

npm install redis-packhash

No runtime dependencies. It works with ioredis, node-redis, and other Redis clients that expose hset, hget, and hdel.

Quickstart

Bring your Redis client and hand it to PackHash, and everything after that is a simple get/set.

import { PackHash } from "redis-packhash";
import Redis from "ioredis";

// Store each user's locale, one small value per user.
const locales = new PackHash(new Redis(), {
  namespace: "user:locale",
  expectedKeys: 1_000_000,
});

const userId = "9f8c4a2b-1e77-4c3d-bd21-7a0e5f6c8d90";
// Values are strings.
await locales.set(userId, "en-US");

const locale = await locales.get(userId);
//=> "en-US"

That's it. Initialize the store with your expectedKeys, and you are ready to go.
You need another dataset with a different use case? Spin up a new instance. Each one takes an optional namespace that prefixes its Redis top level keys, so they will not collide.

The rest of the API

redis-packhash only accepts string keys. You can pass a UUID, an email, or anything as long as it's string. It runs every key through the FNV-1a hash function and buckets on the result, so the spread stays even whatever your ids look like. Values also need to be string. You pick the serialization, so it never silently reshapes your data.

const logins = new PackHash(redis, {
  namespace: "user:logins",
  expectedKeys: 1_000_000,
});

await logins.set(userId, String(42)); // stringify going in

const raw = await logins.get(userId);
const count = raw ? parseInt(raw, 10) : 0; // and parse coming out => 42

Beyond get and set, you also get has and del, plus mset and mget for batches.

await locales.has("4821"); //=> false

await locales.del("4822"); //=> false

await locales.mset([
  ["4821", "en-US"],
  ["4822", "fr-FR"],
]);

const found = await locales.mget(["4821", "4822", "9999"]);
found.get("4821"); //=> "en-US"
found.get("9999"); //=> null

Sizing is IMPORTANT

The one decision that you make at the very beginning is IMPORTANT. Failed to fill the correct input means you are not save anything. That's the expectedKeys param, which is how many keys you expected to store. Set it too low and buckets overflow and the encoding fallback to a hashtable, which means losing the saving entirely. Set it too high and you get more buckets than you need, which means cost you a little extra overhead. When in doubt, estimate high.

const locales = new PackHash(redis, {
  namespace: "user:locale",
  expectedKeys: 1_000_000,
});

locales.buckets; //=> 2605 total buckets

WARNING: changing expectedKeys on a store that already holds data will silently orphans every existing data. You either can migrate it manually before changing it, or spin a new instance if match your case.

Limits that you need to honor

Listpack has two limits, and it will fallback from listpack to hashtable the moment it crosses either one:

Setting (Redis 7.0+)	Redis default	Description
hash-max-listpack-entries	512	Number of fields in the hash
hash-max-listpack-value	64 bytes	Length of any single field name or value

expectedKeys handles the first limit for you by compute the bucket count needed to achieve below the default 512. But the second limit is on you, at least for now.
The 64 bytes limit applies to the field name and the value separately, not shared between them, and your top level key is not included.

// Case #1: Value over 64 byte
const key = "9f8c4a2b-1e77-4c3d-bd21-7a0e5f6c8d90";
const value = JSON.stringify({
  locale: "en-US",
  timezone: "America/New_York",
  dateFormat: "MM/DD/YYYY",
});

await locales.set(key, value);
// HSET user:locale:2312  "<36-byte key>"  <74-byte value>
// bucket user:locale:2312 falls back to hashtable

// Case #2: Key over 64 byte
const key = "tenant:acme-corp/region:eu-west-1/user:9f8c4a2b-1e77-4c3d-bd21-7a0e5f6c8d90";
const value = "en-US";
await locales.set(key, value );
// HSET user:locale:85  "<75-byte key>"  "<5-byte value>"
// bucket user:locale:85 falls back to hashtable

The entry limit, and your Redis version

Rather than assuming your limits, just ask your own server:

127.0.0.1:6379> CONFIG GET hash-max-listpack-*
1) "hash-max-listpack-entries"
2) "512"
3) "hash-max-listpack-value"
4) "64"

Before Redis 7.0 it was hash-max-ziplist-* and the encoding was ziplist.

If you change your default hash-max-listpack-entries or it has different value in your server for whatever reason, you can set it manually using maxListpackEntries option.

const locales = new PackHash(redis, {
  namespace: "user:locale",
  expectedKeys: 1_000_000,
  maxListpackEntries: 256, // match your config
});

That is the whole thing

For now, that's what this library supports. I've some things in mind to improve it, and if do too, I'm more than happy to hear them.