Redis 8.12-rc1's roaring bitmap type shrinks a sparse bitmap from 24MB to 239KB

Redis 8.12-rc1 folds a side project called redis-roaring into core. It gives bitmaps a second, opt-in internal representation based on Roaring Bitmaps, and a new command, BITRANGEINTARRAY, that reads the set-bit positions straight out of it. The release notes describe this as lowering memory use and speeding up bitmap operations. I wanted to know what that actually means for a bitmap that looks like something I'd really store, not a synthetic all-ones or all-zeros string.

There's no official Docker image for 8.12-rc1 yet, so I built it from the tagged source inside a throwaway container (the commands are in "Run it yourself" below). redis-cli INFO server reports this build as redis_version:8.11.240, not 8.12 anything — the pre-release versioning is a full point behind the tag name, which would trip up any script checking the version string before relying on this feature.

The headline number first, then the three things that make it less simple than it sounds.

The sparse case

I set 20,000 bits at pseudo-random offsets across a 200,000,000-bit range — a rough stand-in for "which of 200 million user IDs did something today, and about 20,000 of them did." I built the same bitmap twice, once with the new bitmap-default-roaring config off (the default) and once with it on, using the same seeded random sequence both times so the two bitmaps mark identical bits.

	legacy bitmap	roaring bitmap
MEMORY USAGE	25,165,860 bytes	244,837 bytes
ratio	—	103x smaller
BITCOUNT throughput (3 runs)	1,040–1,056 req/s	14,706–15,152 req/s
OBJECT ENCODING	raw	bitmap-roaring

The memory drop is the one I expected from "roaring bitmap," since that's exactly the data structure's reason to exist: it stores runs and small arrays of set positions instead of one bit per possible value. What I didn't expect going in was that BITCOUNT — a full scan of the bitmap under the old encoding — got roughly 14 times faster too, because counting set bits in a roaring structure means walking a much smaller set of containers instead of every byte.

BITCOUNT numbers came from redis-benchmark -n 5000 -c 1 BITCOUNT <key>, three runs each, reported as the full range I saw across those runs.

The dense case, which argues the other way

Roaring bitmaps compress well when bits are either clustered into runs or sparse enough to list individually. They do not compress a bitmap where roughly half the bits are set with no pattern, because there's nothing to run-length or array-encode. I built a 1,000,000-bit bitmap with every even bit set — deliberately dense, deliberately patterned so run-length encoding gets a fair shot — and it still came out bigger under roaring:

	legacy bitmap	roaring bitmap
MEMORY USAGE	131,103 bytes	132,376 bytes
ratio	—	~1% larger

A 1% regression isn't going to hurt anyone, but it inverts the headline cleanly: the "smaller and faster" release-notes framing holds for the sparse, real-world-shaped case and not for a bitmap that's already dense. BITCOUNT on this dense pair came out close to identical too (23,622–25,000 req/s legacy against 26,087–27,778 req/s roaring) — not nothing, but nowhere near the 14x on the sparse bitmap, because at 125KB the legacy scan was already fast enough that there wasn't much left to win.

It's not a string encoding, it's a different type

This is the part I'd have gotten wrong by assumption. A roaring bitmap is not a new OBJECT ENCODING of the string type the way intset or listpack are encodings of other types. TYPE on a roaring key returns bitmap, not string, and Redis enforces that split:

$ redis-cli GETRANGE dense_roaring 0 10
WRONGTYPE Operation against a key holding the wrong kind of value
$ redis-cli APPEND dense_roaring "x"
WRONGTYPE Operation against a key holding the wrong kind of value
$ redis-cli SETRANGE dense_roaring 0 "x"
WRONGTYPE Operation against a key holding the wrong kind of value

Any command that treats the value as a byte string — GETRANGE, APPEND, SETRANGE, STRLEN — refuses outright. The bit-oriented commands don't care:

$ redis-cli GETBIT dense_roaring 0
(integer) 1
$ redis-cli BITCOUNT dense_roaring
(integer) 500000
$ redis-cli BITFIELD dense_roaring GET u8 0
(integer) 170
$ redis-cli BITOP AND destkey dense_roaring dense_legacy
(integer) 125000

That last one mixes a roaring key and a legacy string-backed bitmap in the same BITOP and it just works. So the practical rule is: anything in the bitmap command family is fine across both representations, anything in the string command family now refuses on a roaring key where it used to succeed. If you have code that does GETRANGE on a key you also SETBIT into — which is a real pattern for packing flags into a string you also read back in chunks — turning on bitmap-default-roaring will break it, silently, the first time Redis picks the new encoding for a fresh key.

The config only affects new keys

bitmap-default-roaring reads like a global switch, and in one sense it is: flip it and every subsequent SETBIT on a key that doesn't exist yet creates a roaring bitmap instead of a string. What it doesn't do is touch anything that already exists. I set a bit on a fresh key with the config on, got bitmap-roaring back from OBJECT ENCODING, then turned the config off and set another bit on that same key:

$ redis-cli CONFIG SET bitmap-default-roaring no
OK
$ redis-cli SETBIT dense_roaring 1000001 1
(integer) 0
$ redis-cli OBJECT ENCODING dense_roaring
"bitmap-roaring"

Still roaring. The config is read once, at key creation, and never again for that key. There's no in-place conversion command either way, as far as I could find — if you want to move an existing bitmap between representations you'd have to read it out and rebuild it under the other setting.

BITRANGEINTARRAY works on anything, and never errors

I'd assumed BITRANGEINTARRAY — which returns the offsets of set bits in a range, instead of making the caller walk BITPOS one position at a time — was exclusive to the new type, since the release notes introduce it in the same breath as roaring bitmaps. It isn't. It runs on a plain legacy bitmap, and on a key that was never meant to be a bitmap at all:

$ redis-cli SET not_a_bitmap "hello"
OK
$ redis-cli BITRANGEINTARRAY not_a_bitmap 0 10
1
2
4
9
10
13
15
...

Those are just the set bits in the ASCII encoding of "hello," which is consistent with how every other Redis bit command already treats a plain string. I also fed it a battery of bad ranges — a nonexistent key, end before start, a negative start, a negative end, an end several orders of magnitude past the bitmap's length — and none of it errored:

$ redis-cli BITRANGEINTARRAY nonexistent_key 0 10
(empty array)
$ redis-cli BITRANGEINTARRAY big_sparse_roaring 10 0
(empty array)
$ redis-cli BITRANGEINTARRAY big_sparse_roaring 0 -1
(19998 results)
$ redis-cli BITRANGEINTARRAY big_sparse_roaring 0 1000000000000
(19998 results)

A negative end behaves like it does in GETRANGE — "to the end of the bitmap" — rather than erroring, and an absurdly large end just gets clamped. That's a reasonable design, but it means a bug in your own range arithmetic (an off-by-one that sends a negative number where you meant a small positive one) will not announce itself. It will just quietly return more or fewer bits than you meant to ask for.

What a single bit costs, with and without roaring

The 14x BITCOUNT win is real, but it's a win for a command that touches the whole bitmap. I wanted to know whether a single SETBIT on an existing key pays any kind of tax for being stored as a roaring container instead of a flat byte string. Repeated SETBIT on the same existing bit, one connection:

	legacy	roaring
throughput (3 runs)	15,789–27,778 req/s	27,273–28,571 req/s

And the same thing with 50 concurrent clients hammering the same key:

	legacy	roaring
throughput (3 runs)	112,360–131,579 req/s	117,647–149,254 req/s

Both ranges overlap enough that I don't think there's a real difference here, in either direction, at one connection or fifty. BITPOS searching from offset 0 came out the same way — 23,622–27,273 req/s legacy against 25,000–29,126 req/s roaring. The speedup from roaring bitmaps is specific to operations that have to look at a large span of the bitmap at once. A point write or an early-exit search doesn't have much span to save on, so it doesn't get much faster, which is a narrower claim than "speeds up bitmap operations" taken at face value.

Watching it in production

INFO everything carries a new keysizes section that separates bitmap-shaped keys from plain strings, bucketed by size:

# Keysizes
db0_distrib_strings_sizes:64K=1,1M=1,16M=1
db0_distrib_bitmaps_sizes:64K=2,1M=1,16M=4

That's useful on its own, independent of roaring: it's a live count of how many of your keys are bitmaps and how big they are, without having to sample keys with MEMORY USAGE yourself. It doesn't currently split roaring bitmaps out from legacy ones in this view, so it tells you "how many big bitmaps do I have" but not "how many of them are roaring" — you'd still need OBJECT ENCODING per key for that.

What I got wrong on the way

My first attempt at timing BITCOUNT used a Lua script that called redis.call('TIME') before and after a loop of 2,000 BITCOUNT calls, on the theory that running the loop server-side would avoid the process-spawn overhead of shelling out to redis-cli thousands of times. The two TIME calls returned exactly the same microsecond value, every run, regardless of how much work happened between them.

Redis scripts run with a frozen clock on purpose — a script has to produce the same result if it's replayed on a replica, so time-dependent commands inside EVAL return a value fixed at script start, not the real clock. I'd read about this behaviour before and genuinely forgot it applied here, so I got a result that looked like "the loop took zero time" and sat with it for a few minutes before realising the instrumentation, not the server, was what had frozen. redis-benchmark, which measures real round trips over the wire rather than asking the script to time itself, doesn't have this problem, which is what the numbers above actually come from.

Run it yourself

This needs Docker and a clone of Redis's own tree, since there's no published image for a release candidate yet:

git clone --depth 1 --branch 8.12-rc1 https://github.com/redis/redis.git redis-src
cat > Dockerfile.redis812 <<'EOF'
FROM debian:bookworm-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential tcl libssl-dev pkg-config && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY redis-src/ .
RUN make -j4 BUILD_TLS=no

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends libssl3 \
    && rm -rf /var/lib/apt/lists/*
COPY --from=build /src/src/redis-server /usr/local/bin/redis-server
COPY --from=build /src/src/redis-cli /usr/local/bin/redis-cli
COPY --from=build /src/src/redis-benchmark /usr/local/bin/redis-benchmark
CMD ["redis-server", "--protected-mode", "no"]
EOF
docker build -f Dockerfile.redis812 -t redis-812rc1 .
docker run -d --name r812 -p 16379:6379 redis-812rc1

Then the sparse-versus-dense comparison itself:

# sparse: 20,000 bits in a 200,000,000-bit range
docker exec r812 redis-cli CONFIG SET bitmap-default-roaring no
docker exec r812 redis-cli EVAL "
  math.randomseed(42)
  for i=1,20000 do redis.call('SETBIT', KEYS[1], math.random(0,199999999), 1) end
  return redis.call('BITCOUNT', KEYS[1])" 1 sparse_legacy
docker exec r812 redis-cli MEMORY USAGE sparse_legacy

docker exec r812 redis-cli CONFIG SET bitmap-default-roaring yes
docker exec r812 redis-cli EVAL "
  math.randomseed(42)
  for i=1,20000 do redis.call('SETBIT', KEYS[1], math.random(0,199999999), 1) end
  return redis.call('BITCOUNT', KEYS[1])" 1 sparse_roaring
docker exec r812 redis-cli MEMORY USAGE sparse_roaring

# throughput comparison
docker exec r812 redis-benchmark -q -n 5000 -c 1 BITCOUNT sparse_legacy
docker exec r812 redis-benchmark -q -n 5000 -c 1 BITCOUNT sparse_roaring

On my machine that prints 25,170,660-ish bytes for the legacy key and 244,837 for the roaring one, and the same gap in BITCOUNT throughput shown above.

What to do with this

If you've got a sparse bitmap today — activity flags over a large ID space, a bloom-filter-adjacent membership check, anything where most bits are zero — bitmap-default-roaring is worth testing against your own data once 8.12 reaches a stable release, with the understanding that it's a type change, not a free encoding swap. Audit for GETRANGE, APPEND, SETRANGE or STRLEN on any key you also SETBIT into, because those will start failing the moment that key happens to land on the new encoding. And if your bitmaps are already dense, don't turn it on by default and assume it helps; measure your own shape, because mine got slightly worse.