Hyper-efficient Cardinality Estimation: Redis HyperLogLog in Production with wredis

Counting unique elements across millions of daily active users, IP addresses, or IoT telemetry events in a traditional relational database or standard Redis Set quickly consumes gigabytes of memory.

A Redis Set storing 100 million UUIDs requires several gigabytes of RAM. In contrast, Redis HyperLogLog (HLL) uses a probabilistic counting algorithm that bounds memory consumption to a constant ~12 KB per key, maintaining a standard error of less than 0.81%.

Why HyperLogLog with wredis?

With wredis, cardinality operations are integrated seamlessly with type safety and automatic TTL management:

from wredis import RedisHLLManager

# Initialize HLL manager with enterprise connection pooling
hll_manager = RedisHLLManager(host="localhost", port=6379, db=0)

# Track unique daily visitors across distributed nodes
daily_visitors_key = "visitors:2026-09-20"

# Add batches of unique IDs with sub-millisecond overhead
user_batch_1 = [f"user_{i}" for i in range(100000)]
hll_manager.add(daily_visitors_key, *user_batch_1, ttl=86400)

user_batch_2 = [f"user_{i}" for i in range(50000, 150000)]
hll_manager.add(daily_visitors_key, *user_batch_2)

# Estimate total unique cardinality with bounded memory (~12 KB)
cardinality = hll_manager.count(daily_visitors_key)
print(f"Estimated unique visitors: {cardinality}")
# Expected ~150,000 unique records with <0.81% standard error

Key Capabilities in Production:

• Constant Memory Footprint: Exactly 12 KB per counter, whether tracking 10,000 or 10,000,000 distinct items.
• Fast Merging: Merge multiple HyperLogLog keys (e.g., daily metrics into weekly/monthly aggregates) without pulling raw records back to the application server.
• Automatic TTL Lifecycle: Protect clusters against uncollected analytical debris.

Explore the complete architecture and test suite on GitHub!