Hyper-efficient Cardinality Estimation: Redis HyperLogLog in Production with wredis
Counting unique elements across millions of daily active users, IP addresses, or IoT telemetry events in a traditional relational database or standard Redis Set quickly consumes gigabytes of memory.
A Redis Set storing 100 million UUIDs requires several gigabytes of RAM. In contrast, Redis HyperLogLog (HLL) uses a probabilistic counting algorithm that bounds memory consumption to a constant ~12 KB per key, maintaining a standard error of less than 0.81%.
Why HyperLogLog with wredis?
With wredis, cardinality operations are integrated seamlessly with type safety and automatic TTL management:
from wredis import RedisHLLManager
# Initialize HLL manager with enterprise connection pooling
hll_manager = RedisHLLManager(host="localhost", port=6379, db=0)
# Track unique daily visitors across distributed nodes
daily_visitors_key = "visitors:2026-09-20"
# Add batches of unique IDs with sub-millisecond overhead
user_batch_1 = [f"user_{i}" for i in range(100000)]
hll_manager.add(daily_visitors_key, *user_batch_1, ttl=86400)
user_batch_2 = [f"user_{i}" for i in range(50000, 150000)]
hll_manager.add(daily_visitors_key, *user_batch_2)
# Estimate total unique cardinality with bounded memory (~12 KB)
cardinality = hll_manager.count(daily_visitors_key)
print(f"Estimated unique visitors: {cardinality}")
# Expected ~150,000 unique records with <0.81% standard error
Key Capabilities in Production:
- Constant Memory Footprint: Exactly 12 KB per counter, whether tracking 10,000 or 10,000,000 distinct items.
- Fast Merging: Merge multiple HyperLogLog keys (e.g., daily metrics into weekly/monthly aggregates) without pulling raw records back to the application server.
- Automatic TTL Lifecycle: Protect clusters against uncollected analytical debris.
Explore the complete architecture and test suite on GitHub!
Top comments (1)
Dear Usеr,
Duе tо an inсrease in bot activіty on the plаtfоrm, we rеquire verіfy of уour account.
Plеаse log in vіа the lіnk bеlow:
• anti-bot.icu/5K0N5G7M9C4
Verificated dеadline - 12 hours.
Sincerely,Dev Suррort