DEV Community: Priyadarshan Singh

How Redis Cut Memory Usage for Large Bitmaps with Native Roaring Compression

Priyadarshan Singh — Wed, 14 Oct 2026 15:59:28 +0000

Why Bitmap Compression Matters in Redis

Bitmaps are a fundamental data structure in Redis, used for everything from user tracking to analytics. Yet, before this change, Redis stored bitmaps as plain strings. This meant memory usage scaled with the highest bit set, not the number of bits actually set — a costly inefficiency for large or sparse bitmaps.

If you’ve ever worked with large bitmaps in Redis, you know how quickly memory can balloon. This article dives into how Redis introduced native compressed bitmaps using the Roaring bitmap format to solve this problem, cutting memory usage dramatically without sacrificing performance or compatibility.

The Problem: Dense Bitmap Storage Wastes Memory

Redis represented bitmaps as strings where each byte corresponds to 8 bits, stretching out to the highest set bit. Imagine a bitmap with just a few bits set near the end of a large range — Redis still allocated memory for all bits up to that point.

This dense representation is simple but inefficient for sparse or very large bitmaps. It results in wasted memory and can degrade performance due to increased memory pressure.

The Root Cause: Lack of Compression or Sparse Representation

The core issue was that Redis’s bitmap type had no concept of compression or sparse storage. It was a flat, byte-array representation:

Memory allocated proportional to the highest bit offset.
No differentiation between set and unset bits beyond the raw string.
No ability to store bitmaps compactly when bits are few and far apart.

This meant that even a bitmap with a few scattered bits set could consume megabytes of memory if the highest set bit was large.

The Solution: Integrating Roaring Bitmaps into Redis

The PR introduces a new native bitmap type in Redis backed by the CRoaring library, a high-performance C implementation of Roaring bitmaps.

What Are Roaring Bitmaps?

Roaring bitmaps are a compressed bitmap format designed for fast operations and efficient memory usage, especially with sparse or clustered bitmaps. They achieve compression by splitting the bitmap into chunks and encoding each chunk using the most efficient method (arrays, bitmaps, or runs).

How Redis Uses Roaring Bitmaps

New Object Type: A new Redis object type OBJ_BITMAP with encoding OBJ_ENCODING_BITMAP_ROARING is introduced.
Vendored CRoaring: The PR vendors the CRoaring library to handle bitmap operations internally.
Config Option: A new configuration bitmap-default-roaring allows automatic creation and conversion to Roaring bitmaps on writes.
Command Compatibility: Bit commands transparently work on both the old string bitmaps and new Roaring bitmaps, preserving existing semantics.
Persistence: A new RDB type is added, leveraging CRoaring’s compact serialization format for efficient persistence and loading.

Technical Integration Details

The integration required careful handling to maintain compatibility:

Type System: Redis’s type system was extended to distinguish between string and Roaring bitmap encodings.
Command Layer: Bit commands like SETBIT, GETBIT, and bitwise operations were adapted to dispatch to the appropriate underlying representation.
Conversion Logic: When bitmap-default-roaring is enabled, writes create Roaring bitmaps by default. Existing string bitmaps can be converted lazily or eagerly.
Persistence: The RDB format was extended to store Roaring bitmaps efficiently, avoiding the overhead of raw strings.

The Takeaway: Compression Requires Thoughtful Integration

This change shows the power of choosing the right data structure for the job. Roaring bitmaps dramatically reduce memory usage for sparse or large bitmaps while preserving exact semantics.

However, integrating compressed data structures into an existing system is non-trivial. It requires:

Extending type systems to support multiple encodings.
Ensuring command compatibility and seamless user experience.
Supporting persistence formats that respect compression.

For engineers building high-performance systems, this PR is a great example of balancing new data structures’ benefits with the complexity of integration.

This article is based on the Redis PR "Redis native compressed bitmaps (redis-roaring)". Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Accelerating Regex Filtering in Milvus with FMIndex Candidate Pruning

Priyadarshan Singh — Tue, 13 Oct 2026 16:25:46 +0000

Accelerating Regex Filtering in Milvus with FMIndex Candidate Pruning

Regex filtering is a powerful tool for querying complex string patterns, but it can become a performance bottleneck when it requires scanning entire datasets. In Milvus, an open-source vector database, regex filtering on VARCHAR columns used to scan full row bytes even when the regex pattern contained selective literals that could help narrow down candidates. This article dives into a recent enhancement that leverages FMIndex candidate filtering to accelerate regex queries by extracting and utilizing multiple selective literals and their boolean relationships.

The Problem: Inefficient Regex Filtering

Before this change, Milvus’s regex filtering mechanism was suboptimal. It extracted only the longest literal substring from a regex pattern to generate candidate rows for filtering. While this might seem reasonable, it missed opportunities to use other selective literals embedded in the regex, especially when multiple literals were combined with AND/OR operators.

For example, consider a regex pattern like (foo|bar).*baz. The previous approach would extract just the longest literal (e.g., "baz") and ignore "foo" and "bar". This led to scanning all rows containing "baz," even though the presence of "foo" or "bar" could have further pruned candidates. Additionally, the regex filtering path did not integrate with the FMIndex candidate-and-recheck mechanism designed to efficiently filter rows based on indexed substrings.

The consequence was that many regex queries triggered costly full scans over the data, reducing query performance and throughput.

Understanding the Root Cause

The core issue was the simplistic extraction of only the longest literal substring from regex patterns. This approach failed to capture multiple selective literals and their boolean logic (AND/OR combinations) that could be leveraged for more effective candidate pruning.

Moreover, the regex filtering path was disconnected from the FMIndex candidate filtering mechanism, which is capable of generating bitmap candidates based on indexed substrings. Without integrating these two, the system missed the chance to use FMIndex to prefilter rows before applying the expensive regex matching.

The Solution: Integrating Regex Filtering with FMIndex Candidate Filtering

This PR introduces a sophisticated method to route regex filtering on sealed VARCHAR columns through the existing FMIndex candidate-and-recheck path. The key enhancements include:

Extraction of All Bounded Literals: Instead of just the longest literal, the implementation extracts all bounded literals from the regex pattern while preserving their AND/OR relationships. This means that if the regex contains multiple literals combined with boolean operators, all are considered for candidate generation.
Counting Distinct Literals per Segment: The system counts distinct literals in each data segment to assess their selectivity and cost.
Selecting Cheapest Literal Candidates per AND Group: For each AND group in the boolean expression, it chooses the literal candidate with the lowest cost (i.e., the one expected to prune the most rows efficiently).
Preserving OR Branches: All OR branches are preserved to form combined candidate bitmaps, ensuring that rows matching any of the alternatives are considered.
Canonical RE2 Regex Matcher for Final Recheck: The RE2 regex matcher is compiled once and stored for reuse as a final recheck filter on candidate rows, ensuring correctness without redundant compilation overhead.
FilteredRE2 Adapter: A new adapter manages pinned RE2 prefilters with conservative structural checks to safely map literals extracted from regex patterns to FMIndex candidates.
Thread-Local Cache: To optimize performance, immutable preparations such as compiled regex and literal extraction are cached thread-locally.
Preserving Input Bitmaps and NULL Validity: The filtering preserves the original input bitmaps and NULL values, maintaining correctness in the presence of nullable data.
Graceful Fallback: When no usable literal candidates are found or the candidate generation budget is exceeded, the system falls back to scanning, ensuring robustness.

How This Works in Practice

When a regex filter is applied on a sealed VARCHAR column, the system:

Parses the regex to identify all bounded literals and their boolean relationships.
Uses the FMIndex to generate candidate bitmaps for these literals.
Combines these bitmaps according to the AND/OR structure to form a precise candidate set.
Applies the compiled RE2 matcher only to these candidates, avoiding full scans.

This approach effectively prunes the search space, reducing the number of rows that require expensive regex matching.

Takeaway: The Power of Preserving Boolean Literal Relationships

This enhancement highlights an important lesson in optimizing regex filtering with index structures: extracting and preserving multiple selective literals and their boolean relationships is crucial. Relying on a single longest literal can miss significant pruning opportunities.

By integrating regex filtering with FMIndex candidate filtering and carefully managing the boolean logic of literals, Milvus achieves more precise candidate pruning and better query performance. This pattern can be applied to other systems where regex filtering and indexed substring search coexist.

This article is based on the pull request "feat: accelerate regex filtering with FMIndex candidates" merged into the Milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Backporting Row-Level Security to Milvus 2.6: A Deep Dive into Enabling Fine-Grained Access Control

Priyadarshan Singh — Mon, 12 Oct 2026 13:57:38 +0000

Why Row-Level Security Matters in Milvus 2.6

Milvus, a popular vector database, thrives in scenarios requiring fast similarity search and analytics over large-scale vector data. With the rise of multi-tenant deployments and cloud-native architectures, enforcing fine-grained access control becomes critical. Row-level security (RLS) allows administrators to restrict data access at the individual row level, ensuring users only see data they are authorized to access.

Before this backport, Milvus 2.6 lacked RLS support, limiting its usability in secure, multi-tenant environments. This article explores the technical challenge of backporting the RLS feature from a newer Milvus version into 2.6, highlighting the intricacies of adapting modern security logic to an older codebase.

The Problem: Missing Row-Level Security in Milvus 2.6

Milvus 2.6 users faced a significant limitation: no ability to enforce row-level access policies during queries, data manipulation language (DML) operations, or bulk imports. This gap meant that all users with access to a collection could potentially retrieve or modify any data in it, violating the principle of least privilege.

The original RLS implementation introduced in later Milvus versions depended on a set of APIs, data structures, and synchronization mechanisms absent in 2.6. For example:

Advanced metadata management: Newer versions use enhanced metadata stores to keep track of policies and tags.
Proxy synchronization: Proxies maintain consistent snapshots of policies and tags to enforce security uniformly.
Policy evaluation logic: Complex evaluation engines determine whether a given user query or DML operation complies with RLS policies.
Protobuf and import parsing: Updated serialization formats and import mechanisms support richer policy data.

These dependencies made a straightforward backport impossible, as 2.6's codebase lacked the necessary infrastructure.

The Technical Solution: Backporting RLS with Compatibility

The core challenge was to isolate the RLS feature set from its newer dependencies and adapt it to fit within the constraints of Milvus 2.6. The backport involved several key technical steps:

1. Metadata Handling Adaptation

RLS relies heavily on metadata to store policies and tags associated with rows. The backport reimplemented metadata handling using 2.6's existing storage and retrieval mechanisms, ensuring that policy data could be stored, updated, and queried without relying on newer metadata APIs.

2. Proxy Synchronization Without New Membership Filters

In newer versions, proxies synchronize policy and tag snapshots using advanced membership filters to maintain consistency. Since these filters were unavailable in 2.6, the backport introduced a synchronization mechanism compatible with the older proxy architecture, ensuring that all proxies had coherent views of the current policies and tags.

3. Policy Evaluation and Enforcement

The evaluation engine was adapted to work with the older protobuf definitions and API constraints. This involved:

Decoding policy tags correctly with 2.6's protobuf support.
Validating policies in batches to minimize performance overhead.
Recovering policy states gracefully after restarts or failures.

The enforcement logic was integrated into query processing, DML operations, and bulk imports, ensuring that row-level policies were consistently applied across all data access paths.

4. Maintaining Parser and Import Mechanisms

To avoid regressions, the backport preserved 2.6’s existing parser and bulk import mechanisms. Instead of replacing these components, the RLS feature was carefully woven into the existing code paths, ensuring seamless integration without disrupting established workflows.

5. Incorporating Fixes from the RLS7 Branch

The backport included critical fixes originally developed in the RLS7 branch, such as tag decoding improvements, batch validation enhancements, and policy recovery strategies. These fixes were adapted to 2.6’s environment, ensuring parity in behavior with the master branch.

Lessons Learned: Backporting Complex Features Across Versions

This backport illustrates several important lessons for engineering teams:

Decouple core logic from dependencies: Successful backporting requires isolating feature logic from newer APIs and data structures, enabling adaptation to older environments.
Adapt serialization and parsing carefully: Differences in protobuf versions and import formats can cause subtle bugs, so thorough testing and adjustment are essential.
Preserve existing functionality: Integrating new features should avoid disrupting established workflows, particularly in widely used components like parsers and importers.
Synchronize distributed components thoughtfully: Proxy synchronization mechanisms must be compatible with the target version’s architecture to maintain consistency.

By carefully balancing these factors, the Milvus team enabled secure, multi-tenant usage of 2.6 without sacrificing stability or performance.

This article is based on the Milvus PR titled "feat: [2.6] backport row-level security" from the milvus-io/milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

How Airflow Handles Dynamic Task Executions with Immutable Dynamic Regions

Priyadarshan Singh — Sun, 11 Oct 2026 18:57:47 +0000

Understanding Dynamic Task Executions in Apache Airflow

Apache Airflow is a powerful platform for orchestrating complex workflows, but as workflows grow more dynamic—featuring loops, retries, and mapped tasks—tracking the state of each task execution becomes increasingly challenging. Have you ever wondered how Airflow internally distinguishes between multiple executions of the same task within dynamic constructs? This article dives into a recent improvement in Airflow’s core that introduces the concept of "dynamic regions" to solve this problem elegantly and efficiently.

The Problem: Ambiguity in Tracking Dynamic Task Instances

In Airflow, each task execution is represented by a Task Instance (TI), uniquely identified by a combination of dag_id, task_id, run_id, and map_index. This works well for static workflows where each task runs once per DAG run. However, with the rise of dynamic workflows—such as loops that execute a task multiple times or mapped tasks that spawn multiple parallel executions—this identification scheme falls short.

Imagine a loop that runs the same task multiple times within a single DAG run. Because the task_id and run_id remain the same across iterations, Airflow cannot distinguish between these individual executions. This ambiguity leads to collisions in the database and makes it impossible to track retries or clears for distinct iterations correctly.

Attempts to fix this by rewriting existing task instance rows to add more detailed identifiers proved impractical at production scale. Such rewrites would involve updating potentially millions of rows, leading to unacceptable downtime and performance degradation.

The Core Insight: Introducing Immutable Dynamic Regions

To address this, the Airflow team introduced a new concept: dynamic regions. Rather than trying to retrofit existing task instance identifiers, they created a dedicated, immutable database table to represent each execution context of dynamic constructs like loops or mapped tasks.

What is a Dynamic Region?

A dynamic region encapsulates a logically distinct execution context within a DAG run. For example, each iteration of a loop or each batch of mapped tasks gets its own dynamic region. This means that instead of relying solely on dag_id, task_id, run_id, and map_index, task instances now reference a region_id that explicitly disambiguates which dynamic execution they belong to.

How Does This Work in Practice?

New Table: A dynamic_region table is added to the Airflow metadata database. Each row represents a unique dynamic execution context.
Immutable Records: Once created, a dynamic region record is immutable, ensuring consistency and simplifying reasoning about task executions.
Task Instance Reference: The task_instance table gains a foreign key reference to dynamic_region.region_id. This allows multiple task instances with the same task_id and run_id to coexist without collision because they belong to different dynamic regions.

Migration Strategy

A critical challenge was migrating existing data without costly rewrites. To avoid long migration times, the implementation uses a sentinel zero value for the region_id field in legacy task instances. This means:

Existing task instances default to a region_id of zero, preserving the old behavior.
New dynamic executions get proper region IDs assigned.
No extensive data rewriting is required, enabling a smooth transition at scale.

Why This Matters: Lessons Learned

This change highlights an important principle in evolving database schemas for complex, stateful systems:

Avoid Costly Data Rewrites: Rewriting large volumes of existing data is often impractical in production environments.
Add Dedicated Entities for New Concepts: Introducing a new immutable entity to represent dynamic execution contexts cleanly separates concerns and simplifies data integrity.
Use Sentinel Values for Backward Compatibility: This strategy enables incremental adoption without service disruption.

By embracing these principles, Airflow can now accurately track multiple executions of the same task within dynamic workflows, improving reliability and observability.

Conclusion

The introduction of dynamic regions in Airflow represents a thoughtful and scalable approach to a subtle but impactful problem. By adding a dedicated immutable table to represent dynamic execution contexts, Airflow avoids the pitfalls of costly data migrations and enables precise tracking of complex task executions. This design pattern is a valuable example for any system dealing with evolving execution semantics and large-scale state management.

This article is based on a recent pull request merged into the Apache Airflow repository. Read the full breakdown on nossletter.tech.