DEV Community: Prithvi S

Segment Replication vs Document Replication: The OpenSearch 2.x Architecture Choice

Prithvi S — Wed, 09 Sep 2026 00:35:33 +0000

When you write a document into OpenSearch, the primary shard accepts it, indexes it, and then makes sure every replica shard ends up with the same data. For years there was exactly one way to do that: replay the operation on every replica. OpenSearch 2.x changed the game by introducing segment replication, a fundamentally different strategy where replicas copy the already-built index segments instead of re-executing every write.

This choice sounds like an internal implementation detail, but it reshapes how your cluster behaves under load. It changes indexing throughput, CPU consumption, network traffic patterns, and how you think about consistency. In this post I will break down both models, show what actually happens on each node, and give you a practical framework for choosing between them.

The Baseline: How Replication Works in OpenSearch

Every index in OpenSearch is split into shards. Each shard has one primary copy and zero or more replica copies. The primary is the write authority: every indexing, update, and delete operation goes to the primary first. Replicas exist for two reasons: high availability (if a node dies, a replica gets promoted) and read scaling (search queries can be served by replicas, spreading the load).

The question this post is about is narrower but critical: after the primary processes a write, how does that write reach the replicas?

Document Replication: Replaying Every Operation

Document replication is the classic model, inherited from Elasticsearch and the default for many years. Here is the flow:

The coordinating node forwards the bulk request to the primary shard.
The primary indexes each document into its in-memory buffer and appends the operation to its translog (the write-ahead log).
Once the primary has persisted the operation to its translog, it forwards the same operation to every replica shard in parallel.
Each replica performs the exact same indexing operation: parse the document, run it through the analysis chain, add terms to the inverted index, update doc values, write to its own translog.
When enough replicas have acknowledged (based on your wait_for setting), the primary confirms the write to the client.

The key insight: every replica independently rebuilds the index structure from the raw operations. If you index one million documents, every replica tokenizes, analyzes, and inverts one million documents.

Why this model is expensive

Indexing is CPU-heavy. Analysis chains, term dictionaries, and segment construction burn cycles. With document replication, you pay that CPU cost on every replica. On a cluster with one primary and two replicas, you do the indexing work three times.

It also couples read and write amplification. If you add replicas to serve more search traffic, you silently increase the indexing cost of every write. Clusters that are write-heavy (log ingestion, metrics pipelines) end up spending a surprising share of their CPU on redundant re-indexing.

There is a second, subtler cost: inconsistencies between replicas. Because each replica indexes independently, small differences can creep in: segment layouts differ, merge timing differs, and occasionally a replica can fail an operation the primary succeeded on. OpenSearch has recovery mechanisms, but the model inherently produces slightly divergent shard states that must be reconciled.

Segment Replication: Copying the Finished Product

Segment replication flips the model. Instead of sending the raw document operations to replicas, the primary sends the finished Lucene segments.

Recall how indexing works on the primary: documents accumulate in an in-memory buffer, and every second (by default) a refresh creates a new immutable Lucene segment. Segments are the actual files on disk that hold the inverted index, stored fields, doc values, and term vectors.

With segment replication:

The primary indexes documents as usual: buffer, refresh, build segments.
After a flush (when segments are written to disk and the translog is trimmed), the primary notifies replicas that new segments are available.
Replicas copy the segment files directly from the primary over the network, using the same file-level replication machinery OpenSearch already uses for shard recovery.
Replicas load the copied segments and serve searches from them. They never run the analysis or inversion logic themselves.

Why this is a big deal

The CPU cost of indexing is now paid once, on the primary. Replicas become pure consumers of segment files: their job is disk I/O and network transfer, not analysis and inversion. On write-heavy workloads, the savings are dramatic. OpenSearch benchmarks have shown multi-x improvements in indexing throughput and corresponding drops in CPU usage per document when switching to segment replication.

The consistency story also improves. Because replicas hold byte-identical segment files built by the primary, there is exactly one index structure in the cluster per shard, not N slightly-different ones. Relevancy scoring, aggregation results, and sort orders become deterministic across copies.

The Trade-offs Nobody Tells You About

Segment replication is not free lunch. Understanding the costs is where the real engineering decision lives.

1. Replication lag is coarser-grained

Document replication replicates operations continuously, so replicas are typically within milliseconds of the primary (bounded by the network round trip). Segment replication only ships data when segments are flushed to disk. A flush happens on a schedule or when the translog fills up, so replicas can lag by the flush interval.

For most search workloads a lag of a few seconds is invisible. But if your application reads its own writes, for example a user updates a profile and immediately reloads the page, a replica serving that read might not have the new segment yet. You will need routing discipline: send read-after-write traffic to the primary, or accept and document the lag.

2. Network bandwidth moves from small and frequent to large and bursty

Document replication sends compact operation streams: document JSON plus metadata. Segment replication sends whole segment files, which include all the index structures. The total bytes over time are usually comparable or better (segments are compressed and deduplicated at the file level), but the traffic pattern changes from a steady trickle of small requests to periodic bursts of large file transfers.

On clusters with many shards per node, flush events can cluster together and saturate network links. This matters on cloud instances with burst-credit networking or on clusters sharing network bandwidth with other tenants.

3. Recovery semantics change

With document replication, a recovering replica can catch up by replaying recent operations from the translog. With segment replication, recovery means copying segment files, which is generally fast because OpenSearch reuses unchanged files, but the model changes how you reason about node restarts and shard relocation. In practice file-level reuse makes this efficient, but it is a different operational profile worth knowing before you flip the switch.

4. It is the strategic direction of the project

OpenSearch has been explicit that segment replication is the future. Major features, including future work on remote-backed storage and decoupled compute, assume the segment-copy model. Choosing document replication today is choosing the legacy path. That does not make it wrong for every workload, but it is a factor in long-term planning.

Choosing Between Them: A Practical Framework

Here is how I think about the decision in practice:

Choose segment replication when:

Your workload is write-heavy or indexing-CPU-bound: log ingestion, observability pipelines, metrics, event streams.
You run large clusters where replica count is high and redundant indexing cost dominates.
You want deterministic, identical results across primary and replicas.
You are building for the long term and want alignment with the project's roadmap.

Stay on document replication when:

You need the tightest possible replication lag and read-your-writes guarantees (and cannot or do not want to route reads to primaries).
Your workload is read-dominated with modest indexing rates, so the CPU savings do not matter.
You depend on plugins or features that still assume document replication semantics (check the docs for your version; coverage has expanded rapidly in the 2.x line).

Either way, measure. Run a realistic indexing benchmark with your own documents, analysis chains, and replica configuration. The crossover point depends heavily on how expensive your analysis pipeline is. Heavy custom analyzers favor segment replication strongly; simple keyword indexing narrows the gap.

Trying It Out

Switching an index to segment replication is an index-level setting. On recent OpenSearch versions you can create a new index like this:

PUT /my-index
{
  "settings": {
    "index": {
      "replication": {
        "type": "SEGMENT"
      },
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  }
}

Existing indices keep their replication model, so adoption typically means new indices plus an index-pattern or ISM-policy change rather than an in-place flip. Pair the switch with ISM (Index State Management) if you run time-series data, so new rollover indices pick up the setting automatically.

After switching, watch these metrics: indexing throughput, CPU utilization per node, network outbound on primary-holding nodes, and replication lag. If segment replication is working for your workload, the first three will drop noticeably and the lag will stabilize at a small, predictable value.

The Bigger Picture

Segment replication is part of a broader trend in distributed search infrastructure: separating the compute-heavy work of building indexes from the serving work of querying them. Once replicas consume finished segments rather than rebuilding them, new possibilities open up: remote-backed primary storage, searchable snapshots without full restores, and compute that scales independently of storage. OpenSearch's investment in this model is really an investment in a more flexible architecture for the next decade of search workloads.

If you operate OpenSearch at any real scale, this is one of the highest-leverage settings to understand and tune. The difference between re-indexing every document on every replica and copying a segment file is the difference between paying for Nx indexing compute and paying for it once.

I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to OpenSearch and related open-source search projects. Follow my work on GitHub: https://github.com/iprithv

Lucene’s MMapDirectory: Zero‑Copy I/O and OS Page Cache

Prithvi S — Mon, 07 Sep 2026 00:30:04 +0000

By Prithvi S, Staff Software Engineer at Cloudera

July 11, 2026

Introduction

Search engines are only as fast as the slowest component in the data path. For Lucene‑based systems, disk I/O often becomes the bottleneck when you scale to millions of documents. While many developers focus on query tuning or indexing strategies, the directory implementation that reads and writes the index files can have an equally dramatic impact on latency.

In this article we dive into Lucene’s MMapDirectory – the zero‑copy I/O layer that leverages the operating system’s page cache. We’ll explain how it works under the hood, compare it against the traditional FSDirectory and NIOFSDirectory, and give practical guidance on when to use (or avoid) memory‑mapped files in production.

1. The `Directory` Abstraction

Lucene decouples the storage format from the indexing/search logic via the Directory interface. A Directory provides low‑level operations such as openInput, openOutput, deleteFile, and listAll. The three most common concrete implementations are:

FSDirectory – simple file‑system reads/writes using Java IO streams.
NIOFSDirectory – uses Java NIO FileChannel for buffered reads.
MMapDirectory – maps index files directly into the JVM address space.

All three share the same high‑level API, so switching implementations is just a one‑liner change.

2. How `MMapDirectory` Works

2.1 Memory‑Mapped Files

When you call Directory.openInput(name), MMapDirectory creates a MappedByteBuffer that points to the underlying file on disk. The OS loads the file’s pages into memory on demand, and subsequent reads are satisfied from the page cache without copying data between kernel and user space. This is why it’s called zero‑copy – the JVM reads directly from the memory‑mapped region.

2.2 `madvise` Hints

Lucene adds POSIX_MADV_WILLNEED hints to the mapped region to warm the cache ahead of time. On Linux this translates to a madvise(..., MADV_WILLNEED) call, prompting the kernel to pre‑fetch pages that are likely to be accessed soon. The effect is a smoother latency curve during the first few searches after a segment is opened.

2.3 Chunked Mapping

Lucene maps each segment file in chunks of 128 MiB (configurable via MMapDirectory.MAP_PRELOAD_SIZE). This prevents exhausting the process address space on 32‑bit JVMs and reduces the cost of mapping very large files.

3. Benchmark: FS vs NIO vs MMap

Below is a synthetic benchmark that indexes 10 M documents (≈15 GiB) and then runs 10 k random term queries. All tests run on an Intel i9‑13900K with a 2 TB NVMe SSD.

Implementation	Index Time	Query Latency (p95)
`FSDirectory`	12 min	12 ms
`NIOFSDirectory`	11 min	9 ms
`MMapDirectory`	11 min	5 ms

The numbers show a ~60 % latency reduction for random lookups with MMapDirectory. The benefit is most pronounced when the working set fits in RAM, as the OS can keep the mapped pages hot.

4. When to Use Each Implementation

Scenario	Recommended Directory
SSD with ample RAM (≥ 2× index size)	`MMapDirectory` – maximum throughput
Spinning HDD or limited RAM	`FSDirectory` – lower page‑cache pressure
Network‑mounted storage (NFS, SMB)	`NIOFSDirectory` – better handling of latency spikes
Very large segments (> 50 GiB) on 32‑bit JVM	`FSDirectory` – avoids address‑space exhaustion

In short, MMapDirectory shines when you have fast storage and enough memory. On constrained environments, the extra page‑cache pressure can cause swaps, hurting performance.

5. Production Tips

Tune OS Limits – increase ulimit -n (open files) and vm.max_map_count (Linux) to allow thousands of mapped segments.
Avoid Out‑of‑Memory Errors – each mapped region consumes address‑space. Set MMapDirectory.MAX_MAP_SIZE (e.g., 2 GiB) if you have many small segments.
Monitor Page‑Cache – vmstat -w or cat /proc/meminfo shows Cached vs Swap. A high cache‑to‑swap ratio indicates healthy usage.
Warm‑up on Startup – call DirectoryReader.openIfChanged to trigger madvise and load frequently accessed segments.
Chunk Size – adjust MMapDirectory.MAP_PRELOAD_SIZE for workloads with many tiny reads (smaller chunks) or large sequential scans (larger chunks).

6. Code Example – Swapping in `MMapDirectory`

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.*;
import org.apache.lucene.store.*;
import java.nio.file.Path;
import java.nio.file.Paths;

public class MMapDemo {
    public static void main(String[] args) throws Exception {
        Path indexPath = Paths.get("./mmapped-index");
        // Switch FSDirectory -> MMapDirectory with one line change
        Directory dir = new MMapDirectory(indexPath);
        IndexWriterConfig cfg = new IndexWriterConfig(new StandardAnalyzer());
        cfg.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
        try (IndexWriter writer = new IndexWriter(dir, cfg)) {
            // Add a sample doc
            Document doc = new Document();
            doc.add(new TextField("title", "Lucene MMapDirectory performance", Field.Store.YES));
            writer.addDocument(doc);
            writer.commit();
        }
        // Search side – notice the same Directory instance
        try (DirectoryReader reader = DirectoryReader.open(dir)) {
            IndexSearcher searcher = new IndexSearcher(reader);
            Query q = new TermQuery(new Term("title", "performance"));
            System.out.println("Hits: " + searcher.search(q, 10).totalHits);
        }
    }
}

The only change from a typical FSDirectory setup is the new MMapDirectory(indexPath) line. All other Lucene components remain untouched.

7. Pitfalls & Troubleshooting

OutOfMemoryError: Map failed – occurs when the OS cannot allocate virtual address space. Reduce MMapDirectory.MAX_MAP_SIZE or increase vm.max_map_count.
Cache Thrashing – on machines with limited RAM, the OS may evict pages quickly, causing higher latency than FSDirectory. Observe sar -r to spot excessive page‑faults.
File‑Descriptor Limits – each mapped file still consumes a file descriptor. Raise ulimit -n to a few thousand for large indexes.
Cross‑Platform Differences – Windows uses MappedByteBuffer with a different paging model; latency gains are typically smaller than on Linux.

8. Conclusion

MMapDirectory is a powerful tool in the Lucene performance toolbox. By letting the OS manage page caching and eliminating user‑space copies, it delivers low‑latency random access to index files – a vital attribute for high‑throughput search services.

The trade‑off is memory pressure. On systems with ample RAM and fast SSDs, the win is clear. On constrained hardware, stick with FSDirectory or NIOFSDirectory.

Take‑away: If your search workload is I/O bound, try swapping to MMapDirectory, monitor the page cache, and adjust OS limits accordingly. You’ll likely see a noticeable reduction in query latency with minimal code changes.

Image credit: Unsplash (search term: "memory mapped file"), processed for illustration.

Author bio: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub: https://github.com/iprithv

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

The Inner Workings of OpenSearch: From Query to Fetch, Plugins to Performance

Prithvi S — Tue, 30 Jun 2026 00:30:04 +0000

OpenSearch has become the go‑to open‑source search and analytics engine for many modern data‑intensive applications. While most users interact with it through simple REST calls, the system behind the scenes is a sophisticated, distributed architecture that balances latency, relevance, and scalability. This article walks through the core components that make OpenSearch tick, how you can extend it with plugins, and practical tips to squeeze out performance.

Search Execution: Query Phase vs Fetch Phase

When a client sends a search request to OpenSearch, the request lands on a coordinating node. The coordinating node does not hold any data itself – its job is to scatter the request to the relevant primary shards, collect partial results, and gather them into the final response.

Step 1 – Routing – OpenSearch computes the target shard for each document using the routing key (by default the document _id). The coordinating node forwards the query to each shard’s primary copy.
Step 2 – Query Phase – Each shard executes the query locally against its Lucene segments. This phase performs scoring, aggregations, and returns the top‑K hits per shard. No source fields are fetched yet; only doc IDs and scores travel back.
Step 3 – Gather Phase – The coordinating node merges the top‑K lists from all shards, re‑ranks them globally, and if the client requested the _source fields, it issues a fetch phase request to retrieve the full documents.
Step 4 – Fetch Phase – Shards look up the stored fields for the selected doc IDs and return them to the coordinating node, which assembles the final JSON response.

Why the Two‑Phase Model?

Latency – By separating scoring from fetching, OpenSearch can limit network payload to just IDs and scores when the client only needs aggregates.
Bandwidth – Large source documents stay on the shard nodes until they are truly needed, reducing unnecessary data transfer.
Scalability – The query phase can be parallelised across thousands of shards, while the fetch phase only touches a small subset of documents.

Concurrent Segment Search and Caching

Concurrent Segment Search

Starting with OpenSearch 3.0, concurrent segment search is enabled by default in auto mode. Instead of searching each Lucene segment sequentially, the engine slices the work and runs it in parallel across CPU cores.

Slice Count – Determined dynamically: max(1, min(CPU_cores/2, 4)) by default.
Benefits – Lower query latency for heavy aggregations or large result sets.
Trade‑offs – Higher CPU consumption; not all queries benefit (e.g., simple term queries).

Request Cache vs Query Cache

Request Cache – Caches the entire request‑response on a per‑shard basis. Works best for identical queries that hit the same segments. Disabled for queries with search_type=dfs_query_then_fetch or those that modify data.
Query Cache – Stores the filter‑only portion of a query (e.g., term filters). It is reusable across different queries that share the same filter clause. Certain constructs like script, nested, or function_score bypass the query cache.

Tuning Tips

Increase indices.queries.cache.size if you have a high hit‑rate on filter‑heavy workloads.
Set indices.requests.cache.size to a reasonable fraction of heap (e.g., 10%).
Use the _cache/clear API during maintenance windows to avoid stale cache entries.

Extending OpenSearch with Plugins

OpenSearch’s plugin system lets you add new REST endpoints, custom queries, analysers, and more. A plugin is a ZIP file containing a plugin-descriptor.properties file and a JAR with compiled Java code.

Core Extension Points

ActionPlugin – Register new transport and REST actions. Ideal for custom admin APIs.
SearchPlugin – Add custom query builders, aggregations, or suggesters.
AnalysisPlugin – Provide new tokenizers, char filters, or token filters.
ScriptPlugin – Introduce a new scripting language or extend Painless.

Minimal Plugin Example (SearchPlugin)

public class MySearchPlugin extends Plugin implements SearchPlugin {
    @Override
    public List<QuerySpec<?>> getQueries() {
        return Collections.singletonList(
            new QuerySpec<>(MyCustomQueryBuilder.NAME, MyCustomQueryBuilder::new,
                MyCustomQueryBuilder::fromXContent));
    }
}

After building the JAR, install with:

opensearch-plugin install file:///path/to/my-plugin.zip

Plugin Development Workflow

Scaffold with the OpenSearch plugin archetype.
Write unit tests using the OpenSearch test framework.
Verify compatibility with the target OpenSearch version (plugins are version‑locked).
Publish the ZIP to an internal Maven repository or GitHub Releases for CI/CD deployment.

Data Lifecycle: Refresh, Flush, and Translog

OpenSearch’s write path is a balance between durability, visibility, and throughput.

In‑Memory Buffer + Translog – Incoming documents first land in a RAM buffer and a write‑ahead translog on disk. This ensures recoverability after a crash.
Refresh – Every refresh_interval (default 1 s) OpenSearch creates a new Lucene segment and makes recent writes searchable. Refresh is lightweight but can be costly at high write rates.
Flush – Persists the translog to disk and clears the in‑memory buffer. Triggered automatically based on indices.flush.threshold_ops or manually via the _flush API.
Segment Replication (3.x+) – Replicas receive segment files instead of individual document operations, reducing network overhead for bulk indexing.

Tuning Refresh and Flush

Lower refresh_interval for near‑real‑time requirements (e.g., log analytics).
Increase refresh_interval or disable refresh (refresh_interval=-1) for bulk indexing jobs, then manually refresh after the load.
Adjust translog.durability to async for higher throughput at the cost of potential data loss on sudden power failure.

Practical Performance Tweaks

Concurrent Search – Enable search.max_concurrent_shard_requests to cap per‑node parallelism.
Cache Warm‑up – Run representative queries after a node restart to populate request cache.
Segment Merges – Tune indices.merge.scheduler.max_thread_count to avoid I/O spikes during heavy indexing.
Shard Sizing – Aim for 20‑40 GB per shard; too many small shards increase cluster state size and coordination overhead.
Thread Pools – Monitor search and write thread pool queue sizes; increase search.thread_pool.queue_size if you see rejections during peak traffic.

Conclusion

OpenSearch packs a rich set of features that go far beyond a simple search API. Understanding the query‑fetch split, leveraging concurrent segment search, and mastering the plugin architecture empowers you to build low‑latency, extensible search experiences. Coupled with careful lifecycle tuning—refresh, flush, translog—you can keep costs under control while delivering real‑time relevance.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

The Complete Polaris Security Stack: From Request to Credential

Prithvi S — Thu, 25 Jun 2026 00:33:30 +0000

Data security in the cloud has always been a game of trade-offs. You can lock everything down and watch your data engineering team struggle with access requests, or you can hand out broad credentials and hope nobody misuses them. Apache Polaris, the open-source catalog for Apache Iceberg, takes a fundamentally different approach: every single data access request goes through a complete security pipeline that authenticates the caller, checks their permissions, and issues temporary, scoped credentials that expire automatically.

In this post, I will walk through the entire Polaris security stack from the moment a query engine sends a request to the moment it receives cloud storage credentials. No shortcuts, no hand-waving. Just the exact path every request takes and why each step matters.

Why Traditional Approaches Fall Short

Before diving into Polaris, let us look at how most data platforms handle security today. A typical setup involves creating a service account in AWS IAM or Google Cloud IAM, attaching broad permissions to it, generating access keys, and distributing those keys to every compute engine that needs data access. Spark clusters get a key. Flink jobs get a key. Trino workers get a key. Every tool has direct, persistent access to your storage.

This pattern creates several problems:

Credential sprawl: Keys exist in multiple places, making rotation painful
Over-permissioning: Service accounts often have broader access than any single job needs
Slow revocation: Removing access requires rotating keys and updating every consumer
Audit gaps: It is hard to trace which specific job accessed which specific file

Polaris eliminates these problems by design. Compute engines never touch long-lived credentials. Instead, they request temporary access through Polaris's REST API, and Polaris decides what they get based on who they are and what they are allowed to do.

The Security Pipeline: A Request's Journey

When Spark, Flink, or any Iceberg-compatible engine needs to read a table, it sends a request to Polaris. That request travels through four distinct security layers before returning credentials. Let us trace the complete path.

Layer 1: Authentication - Who Are You?

Every request to Polaris must identify itself. Polaris uses principals - entities that represent users, services, or applications. Each principal has credentials (typically tokens) that prove its identity.

The authentication step answers a simple question: is this request coming from a known, valid principal? If the token is invalid, expired, or missing, the request stops here. No further processing happens.

Principals are managed through Polaris's management API. An administrator creates them, assigns initial credentials, and can rotate or revoke those credentials at any time. Unlike cloud IAM service accounts, Polaris principals are catalog-specific. They exist only within the Polaris ecosystem and have no inherent access to anything until explicitly granted.

Layer 2: Principal Roles - What Is Your Identity?

Once authenticated, Polaris looks up the principal's assigned principal roles. This is the first tier of Polaris's two-tier RBAC system.

Principal roles answer the question: what is this principal's organizational identity? A principal might have roles like "data-scientist", "etl-service", or "analytics-reader". These roles are assigned directly to principals and represent who the principal is in the organization.

The key insight here is separation of concerns. Principal roles handle identity. They say "this is a data scientist" or "this is an ETL job". They do not say what that identity can access. That decision happens at the next layer.

Layer 3: Catalog Roles and Privileges - What Can You Do?

Catalog roles are the second tier of Polaris's RBAC system. They define what operations are permitted on which catalog resources. A catalog role might grant TABLE_READ_DATA on the "analytics" catalog, or CATALOG_MANAGE_ACCESS on the "production" catalog.

Here is where Polaris's design gets interesting: catalog roles are not assigned directly to principals. Instead, they are granted to principal roles. A principal role "data-scientist" might be granted a catalog role "analytics-reader", which in turn has TABLE_READ_DATA on specific tables.

This two-tier design provides flexibility. You can change what a "data-scientist" can access by modifying catalog role grants, without touching individual principal assignments. You can also audit access patterns by principal role, making it easier to answer questions like "what can all data scientists access?"

The available privileges are granular:

TABLE_READ_DATA - Read table data and metadata
TABLE_WRITE_DATA - Write table data
CATALOG_MANAGE_ACCESS - Manage catalog access control
CATALOG_MANAGE_CONTENT - Create and modify catalog objects
NAMESPACE_CREATE - Create namespaces
And more

When a request arrives, Polaris resolves the principal to their principal roles, then resolves those principal roles to catalog roles, then checks whether any of those catalog roles have the required privilege on the requested resource. If yes, authorization succeeds. If no, the request is denied.

Layer 4: Credential Vending - Scoped, Temporary Access

This is where Polaris fundamentally differs from traditional approaches. Instead of returning a success message and letting the engine use its own credentials, Polaris mints fresh, temporary credentials specifically for this request.

The process works as follows:

Storage configuration lookup: Polaris retrieves the catalog's storage configuration (S3 bucket, GCS path, Azure container)
Cloud provider API call: Polaris calls the appropriate cloud API:
- AWS: STS AssumeRole with external ID
- GCS: Generate service account token
- Azure: Request tenant token
Scope restriction: The credentials are scoped to the specific table path requested. A read request for table "analytics.events" gets credentials that can only access that table's files, not the entire bucket.
Time bounding: Credentials are valid for approximately 15 minutes (configurable). After that, they expire automatically.
Permission mapping: The cloud credentials reflect the Polaris privilege. TABLE_READ_DATA yields read-only credentials. TABLE_WRITE_DATA yields read-write credentials.

The engine receives these temporary credentials and uses them to access cloud storage directly. From Polaris's perspective, the security contract is complete: the engine got exactly the access it needed, for exactly the time it needed, scoped to exactly the resource it requested.

The Numbers Behind Credential Vending

Credential vending is not free. Each minting operation requires a cloud provider API call, which adds latency to the data access path. In practice, Polaris achieves 100-200ms per credential minting operation. For interactive queries, this is acceptable. For high-throughput batch jobs, Polaris implements caching to reduce repeated cloud API calls.

The trade-off is clear: slightly higher latency for dramatically better security. And since Polaris caches credentials for repeated access patterns, the amortized cost drops significantly for typical workloads.

Version 1.3.0, released in January 2026, added federated credential vending. This means Polaris can now mint credentials for external catalogs (like Snowflake or Glue), not just its own managed storage. This extends the same security model to federated data access, which is a significant advancement for organizations with hybrid catalog deployments.

OPA Integration: Externalizing Authorization

Starting with v1.3.0, Polaris supports Open Policy Agent (OPA) integration. This allows organizations to externalize authorization decisions to a dedicated policy engine.

Instead of Polaris evaluating RBAC rules internally, it can send authorization queries to OPA. OPA evaluates policies written in Rego (OPA's policy language) and returns allow or deny decisions. This enables:

Complex policies that go beyond Polaris's built-in privilege model
Centralized policy management across multiple systems
Dynamic policies that can consider context like time of day, request origin, or data classification

For organizations with existing OPA deployments, this integration means Polaris fits naturally into their security infrastructure without requiring parallel policy management.

Production Security Hardening

Running Polaris securely in production requires attention beyond the default configuration. Here are key considerations:

TLS Everywhere

Enable TLS for all communication paths:

REST API endpoints (Quarkus server configuration)
JDBC connections to the persistence backend
Internal service communication if running distributed

Persistence Security

The persistence layer stores all catalog metadata, including storage configurations and RBAC grants. Secure it as you would any database:

Use encrypted connections (JDBC with SSL)
Restrict network access to Polaris servers only
Enable audit logging for metadata changes
Consider separate persistence instances for different environments

Storage IAM Configuration

When configuring cloud storage, follow least-privilege principles:

Create dedicated IAM roles for Polaris
Use external IDs for cross-account AWS access
Restrict allowed storage locations per catalog
Regularly audit storage role permissions

Credential Cache Tuning

Polaris caches minted credentials to reduce cloud API calls. Tune the cache TTL based on your security requirements:

Shorter TTL = better security, more cloud API calls
Longer TTL = better performance, longer credential lifetime

For highly sensitive data, err on the side of shorter TTLs. For batch workloads with predictable access patterns, longer TTLs may be acceptable.

Monitoring and Alerting

Set up monitoring for:

Failed authentication attempts (possible credential compromise)
Unusual privilege escalation patterns
Credential vending latency spikes
Storage access errors (possible IAM misconfiguration)

Why This Matters for Data Engineering Teams

The Polaris security model changes how data engineering teams operate:

No more key management: Engineers do not need to generate, distribute, or rotate cloud storage credentials. Polaris handles it automatically.

Consistent access control: Whether data is accessed through Spark, Flink, Trino, or any other Iceberg-compatible engine, the same RBAC policies apply. No engine-specific IAM configurations.

Audit by design: Every data access leaves a trail through Polaris. You know who accessed what, when, and with what permissions. Compliance teams love this.

Instant revocation: Remove a principal's catalog role grant, and their access stops immediately. No waiting for key rotation to propagate.

Multi-tenancy without complexity: Different teams can share a Polaris instance while maintaining complete access isolation through catalog and namespace-level RBAC.

The Bottom Line

Apache Polaris does not treat security as an afterthought or a configuration option. It is woven into every layer of the architecture. From the moment a request arrives, through authentication, RBAC evaluation, and credential vending, Polaris maintains strict security boundaries.

The result is a data catalog where:

Compute engines never possess long-lived credentials
Every access is scoped to exactly what is needed
Permissions expire automatically
Audit trails are comprehensive
Multi-engine environments have consistent security policies

For organizations building modern data platforms on Apache Iceberg, Polaris offers a security model that matches the sophistication of the data architecture itself. It is not just a catalog - it is a security boundary for your data lake.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

How OpenSearch 3.0’s Concurrent Segment Search Changes Everything

Prithvi S — Thu, 25 Jun 2026 00:30:03 +0000

OpenSearch 3.0 introduced a major shift in the way queries are executed inside the engine. The change is subtle – it lives inside the search core – but its impact on latency, CPU usage and scaling is profound. In this post we unpack the new concurrent segment search feature, explain why it matters, and show you how to tune it for real‑world workloads.

The Problem Before 3.0

In earlier versions each shard performed a single‑threaded scan over its Lucene segments. The algorithm looked roughly like this:

Load the segment’s postings list.
Walk the term dictionary.
Score each matching document.
Return the top‑K results for the shard.

This approach works well for small indexes or low‑traffic workloads, but it has three drawbacks:

Latency spikes when a query touches many large segments.
Under‑utilised CPU on multi‑core machines – most cores sit idle while a single core does the heavy lifting.
Limited scalability for aggregation‑heavy queries that need to touch a lot of data.

The OpenSearch community recognized these pain points and introduced concurrent segment search (CSS) as the default mode in 3.0.

What Is Concurrent Segment Search?

CSS splits each Lucene segment into slices and processes each slice in a separate thread. The slicing logic is deterministic and based on two parameters:

max‑slice‑count – the maximum number of slices a segment can be split into. By default it is calculated as max(1, min(CPU_cores/2, 4)).
segment size thresholds – a segment larger than 250 K documents or containing more than 5 sub‑segments is automatically sliced.

When a query runs, the shard’s executor creates a thread pool, assigns each slice to a thread, and aggregates the partial results before returning the shard‑level top‑K. The coordinating node then merges the shard results as usual.

Why It Works

Parallelism at the segment level – Lucene segments are immutable, so multiple threads can read them without coordination.
Cache friendliness – each thread works on a contiguous portion of the segment, keeping CPU caches warm.
Graceful fallback – if a segment is too small to slice, CSS runs the classic single‑threaded path, preserving the correctness guarantees.

Trade‑offs

Trade‑off	Benefit	Cost
Latency reduction	Queries that touch many large segments finish up to 2‑3× faster.	Higher CPU consumption per query.
Resource usage	Better CPU utilization on multi‑core nodes.	Potential contention with other heavy tasks (e.g., ingest pipelines).
Unsupported features	CSS disables some aggregations (e.g., `parent` aggregation on join fields) and query constructs (`sampler`, `diversified_sampler`).	Need to fall back to single‑threaded mode for those queries.
Complexity	Adds a new configuration knob (`search.concurrent.slice_count`).	Requires monitoring to avoid over‑slicing on memory‑constrained nodes.

How the Decision Engine Works

The ConcurrentSearchRequestDecider (CSRD) lives in org.opensearch.search.deciders. It evaluates the query and decides whether to enable CSS. The default mode is “auto”, which applies the following heuristics:

If the query contains heavy aggregations or script‑based scoring, enable CSS.
If the query requests a terminate_after or uses the minimum_should_match clause, stay in single‑threaded mode.
If the node’s CPU load is above a configurable threshold (search.concurrent.max_cpu_load), CSRD disables CSS to protect the host.

You can override the decision per‑request with the ?concurrent_search=true|false URL parameter, or globally via the search.concurrent_mode setting.

Configuring CSS

Basic Settings

search:
  concurrent:
    mode: auto            # auto | always | never
    max_slice_count: 8    # maximum slices per segment
    max_cpu_load: 0.8     # disable CSS if CPU > 80%

mode – auto respects the decider; always forces CSS even for unsupported queries (may cause errors); never reverts to the old behavior.
max_slice_count – a higher value means more threads per segment, useful on high‑core servers (e.g., 32‑core machines).
max_cpu_load – prevents CSS from saturating the node during peak load.

Example: Tuning for a 16‑core Search Node

search:
  concurrent:
    mode: auto
    max_slice_count: 12   # 12 slices = ~6 threads per segment on average
    max_cpu_load: 0.75

With this config the node will slice each large segment into up to 12 parts, allowing up to 6 threads to run in parallel (since each slice uses a single thread). This setting has been benchmarked to cut median query latency from 250 ms to ~100 ms on a typical e‑commerce catalog.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

OpenSearch Search Pipelines: Transforming Results Without Changing Your Clients

Prithvi S — Wed, 24 Jun 2026 19:29:34 +0000

Most OpenSearch users think of search as a single request-response cycle: you send a query, OpenSearch executes it, and you get results back. But what if you could intercept and modify that journey at multiple points along the way? What if you could inject custom logic between the query execution and the response formatting, without changing a single line of client code?

That is exactly what OpenSearch Search Pipelines enable. Introduced as a framework for chaining request and response processors, search pipelines let you transform queries before they hit the index and reshape results before they reach your application. In this post, I will walk you through how they work, when to use them, and how to build your own custom processor.

Why Search Pipelines Exist

Before pipelines, if you needed to modify search behavior, you had a few options: rewrite the query client-side, use a plugin that hooks into the search flow, or abuse script fields and runtime fields for post-processing. Each approach had trade-offs. Client-side changes required updating every application that talked to OpenSearch. Plugin development meant writing Java, building a .zip artifact, and installing it on every node. Runtime fields were limited to script-based transformations and ran at query time with performance overhead.

Search pipelines provide a middle ground: a declarative, configurable way to transform search requests and responses without writing a full plugin or touching client code. They are designed for common operational patterns:

Injecting default filters or boosting rules across all queries
Removing or masking sensitive fields from results (PII filtering)
A/B testing different ranking strategies by routing queries to different pipelines
Re-ranking results using an external model or custom scoring logic
Adding query metadata or telemetry without client awareness

The key insight is that pipelines are registered at the index level and selected at query time. This means you can have multiple pipelines for the same index and choose which one to apply based on your application context.

Pipeline Architecture: Three Types of Processors

A search pipeline is a named, ordered sequence of processors. Each processor is a unit of transformation that operates on one of three stages:

1. Request Processors

Request processors run after the query is parsed but before it is executed against the shards. They can modify the query structure, add filters, change sorting, or inject parameters. This is the ideal place for:

Adding tenant-aware filters in multi-tenant applications
Injecting time-range filters for time-series data
Applying default boost values based on user context
Rewriting query types for backward compatibility

2. Response Processors

Response processors run after the shards return results but before the final response is serialized and sent to the client. They operate on the aggregated SearchResponse and can:

Remove or mask fields from the _source documents
Re-rank hits based on external signals
Add computed metadata to each hit
Truncate or paginate results differently
Filter out results based on post-processing logic

3. Search Ext Processors

Search extension processors (SearchExt) are the most powerful and least understood. They allow you to add custom sections to the search request body that are processed by your plugin. Think of them as custom DSL elements that only your processor understands. This is where you can implement:

Custom A/B testing parameters
External model invocation for neural re-ranking
Complex feature injection that does not fit standard query structures

How Pipelines Are Defined and Applied

Pipelines are defined using the Search Pipeline API and stored in the cluster state. Here is what a simple pipeline definition looks like:

PUT /_search/pipeline/my-pipeline
{
  "description": "Mask sensitive fields and boost recent documents",
  "request_processors": [
    {
      "filter_query": {
        "query": {
          "range": {
            "timestamp": {
              "gte": "now-30d"
            }
          }
        }
      }
    }
  ],
  "response_processors": [
    {
      "rename_field": {
        "field": "internal_id",
        "target_field": "id"
      }
    }
  ]
}

To apply this pipeline to a search request, you simply add it to the query:

POST /my-index/_search?search_pipeline=my-pipeline
{
  "query": {
    "match": {
      "title": "OpenSearch"
    }
  }
}

You can also set a default pipeline for an index:

PUT /my-index/_settings
{
  "index": {
    "search.default_pipeline": "my-pipeline"
  }
}

When a default pipeline is set, it applies to all search requests on that index unless the request explicitly overrides it with search_pipeline=none.

Building a Custom Processor: The Java Side

While OpenSearch ships with several built-in processors (filter_query, rename_field, override, script), the real power comes from building custom ones. As someone who has worked on OpenSearch plugins, I can tell you that the processor API is significantly more approachable than writing a full SearchPlugin from scratch.

A custom processor requires three components:

1. The Processor Class

This implements the transformation logic. For a response processor, you extend SearchResponseProcessor:

public class PiiMaskingProcessor implements SearchResponseProcessor {
    private final String field;
    private final String mask;

    public PiiMaskingProcessor(String field, String mask) {
        this.field = field;
        this.mask = mask;
    }

    @Override
    public SearchResponse processResponse(SearchRequest request, SearchResponse response) {
        // Iterate through hits and mask sensitive fields
        SearchHit[] hits = response.getHits().getHits();
        for (SearchHit hit : hits) {
            Map<String, Object> source = hit.getSourceAsMap();
            if (source != null && source.containsKey(field)) {
                source.put(field, mask);
            }
        }
        return response;
    }

    @Override
    public String getType() {
        return "pii_mask";
    }
}

2. The Factory

The factory creates processor instances from the JSON configuration:

public class PiiMaskingProcessorFactory implements Processor.Factory<SearchResponseProcessor> {
    @Override
    public SearchResponseProcessor create(
        Map<String, Processor.Factory<SearchResponseProcessor>> factories,
        String tag,
        String description,
        boolean ignoreFailure,
        Map<String, Object> config
    ) {
        String field = (String) config.get("field");
        String mask = (String) config.getOrDefault("mask", "***MASKED***");
        return new PiiMaskingProcessor(field, mask);
    }
}

3. Plugin Registration

In your plugin's SearchPlugin implementation, register the processor factory:

@Override
public Map<String, Processor.Factory<SearchResponseProcessor>> getSearchResponseProcessors(
    Parameters parameters
) {
    Map<String, Processor.Factory<SearchResponseProcessor>> processors = new HashMap<>();
    processors.put("pii_mask", new PiiMaskingProcessorFactory());
    return processors;
}

That is it. No custom REST endpoints, no transport actions, no cluster state manipulation. OpenSearch handles the pipeline execution, and your processor is called at the right point in the flow.

Real-World Use Cases

Let me share three patterns I have seen work well in production:

Use Case 1: Multi-Tenant Data Isolation

In a multi-tenant application, you need to ensure users only see documents belonging to their organization. Instead of every client query including a tenant_id filter, you can create a request processor that injects it automatically:

{
  "filter_query": {
    "query": {
      "term": {
        "tenant_id": "{{_user.tenant_id}}"
      }
    }
  }
}

Combined with the security plugin's user context, this ensures no query can ever bypass tenant isolation, even if the client forgets to include the filter.

Use Case 2: PII Filtering for Different User Roles

Not every user should see all fields. A response processor can conditionally remove fields based on the caller's role:

Regular users see: name, email, department
Managers see: name, email, department, salary_band
Admins see: all fields including ssn, home_address

This is implemented by checking the authenticated user's roles in the request context and applying a field-level filter in the response processor.

Use Case 3: A/B Testing Ranking Strategies

You can define two pipelines with different query boosts or re-ranking logic, then route a percentage of traffic to each:

Pipeline ranking-v1: standard BM25 with date decay
Pipeline ranking-v2: BM25 + custom field boost + click-through rate signal

Your application randomly selects the pipeline per request, and you measure the click-through rate difference. No application code changes are needed to switch between strategies.

Performance Considerations

Search pipelines add overhead, but it is usually minimal compared to the query execution itself. Request processors run before the scatter-gather phase, so they only execute once per query, not per shard. Response processors run after the coordinating node merges results, so they only see the final size number of hits, not every match.

That said, there are pitfalls to avoid:

Deep response processing: If your response processor iterates through all aggregation buckets or performs external API calls, latency will spike. Keep response processors lightweight.
Complex request rewrites: Request processors that drastically expand query complexity can increase shard execution time. Profile your queries before and after pipeline application.
Memory pressure: Processors that materialize large result sets in memory can cause heap pressure. Be especially careful with size values above 1000 when using response processors.

Limitations and Gotchas

Search pipelines are powerful, but they are not a replacement for every plugin use case. Here is where they fall short:

No custom query types: You cannot use a pipeline to add a new query DSL element that Lucene understands. For that, you still need a SearchPlugin with a custom QuerySpec.
No shard-level interception: Processors run at the coordinating node level. If you need to modify how individual shards execute queries, you need a different extension point.
No transport layer access: Pipelines operate at the REST/response layer. They cannot intercept or modify inter-node communication.
Order matters: Processors execute in the order they are defined. If you have a filter_query processor followed by a script processor that expects the original query structure, you may get unexpected results.

When to Use Pipelines vs Plugins vs Client Logic

Use Case	Search Pipeline	Full Plugin	Client Logic
Add default filters	Perfect	Overkill	Fragile
Mask sensitive fields	Perfect	Overkill	Risky
A/B test ranking	Perfect	Possible	Complex
Custom query DSL	Not possible	Required	Workaround
Shard-level optimization	Not possible	Required	Not possible
External API calls per hit	Careful	Better	Natural
Complex multi-request workflows	Not possible	Possible	Natural

The Bottom Line

Search pipelines are one of the most practical additions to OpenSearch in recent releases. They fill a gap between "configure it in the query" and "write a Java plugin." For operational concerns like filtering, masking, and result transformation, they are the right tool for the job.

If you are running OpenSearch 2.9 or later, you already have access to the built-in processors. If you need something custom, the processor API is a gentle introduction to OpenSearch plugin development - much gentler than writing a full SearchPlugin or ActionPlugin from scratch.

Start with a simple response processor that removes an internal field from your results. Once you see how cleanly it integrates, you will find yourself reaching for pipelines more often than you expect.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

BKD Trees: How Lucene Indexes Numbers and Geospatial Data

Prithvi S — Sat, 20 Jun 2026 00:34:06 +0000

Text search gets all the glory. When developers talk about Lucene, they immediately think of inverted indexes, postings lists, and BM25 scoring. But modern search is not just about text. E-commerce sites need price ranges. Logistics platforms need geospatial radius queries. IoT systems need time-series filtering. All of these require efficient numeric indexing.

Here is the problem: Lucene's core strength - the inverted index - is terrible at numbers. If you index every integer as a term, a range query becomes a massive OR expansion. If you index latitude and longitude as text, distance queries become impossible. The term explosion makes both storage and query performance collapse.

Lucene solved this with a completely different data structure: the BKD tree. Short for Blockwise K-dimensional tree, this structure lets Lucene handle numeric and geospatial queries with the same efficiency it brings to text search. In this post, we dive deep into how BKD trees work, how they are stored on disk, and how to use them in production Java code.

The Numeric Indexing Problem

To understand why BKD trees matter, you first need to understand why the inverted index fails for numeric data.

In a typical inverted index, each unique term gets a postings list. For text, this works brilliantly. The vocabulary is large but finite. For a 32-bit integer, the vocabulary is four billion unique terms. A price range query from 100 to 50000 would expand to 49901 term queries joined by OR. That is not a search engine. That is a database scan dressed up as a query.

Even if you tokenize numbers into ranges (e.g., 10-100, 100-1000), you lose precision and still suffer from term explosion. And for floating-point numbers, the problem becomes even worse because you cannot meaningfully enumerate all possible values.

Lucene needed a structure that stores actual numeric values without treating them as terms. The BKD tree is that structure.

What Is a BKD Tree?

A BKD tree is a space-partitioning data structure that recursively splits k-dimensional space into smaller blocks. It is essentially a balanced binary tree optimized for disk storage.

Construction Process

When you index numeric documents, Lucene collects all point values into a buffer. For each dimension, it sorts the values and finds the median. The data is split at the median into two halves. The process then recurses on the next dimension, cycling through dimensions until each leaf block contains at most 1024 points (configurable via maxPointsInLeafNode).

Here is what that looks like for a 2D geospatial index (latitude, longitude):

Collect all lat/lon pairs.
Sort by latitude and find the median. Split into northern and southern halves.
Sort each half by longitude and find the median. Split into eastern and western quadrants.
Repeat, alternating dimensions, until each block has 1024 or fewer points.

The result is a balanced tree where every leaf block contains a compact cluster of points in space. This clustering is what makes range queries fast: if a query range does not intersect a block's bounding box, the entire block is skipped without examining individual points.

Why BKD and Not R-Tree or Quadtree?

Lucene could have used an R-tree or quadtree. Both are proven spatial indexes. The BKD tree was chosen for several reasons:

Deterministic construction: BKD trees are built bottom-up from sorted data. No unpredictable splitting heuristics.
Perfect balance: Every leaf is at the same depth. Query time is predictable.
Excellent compression: Points are stored in sorted, packed blocks. Gaps are small, compression is high.
Cache-friendly: Block-based storage maps well to OS page cache and memory-mapped I/O.
Single-pass bulk loading: The entire tree is built from a single sorted scan, making it efficient for batch indexing.

On-Disk Format: .dii and .dim

The separation of index and data files allows Lucene to prune entire branches without reading point data.

BKD trees have their own file format in Lucene, separate from the inverted index. When you index point values, two files are created per segment:

.dii - The BKD index file. Contains the inner tree nodes, leaf block pointers, and bounding boxes.
.dim - The BKD data file. Contains the actual leaf blocks with packed points and document IDs.

Leaf Block Layout

Each leaf block in the .dim file stores:

The minimum and maximum values per dimension (bounding box).
The number of points in the block.
The points themselves, sorted and packed using delta encoding.
The document IDs associated with each point.

Because points within a leaf are spatially close, delta compression is extremely effective. A block of 1024 lat/lon pairs might compress to just a few kilobytes.

Inner Node Layout

The .dii file stores the tree structure:

Split dimension (which axis was used to split).
Split value (the median value at the split).
File pointer to the child block in .dim.
Bounding box (min/max for all dimensions of the subtree).

During a query, Lucene reads the .dii file to traverse the tree. It only touches .dim for leaf blocks that survive pruning. This separation means that for selective queries, only a tiny fraction of the point data is ever read from disk.

Indexing Numeric Data in Java

Lucene provides dedicated field types for point values. Here is how you index integers and longs:

import org.apache.lucene.document.*;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.FSDirectory;
import java.nio.file.Paths;

public class NumericIndexingExample {
    public static void main(String[] args) throws Exception {
        FSDirectory dir = FSDirectory.open(Paths.get("/tmp/numeric-index"));
        IndexWriterConfig config = new IndexWriterConfig();
        IndexWriter writer = new IndexWriter(dir, config);

        // Index a product with price and quantity
        Document doc = new Document();
        doc.add(new IntPoint("price", 299));
        doc.add(new StoredField("price", 299)); // for retrieval
        doc.add(new IntPoint("quantity", 150));
        doc.add(new StoredField("quantity", 150));

        writer.addDocument(doc);
        writer.close();
    }
}

Key classes:

IntPoint - 32-bit integer points
LongPoint - 64-bit integer points
FloatPoint - 32-bit floating point
DoublePoint - 64-bit floating point
LatLonPoint - 2D geospatial (latitude, longitude)
XYPoint - 2D Cartesian coordinates

Important: Point fields are not stored by default. If you need to retrieve the original value in search results, add a separate StoredField as shown above. Point fields are indexed-only and optimized for query filtering.

Range Queries: The BKD Pruning Advantage

Once your data is indexed as points, range queries become tree traversals instead of term expansions. Here is how you run a numeric range query:

import org.apache.lucene.search.*;
import org.apache.lucene.index.DirectoryReader;

public class RangeQueryExample {
    public static void main(String[] args) throws Exception {
        DirectoryReader reader = DirectoryReader.open(
            FSDirectory.open(Paths.get("/tmp/numeric-index"))
        );
        IndexSearcher searcher = new IndexSearcher(reader);

        // Find products priced between 100 and 500
        Query priceRange = IntPoint.newRangeQuery("price", 100, 500);

        TopDocs results = searcher.search(priceRange, 10);
        System.out.println("Found " + results.totalHits.value + " products");

        for (ScoreDoc sd : results.scoreDocs) {
            Document doc = searcher.doc(sd.doc);
            System.out.println("Price: " + doc.get("price"));
        }

        reader.close();
    }
}

How the Query Executes Internally

When IndexSearcher executes a point range query, the following happens:

Load the BKD tree root from .dii.
Check if the query range intersects the root's bounding box.
If yes, recurse into both children. If no, prune the entire subtree.
At leaf blocks, check the block's bounding box. If it intersects, read the .dim file and test individual points.
Collect all matching document IDs.

For a selective range on a large dataset, this prunes 90-99% of the tree. The query touches only a handful of leaf blocks. This is why BKD-based range queries are 10-100x faster than equivalent term-based range queries on large indexes.

Geospatial Indexing and Queries

BKD trees truly shine with geospatial data. Lucene's LatLonPoint stores latitude and longitude as a 2D point and uses the BKD tree for both bounding box and distance queries.

Indexing Geospatial Data

import org.apache.lucene.document.*;

public class GeoIndexingExample {
    public static void main(String[] args) throws Exception {
        FSDirectory dir = FSDirectory.open(Paths.get("/tmp/geo-index"));
        IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());

        // Index a location: Bangalore, India
        Document doc = new Document();
        doc.add(new LatLonPoint("location", 12.9716, 77.5946));
        doc.add(new StoredField("name", "Bangalore Office"));

        writer.addDocument(doc);
        writer.close();
    }
}

Distance Queries

import org.apache.lucene.search.*;

public class GeoQueryExample {
    public static void main(String[] args) throws Exception {
        DirectoryReader reader = DirectoryReader.open(
            FSDirectory.open(Paths.get("/tmp/geo-index"))
        );
        IndexSearcher searcher = new IndexSearcher(reader);

        // Find locations within 10 km of Bangalore center
        Query distanceQuery = LatLonPoint.newDistanceQuery(
            "location",
            12.9716,   // latitude
            77.5946,   // longitude
            10_000     // radius in meters
        );

        TopDocs results = searcher.search(distanceQuery, 50);
        System.out.println("Found " + results.totalHits.value + " nearby locations");

        reader.close();
    }
}

Geo3D: Accurate Spherical Distance

For applications requiring high-precision spherical distance (aviation, maritime, satellite tracking), Lucene provides Geo3DPoint. This model stores points on a 3D unit sphere using normalized Cartesian coordinates (x, y, z). Distance queries use great-circle math without the distortions of flat 2D projections.

import org.apache.lucene.spatial.util.Geo3DPoint;

// Index a point using geo3d
Document doc = new Document();
doc.add(new Geo3DPoint("location3d", 12.9716, 77.5946));

Geo3D uses the same BKD tree infrastructure but with 3D coordinates. The BKD tree prunes in 3D space, making complex polygon and distance queries efficient even at global scale.

BKD vs Inverted Index: A Concrete Comparison

The choice of data structure determines whether a query completes in milliseconds or minutes.

Let us put numbers to the claim. Consider an index of 10 million documents, each with a random integer field value between 0 and 1,000,000.

Approach	Index Size	Range Query (1000-5000)	Build Time
Term-based (StringField)	890 MB	340 ms	4.2 min
BKD Tree (IntPoint)	45 MB	4.2 ms	3.8 min

The BKD tree is 20x smaller and 80x faster for this workload. The gap widens as the index grows because the inverted index scales with the number of unique terms, while the BKD tree scales with the number of points.

For text data, the inverted index remains unbeatable. For numeric data, the BKD tree is the clear winner.

Production Tips and Best Practices

When to Use Points vs DocValues vs Stored Fields

Use Case	Use	Why
Range queries, filtering	`IntPoint`, `LongPoint`, `DoublePoint`	BKD tree is optimized for this
Sorting by numeric field	`SortedNumericDocValuesField`	DocValues are columnar and fast for sorting
Aggregations (facets)	`SortedNumericDocValuesField`	DocValues support efficient aggregation
Retrieve original value	`StoredField`	Only stored fields return in search results
Geospatial radius/box	`LatLonPoint`	2D BKD tree with haversine distance
High-precision geo	`Geo3DPoint`	3D spherical model without projection errors

Multi-Valued Points

A document can have multiple points in the same field. Lucene stores each point independently and associates them all with the same document ID. Range queries will match if any point in the document falls within the range.

// A store has multiple locations
doc.add(new LatLonPoint("locations", 12.9716, 77.5946));
doc.add(new LatLonPoint("locations", 13.0827, 80.2707));

Memory Considerations

BKD tree files are memory-mapped via MMapDirectory. The .dii index file is small and typically stays in the OS page cache. The .dim data file is larger but only the accessed leaf blocks are read into memory. For hot indexes, the entire tree may be cached by the OS, giving near-in-memory query performance without JVM heap pressure.

Choosing the Right Point Type

Use IntPoint for counts, quantities, IDs, and small ranges.
Use LongPoint for timestamps, large IDs, and financial amounts in cents.
Use DoublePoint for prices, measurements, and scientific values.
Use FloatPoint only when memory is extremely constrained and precision loss is acceptable.

Never use StringField or TextField for numeric data that will be queried with ranges. It is a common mistake that silently destroys performance at scale.

Conclusion

The BKD tree is one of Lucene's most important architectural innovations. It extends Lucene's capabilities from text search into the domains of numeric filtering, time-series analysis, and geospatial search - all while maintaining the same segment-based, immutable, memory-mapped design principles that make Lucene fast and reliable.

Understanding BKD trees helps you make better indexing decisions. It tells you why IntPoint exists, why LatLonPoint is the right choice for geo queries, and why you should never index numeric ranges as strings. The next time you build a search index with prices, dates, or coordinates, you will know exactly how Lucene makes those queries fast.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

How Polaris Locks Down Cloud Storage: IAM, Trust, and the Anatomy of a Secure Table Request

Prithvi S — Thu, 18 Jun 2026 00:33:07 +0000

Most data catalogs handle the easy part: tracking tables, columns, and schemas. The hard part is making sure that when Spark, Trino, or Flink asks to read a table, it gets exactly the right credentials for exactly the right files and nothing else. No long-lived keys floating around. No blanket access to entire buckets. No hoping that someone rotated the IAM policy last quarter.

Apache Polaris takes a fundamentally different approach. Instead of handing out persistent credentials to every engine and hoping for the best, Polaris mints short-lived, scoped credentials on demand. But before a single credential gets vended, Polaris has already done substantial security work: establishing trust relationships with your cloud provider, validating storage locations, enforcing a two-tier RBAC model, and making sure every request passes through multiple authorization layers.

Let me walk you through how Polaris secures cloud storage from the ground up, from the moment you create a catalog to the moment a query engine reads a single Parquet file.

The Setup: Establishing Trust Before the First Request

Before Polaris can vend credentials for S3, GCS, or Azure, it needs to know who it is talking to and prove that it has the right to request access on your behalf. This happens during catalog creation, and the details vary by cloud provider.

S3: ARN, External ID, and the Trust Triangle

For AWS, Polaris uses a cross-account IAM role assumption pattern. When you configure an S3-backed catalog, you provide an IAM Role ARN. Polaris then uses AWS STS AssumeRole to request temporary credentials for that role. But AWS does not hand out credentials blindly. The trust policy on your IAM role must explicitly allow Polaris to assume it, and Polaris supports an external ID for additional protection against the confused deputy problem.

Here is the trust flow:

You create an IAM role in your AWS account with a trust policy that allows Polaris to assume it
You provide the role ARN and optional external ID to Polaris during catalog creation
Polaris stores this configuration in the catalog metadata
When an engine requests table access, Polaris calls sts:AssumeRole with that ARN
AWS returns temporary credentials scoped to the role's permissions
Polaris further restricts those credentials to the specific table path before handing them to the engine

The external ID is a subtle but critical detail. Without it, a third party who knows your role ARN could potentially convince AWS to issue credentials to them instead of Polaris. The external ID acts as a shared secret between you and Polaris, ensuring that only your Polaris instance can assume the role.

GCS: Service Account Delegation

For Google Cloud Storage, Polaris uses service account impersonation. You create a dedicated Google Cloud service account with the minimal permissions needed for your catalog and grant Polaris permission to impersonate it. During catalog creation, you provide the service account email address. When credential vending is needed, Polaris uses the Google Cloud IAM API to generate short-lived OAuth 2.0 access tokens for that service account.

The key difference from S3 is that GCS tokens are OAuth-based rather than session-based, but the principle is the same: Polaris never stores long-lived credentials, and the engine never sees the service account's private key.

Azure: Tenant ID and Managed Identity

For Azure Blob Storage, Polaris connects via tenant ID and either a managed identity or service principal. You configure the tenant ID, client ID, and client secret (or use a managed identity) during catalog setup. Polaris then requests tokens from Azure AD and uses them to generate SAS (Shared Access Signature) tokens for the engine. These SAS tokens are time-bound and scoped to specific containers or blobs, providing fine-grained access control.

Allowed Locations: The Perimeter Guard

Regardless of the cloud provider, Polaris enforces an "allowed locations" policy on every catalog. When you create a catalog, you specify the base storage locations that the catalog is permitted to access. Polaris validates every storage path against this whitelist before vending credentials. If an engine somehow requests access to a path outside the allowed locations, Polaris blocks the request at the storage integration layer, before any cloud API calls are made.

This is a defense-in-depth measure. Even if an attacker compromises the RBAC layer and somehow gets a credential vending request approved, they still cannot access storage outside the catalog's configured locations. The allowed locations act as a hard perimeter around the catalog's data footprint.

The Two-Tier RBAC Wall: Identity and Permissions, Separated

Once storage is configured, the next security layer is Polaris's two-tier RBAC model. Most systems conflate "who you are" with "what you can do." Polaris separates them explicitly.

Principal Roles: Who You Are

Principals in Polaris are service accounts or users. Each principal gets assigned one or more principal roles. These roles define the principal's identity within the system: a data scientist, a pipeline service account, a monitoring agent. Principal roles are global and answer the question: "who is making this request?"

Catalog Roles: What You Can Do

Catalog roles define permissions. A catalog role is a collection of privileges like TABLE_READ_DATA, TABLE_WRITE_DATA, CATALOG_MANAGE_ACCESS, or CATALOG_MANAGE_CONTENT. These roles are scoped to specific catalogs and can be granted to principal roles.

The separation works like this: a data scientist principal might have the "Data Scientist" principal role. That principal role is granted the "Read-Only" catalog role on the "Production Analytics" catalog. The same principal role could also be granted the "Full Access" catalog role on the "Development" catalog. The principal's identity is consistent, but their permissions vary by catalog.

This two-tier model has practical security benefits. When someone leaves the team, you revoke their principal role assignments, and all their catalog access disappears immediately. When a catalog's sensitivity changes, you modify the catalog role's privileges without touching any principal definitions. The separation of concerns makes audits simpler and reduces the blast radius of access changes.

Privileges: The Permission Matrix

Polaris defines a comprehensive privilege hierarchy. Key data access privileges include:

TABLE_READ_DATA - read table data via SELECT
TABLE_WRITE_DATA - insert, update, delete, or merge table data
TABLE_READ_PROPERTIES - read table metadata and properties
TABLE_WRITE_PROPERTIES - modify table metadata
VIEW_READ_DATA - read view data
VIEW_WRITE_DATA - modify view definitions
CATALOG_MANAGE_ACCESS - grant or revoke roles and privileges
CATALOG_MANAGE_CONTENT - create, drop, or alter tables and namespaces

Privileges are enforced by PolarisAuthorizer, which evaluates the principal's catalog roles against the requested action on the target entity. If any catalog role grants the required privilege, the request proceeds. If none do, the request is rejected before any storage operations are considered.

The Anatomy of a Secure Table Request

Now that we understand the setup, let us trace a single table read request through Polaris's security layers. This is where all the pieces come together.

Step 1: Authentication

The query engine sends an Iceberg REST API request to Polaris, typically with a Bearer token or mutual TLS. Polaris validates the token against its configured identity provider. The result is a Principal object representing the authenticated caller.

Step 2: Principal Role Resolution

Polaris looks up the principal's assigned principal roles. For this example, let us say our principal has the "Data Scientist" principal role.

Step 3: Catalog Role Resolution

The request targets a specific catalog. Polaris looks up which catalog roles the principal's roles have been granted on this catalog. If the "Data Scientist" principal role has been granted the "Read-Only" catalog role on this catalog, that catalog role is collected.

Step 4: Privilege Check

The request asks to read table data, which requires TABLE_READ_DATA. Polaris checks if any of the resolved catalog roles grant this privilege. If not, the request is rejected immediately with a 403. No storage APIs are called, no credentials are vended, and no cloud costs are incurred.

Step 5: Entity Resolution

With the privilege check passed, Polaris resolves the target entity: the catalog, namespace, and table. It fetches the table metadata from the persistence layer, typically via AtomicMetaStoreManager and a JDBC backend. This metadata includes the table's location in cloud storage.

Step 6: Storage Configuration Lookup

Polaris retrieves the catalog's storage configuration: the S3 role ARN, GCS service account, or Azure tenant ID that was configured during setup. It also retrieves the allowed locations list for this catalog.

Step 7: Location Validation

Before any credential vending happens, Polaris validates that the table's storage location falls within the catalog's allowed locations. If the table is stored at s3://production-analytics/fact_orders/ and the allowed location is s3://production-analytics/, validation passes. If the table somehow resolved to s3://other-bucket/, Polaris would reject the request here, even though the RBAC check passed.

Step 8: Credential Vending

Now Polaris calls the cloud provider's credential API. For S3, it calls sts:AssumeRole with the catalog's role ARN. For GCS, it requests an OAuth token via the IAM API. For Azure, it generates a SAS token.

But here is the critical detail: Polaris does not just pass through whatever credentials the cloud provider returns. It restricts them further. The cloud credentials are scoped to the specific table path, not the entire catalog or bucket. And they are time-bound, typically to about 15 minutes (configurable).

For a read request, Polaris ensures the credentials are read-only. For a write request, it ensures they have write access. The principle of least privilege is enforced at the cloud credential level, not just the Polaris privilege level.

Step 9: Response

Polaris returns the Iceberg REST API response to the engine, including the short-lived credentials and the table metadata. The engine uses those credentials to read the actual data files from cloud storage. The credentials expire automatically, and the engine must return to Polaris for fresh credentials on subsequent requests.

Step 10: Audit and Revocation

Every step in this flow is auditable. Polaris logs the principal, the request, the privilege check result, the entity accessed, and the credential vending action. Because credentials are short-lived and Polaris is the single point of issuance, revocation is instantaneous. If a principal's access is revoked at step 2, any credentials they previously received will expire within minutes and cannot be renewed.

Federated Credentials: Extending Security Beyond Polaris (v1.3.0)

Polaris 1.3.0 introduced a significant security enhancement: federated credential vending. Previously, when Polaris managed an external catalog (like a Hive or Hadoop-backed catalog), the query engine would use the external catalog's credentials directly. Polaris acted as a metadata pass-through, but the actual storage access was governed by the external system's credential model.

With v1.3.0, Polaris can mint credentials for external catalogs just as it does for internal ones. This means:

Unified security model: Whether a table is in an internal Polaris-managed catalog or a federated external catalog, the credential vending flow is the same: RBAC check, location validation, scoped credentials, automatic expiration.
No credential leakage: Engines never see the external catalog's long-lived credentials. They only see the short-lived, scoped credentials that Polaris mints on their behalf.
Centralized audit: All credential access flows through Polaris, even for data that lives in external systems. This gives security teams a single point of observability.
Instant revocation: Revoking a principal's Polaris role immediately cuts off their access to federated data, without needing to touch the external system's IAM configuration.

This is a subtle but important shift. Polaris is evolving from a metadata catalog into a unified security control plane for all tabular data, regardless of where it lives.

Why This Matters: The Alternative Is Credential Chaos

Without Polaris's model, organizations typically fall into one of two traps:

Trap 1: The Shared Key. Every engine gets the same long-lived IAM key with broad bucket access. Rotation is painful, so it rarely happens. When someone leaves, the key stays the same. When an engine is compromised, the attacker has access to everything.

Trap 2: The IAM Sprawl. Every team maintains their own IAM roles, service accounts, and policies. A data scientist needs access to a table, so they open a ticket. Someone creates a role. The role gets attached to a service account. The service account gets shared. Six months later, nobody knows which roles are still needed, but nobody dares delete them. This is the classic IAM debt spiral.

Polaris avoids both traps by making the catalog the single point of access control. The cloud IAM roles are minimal: they only need to allow Polaris to assume them. Polaris handles the rest: identity, permissions, scoping, expiration, and audit. The IAM surface area shrinks dramatically, and the access model becomes comprehensible.

Performance and Security: Not a Trade-off

A common objection to credential vending is latency. If every table read requires a cloud API call to mint credentials, does that not slow things down? The answer is yes, but Polaris mitigates it aggressively.

Credential minting takes roughly 100-200 milliseconds per cloud API call. Polaris caches credentials keyed by the principal, catalog, table, and operation type. If the same engine requests the same table again within the cache window, Polaris returns the cached credentials without calling the cloud provider. The cache TTL is shorter than the credential expiration time, so there is no risk of serving expired credentials.

For most query patterns, the cache hit rate is high. Engines tend to read the same tables repeatedly during a query session, and batch reads for the same table are common. The practical overhead is minimal, and the security benefit is substantial.

Operational Security: Deploying Polaris Securely

Security is not just about the architecture; it is also about how you run it. Polaris provides several operational controls to keep the deployment secure:

TLS everywhere: The Iceberg REST API and all internal communications should run over TLS 1.2 or higher. Polaris supports certificate-based authentication for service-to-service communication.
Admin bootstrapping: The polaris-admin tool creates the initial principal and role assignments during first-time setup. This should be run in a secure environment, and the initial credentials should be rotated immediately after setup.
Docker and Helm: Polaris distributes Docker images and Helm charts. Security teams should scan these images, run them with non-root users, and restrict their network access to only the necessary ports and peers.
Persistence encryption: The polaris-relational-jdbc persistence layer should connect to a database with TLS encryption. The database itself should be encrypted at rest and protected by its own access controls.
Health and metrics endpoints: Polaris 1.3.0 standardized /q/health and /q/metrics endpoints. These should be exposed to monitoring systems but protected from public access, as they may reveal operational details.

Conclusion

Polaris approaches cloud storage security with a simple but powerful principle: the catalog should be the gatekeeper, not just the librarian. By combining cloud-native trust relationships, two-tier RBAC, location validation, short-lived credential vending, and comprehensive audit logging, Polaris creates a security model that is both strong and operable.

The next time you are evaluating a catalog for your data lake, ask not just "can it track my tables?" but also "can it keep my cloud credentials under control?" Polaris answers the second question with a resounding yes.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Distributed Scoring in OpenSearch: Why Your Results Aren't Perfectly Ranked

Prithvi S — Wed, 17 Jun 2026 15:48:13 +0000

Understanding the trade-off between speed and accuracy in distributed search and the one query parameter that fixes it.

You index a million documents, run a search, and the top result looks right. You scale to a billion documents across fifty shards, run the same search, and suddenly the ranking feels off. Not broken just subtly wrong. The best result is third. A mediocre result is first. You re-run the query and get a different order.

Welcome to distributed scoring in OpenSearch.

This is not a bug. It is a fundamental property of how OpenSearch and any distributed search engine built on Lucene computes relevance. The good news: it is fixable. The bad news: the fix costs you a round-trip. In this post, I will explain why scores diverge across shards, when it matters, and how to decide whether to pay the latency tax for perfect ranking.

The Assumption That Breaks Everything

Most engineers assume search ranking works like this:

Collect every document that matches the query
Score all of them using global statistics
Return the top N in exact rank order

That is how it works in a single-node database. OpenSearch does not work that way. OpenSearch is distributed by design an index is split into shards, shards live on different nodes, and each shard is an independent Lucene index with its own local statistics.

When you send a query, the coordinating node does not gather every matching document from every shard and score them centrally. That would be catastrophically slow at scale. Instead, OpenSearch uses a scatter-gather pattern with a critical optimization: scores are computed locally on each shard, using local term statistics.

How Query Execution Actually Works

Here is what happens when you run a match query against a multi-shard index:

Phase 1: Query Phase (Scatter)

The coordinating node sends the query to every shard that holds data for the index. Each shard executes the query independently against its own Lucene segments. For each document, it computes a relevance score using BM25 or your chosen similarity function.

BM25 depends on two key statistics:

Term Frequency (TF): How often the term appears in this document
Inverse Document Frequency (IDF): How rare the term is across the entire collection

Here is the problem: each shard only knows about its own documents. It computes IDF using local document frequencies, not global ones. A term that appears in 1% of documents globally might appear in 5% of documents on one shard and 0.1% on another. The shard with the 5% local frequency will score matching documents lower. The shard with the 0.1% frequency will score them higher.

The result? Two identical documents on different shards can receive different scores for the same query.

Phase 2: Merge and Fetch (Gather)

Each shard returns its top-K results (document IDs + scores) to the coordinating node. The coordinating node merges these shard-local rankings into a single global ranking. But it is merging rankings that were computed with inconsistent scoring baselines.

Finally, if the client requested full documents (_source), the coordinating node issues a fetch phase to retrieve the actual document bodies from the relevant shards.

The default search type QUERY_THEN_FETCH optimizes for speed. It assumes that for most use cases, "close enough" ranking is acceptable. And for many use cases, it is.

When Local Scoring Bites You

The ranking error is usually small and random. But there are conditions where it becomes material:

1. Skewed Shard Distributions

If your data is not uniformly distributed for example, time-based indices where recent shards have different term distributions than old shards local IDF diverges significantly. A query for "Kubernetes" on a shard full of 2024 DevOps logs will score differently than on a shard of 2019 monolith logs.

2. Small Shards with Rare Terms

With many small shards, rare terms have volatile local document frequencies. A term that appears in exactly one document globally might appear in zero documents on most shards and one document on a single shard. That lucky document gets an enormous IDF boost and shoots to the top.

3. High-Precision Ranking Requirements

E-commerce search, legal discovery, and academic search often need deterministic, reproducible ranking. If you are A/B testing ranking algorithms or comparing results across replicas, non-deterministic shard-local scoring introduces noise that masks real signal.

4. Routing Keys and Custom Distributions

If you use custom routing (e.g., routing all documents for tenant X to the same shard), you create isolated term-frequency bubbles. A term common in tenant X's documents is rare globally but frequent locally and local scoring does not know the difference.

The Fix: DFS_QUERY_THEN_FETCH

OpenSearch provides an alternative search type: DFS_QUERY_THEN_FETCH. The "DFS" stands for Distributed Frequency Search a pre-query phase that gathers global term statistics before executing the actual search.

Here is the flow:

DFS Phase: The coordinating node sends a lightweight request to all shards. Each shard returns its local term frequencies and document frequencies for the query terms. The coordinating node aggregates these into global statistics.
Query Phase: The coordinating node re-sends the query to all shards, this time accompanied by the global term statistics. Every shard now scores documents using the same IDF values.
Fetch Phase: Identical to the default retrieve full documents for the globally-merged top results.

The result: perfectly consistent scoring across shards. Identical documents on different shards receive identical scores. Ranking is deterministic. The A/B test noise disappears.

The Cost: One Extra Round-Trip

DFS_QUERY_THEN_FETCH adds a network round-trip before the query phase. For simple term queries on small clusters, this overhead is negligible a few milliseconds. For complex queries with many terms on large clusters, it can add tens of milliseconds.

More importantly, the DFS phase does not reduce total work. It redistributes it. The coordinating node must collect, merge, and forward statistics. If your cluster is already network-bound or the coordinating node is under CPU pressure, DFS can amplify the bottleneck.

Here is my practical guidance:

Scenario	Recommendation
Few large shards, uniform data, speed matters	Stick with QUERY_THEN_FETCH
Many small shards, skewed distributions, precision matters	Use DFS_QUERY_THEN_FETCH
Reproducible ranking required (A/B tests, audits)	Use DFS_QUERY_THEN_FETCH
Rare-term-heavy queries (specialized domains)	Use DFS_QUERY_THEN_FETCH
High-traffic, latency-sensitive, "good enough" ranking	Stick with QUERY_THEN_FETCH

How to Use It

In OpenSearch, you specify the search type at query time:

GET /my-index/_search?search_type=dfs_query_then_fetch
{
  "query": {
    "multi_match": {
      "query": "distributed scoring",
      "fields": ["title^3", "content"]
    }
  }
}

There is no cluster-wide default to change this. It is a per-query decision, which is the right design you pay the cost only when you need the precision.

A Deeper Realization: This Is Inherent to Distributed Search

The local-vs-global scoring problem is not an OpenSearch quirk. It is inherent to any distributed search system that shards data independently. Solr has the same trade-off. Elasticsearch had it before the fork. The only engines that avoid it entirely are those that do not shard at the indexing level or those that accept the latency cost of global scoring on every query.

What OpenSearch gives you is transparency and control. You can see the behavior, measure it, and opt into the fix when your use case demands it. That is the difference between a system that hides complexity and one that exposes it with clear knobs.

What I Have Learned Maintaining Search Plugins

As a maintainer of the OpenSearch dashboards-search-relevance plugin, I have spent a lot of time in the scatter-gather path. One thing that surprised me early on: the concurrent segment search improvements in OpenSearch 3.0 do not change this trade-off. They parallelize execution within a shard, but each shard still operates with local statistics unless you request DFS.

Another surprise: most ranking anomalies reported as "bugs" are actually local IDF effects. Before you debug your analyzer or your BM25 parameters, check whether dfs_query_then_fetch makes the anomaly disappear. If it does, you have found your culprit and your fix.

Summary

OpenSearch scores documents using local shard statistics, not global ones
This causes ranking inconsistencies that are usually small but can be significant with skewed data or rare terms
DFS_QUERY_THEN_FETCH adds a pre-query phase to gather global statistics, enabling consistent cross-shard scoring
The cost is one extra round-trip acceptable for precision-critical workloads, unnecessary for speed-critical ones
This is not a bug. It is a documented, controllable trade-off in distributed search design

The next time your search results feel subtly wrong, ask yourself: am I seeing the true global ranking or fifty local opinions merged together?

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

OpenSearch Query Phase vs Fetch Phase: What Actually Happens on Each Shard

Prithvi S — Wed, 17 Jun 2026 15:43:40 +0000

Understanding the two-phase search architecture that powers every OpenSearch query

Every time you run a search query in OpenSearch, the engine executes a carefully orchestrated two-phase dance across your cluster. The query phase finds the right documents. The fetch phase retrieves their contents. Understanding how these phases work - and how they interact - is the difference between a search that returns in milliseconds and one that times out.

As someone who has spent the last year maintaining the search-relevance plugin and contributing to OpenSearch core, I have seen how often engineers conflate these two phases. They optimize the wrong things, tune the wrong parameters, and wonder why their queries still lag. This post is what I wish I had read when I started.

Why Two Phases?

OpenSearch distributes data across multiple shards, often on multiple nodes. When you search, the coordinating node must ask every shard to participate. But there is a critical insight: finding the most relevant documents is lighter than retrieving their full contents.

If OpenSearch retrieved the full _source of every document from every shard just to find the top 10 results, the network bandwidth and I/O would be catastrophic. Instead, the engine splits the work:

Phase 1 - Query: Each shard finds its local top-N results, returning only lightweight metadata (doc IDs and scores).

Phase 2 - Fetch: The coordinating node identifies the true global top-N, then asks only the relevant shards for the full document contents.

This is the scatter-gather pattern in practice. The query phase is the scatter. The fetch phase is the gather.

Phase 1: The Query Phase

What Happens on Each Shard

When a search request arrives at the coordinating node, it is forwarded to every shard in the index (or the relevant shards if routing is specified). Here is what each shard does:

Parse the query - The query DSL is converted into a Lucene Query object.
Execute against segments - The query runs against all Lucene segments in the shard. With concurrent segment search (enabled by default in OpenSearch 3.0), these segments may be searched in parallel across CPU slices.
Score documents - BM25 scoring is computed for each matching document. Term frequency and inverse document frequency are calculated locally within that shard.
Collect top-K - Using a priority queue, the shard keeps only the top N results by score (where N is from + size).
Return metadata - The shard returns only document IDs, scores, and sort values - never the full _source.

The Local Scoring Problem

Here is a subtle but important detail: BM25 scores are computed using local shard statistics. Each shard calculates term frequency and document frequency based only on the documents it holds. This means a term that is rare globally but common on one shard will be scored differently there.

For most use cases, this approximation is good enough. But when you need truly accurate global scoring, OpenSearch offers dfs_query_then_fetch:

GET /my-index/_search?search_type=dfs_query_then_fetch
{
  "query": {
    "match": { "title": "distributed search" }
  }
}

This adds a pre-query round-trip: the coordinating node fetches term statistics from all shards, then broadcasts them back so every shard uses global IDF. The trade-off? One extra network hop for scoring accuracy.

Aggregation Execution

If your query includes aggregations, they execute during the query phase too. Each shard builds its local aggregation buckets, then the coordinating node merges them. This is why deeply nested aggregations on high-cardinality fields can be so expensive - every shard is building and serializing large bucket structures.

Phase 2: The Fetch Phase

The Document Retrieval Problem

After the query phase completes, the coordinating node has a sorted list of global top-N results. But it only has document IDs and scores. If the client requested _source fields, highlighting, or script fields, the coordinating node must now fetch those.

What Happens During Fetch

Merge and rank - The coordinating node merges all shard results, re-sorting by score to produce the true global top-N.
Request documents - For each of the top-N document IDs, the coordinating node asks the owning shard for the full document content.
Retrieve stored fields - Each shard fetches the requested fields from its stored field cache or disk.
Apply highlighting - If requested, the highlighter (unified, plain, or fast vector) generates match snippets by re-analyzing the relevant text.
Return results - The final documents are assembled and returned to the client.

The Hidden Cost

The fetch phase is where hidden costs emerge. If you request:

Large _source documents (10KB+ each)
Highlighting on large fields
Script fields that execute at fetch time
Deep pagination (from: 10000, size: 10)

...the fetch phase can dominate your query latency. The query phase might finish in 5ms, but the fetch phase takes 200ms because it is retrieving and highlighting 10,000 documents just to return 10.

How `size` and `from` Control Everything

The size and from parameters are the primary levers for controlling both phases:

GET /my-index/_search
{
  "from": 0,
  "size": 10,
  "query": { "match_all": {} }
}

Query Phase Impact

During the query phase, each shard must return from + size results. If you have 5 shards and request from: 100, size: 10, every shard must find and return its top 110 results. The coordinating node then merges 550 results to find the true global top 10.

With 10 shards and from: 10000, size: 10? Every shard returns 10,010 results. The coordinating node merges 100,100 results. This is why deep pagination is so expensive in OpenSearch.

Fetch Phase Impact

The fetch phase only retrieves documents for the final returned set. So with size: 10, only 10 documents are fetched, regardless of from. But the query phase already paid the price of finding and ranking all those candidate documents.

The `terminate_after` Escape Hatch

For use cases where you only need approximate results, terminate_after tells each shard to stop after finding N documents:

GET /my-index/_search
{
  "terminate_after": 1000,
  "query": { "match": { "status": "active" } }
}

This dramatically speeds up the query phase but means you might miss better-scoring documents that were found later. Use it for analytics, not for user-facing search.

Performance Tuning: Query Phase

1. Reduce Shard Count When Possible

More shards = more work in the query phase. If you have 50 shards for a 10GB index, you are creating unnecessary scatter-gather overhead. Aim for shards in the 20-50GB range.

2. Use Routing for Targeted Queries

If your queries always filter by a specific field (like user_id), use custom routing to ensure all of a user's documents land on the same shard:

POST /my-index/_doc?routing=user_123
{
  "user_id": "user_123",
  "content": "..."
}

Now searches with routing=user_123 hit only one shard. The query phase just got 10x faster.

3. Leverage Concurrent Segment Search

OpenSearch 3.0 enables concurrent segment search by default. For long-running queries with aggregations, this parallelizes segment processing within each shard. Ensure your CPU has cores to spare, or the overhead of thread coordination may not be worth it.

4. Be Careful with Aggregations

Terms aggregations on high-cardinality fields (like user_id or ip_address) force each shard to build massive bucket arrays. If you do not need exact counts, use shard_size to limit the buckets each shard returns:

{
  "aggs": {
    "top_users": {
      "terms": {
        "field": "user_id",
        "size": 10,
        "shard_size": 100
      }
    }
  }
}

Performance Tuning: Fetch Phase

1. Use `docvalue_fields` Instead of `_source`

If you only need specific fields that are stored as doc values (keyword, numeric, date, boolean), request them directly instead of parsing the full _source:

GET /my-index/_search
{
  "query": { "match_all": {} },
  "docvalue_fields": ["status", "created_at"]
}

Doc values are columnar and much faster to retrieve than the full JSON _source.

2. Disable `_source` if You Do Not Need It

For purely analytic queries where you only care about aggregation results, explicitly disable _source retrieval:

GET /my-index/_search
{
  "_source": false,
  "aggs": {
    "status_counts": {
      "terms": { "field": "status" }
    }
  }
}

This skips the fetch phase entirely for hits.

3. Avoid Highlighting Large Fields

Highlighting requires re-analyzing the matched text. If your documents have 100KB text fields and you highlight them, the fetch phase will grind. Consider:

Using the fast_vector_highlighter with term_vector storage (trades index size for fetch speed)
Limiting fragment size and count
Storing a truncated summary field specifically for highlighting

4. Use Search After for Deep Pagination

Instead of from: 10000, use search_after with a point-in-time (PIT) for efficient deep pagination:

GET /my-index/_search
{
  "size": 10,
  "pit": { "id": "my-pit-id", "keep_alive": "1m" },
  "search_after": [1625097600000, "doc_123"],
  "sort": [{ "created_at": "asc" }, { "_id": "asc" }]
}

This avoids the from + size explosion in the query phase and is the recommended approach for scrolling through large result sets.

Real-World Example: Debugging a Slow Query

Let me walk through how I debugged a slow search in production. The query was timing out at 30 seconds:

GET /logs-*/_search
{
  "from": 0,
  "size": 50,
  "query": { "match": { "message": "error" } },
  "highlight": { "fields": { "message": {} } },
  "aggs": {
    "by_service": { "terms": { "field": "service.name", "size": 100 } }
  }
}

The problem: 100 shards, each with 5GB of logs. The query phase was fast (BM25 on "error" is selective). But the fetch phase was retrieving 50 full log messages, each 50KB, and highlighting them across 100 shards.

The fix:

Reduced size from 50 to 10 (users do not need 50 highlighted logs)
Added "summary": { "type": "text" } field with 500-char limit for highlighting
Switched highlighting to the summary field only
Added "shard_size": 20 to the aggregation

Query time dropped from 30 seconds to 800ms. The query phase was never the problem - the fetch phase was.

The Key Insight

Most OpenSearch performance problems are not query problems. They are fetch problems. The query phase is highly optimized - Lucene is incredibly fast at finding and scoring documents. The fetch phase is where your data model, field sizes, and retrieval strategy matter.

When you are debugging slow searches, ask yourself:

Is the query phase slow? (Check with _source: false and no aggregations)
Is the fetch phase slow? (Compare with and without _source retrieval)
Am I retrieving too many documents? (Reduce size)
Am I highlighting too much text? (Use dedicated summary fields)
Am I forcing deep pagination? (Use search_after)

Conclusion

The two-phase search architecture in OpenSearch is not an implementation detail - it is the fundamental design that makes distributed search feasible. The query phase is about finding and ranking. The fetch phase is about retrieving and formatting. They have different performance characteristics, different tuning parameters, and different failure modes.

Understanding this split lets you diagnose problems faster, tune the right parameters, and build searches that scale. The next time a query is slow, do not just stare at the query DSL. Ask: is this a query phase problem, or a fetch phase problem?

The answer will guide you to the right fix.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Vector Search in Elasticsearch: From Keywords to Meaning - Building Semantic Search and RAG Pipelines

Prithvi S — Wed, 17 Jun 2026 15:40:14 +0000

You type "k8s deployment troubleshooting" into your documentation search. The top result is a page about Kubernetes architecture that never mentions the word "troubleshooting." It is exactly what you need. BM25 would have missed it entirely.

This is the promise of vector search: finding documents by meaning, not just matching words. In 2025 and 2026, vector search has moved from niche ML engineering to a core Elasticsearch capability. If you are building search for AI applications - RAG pipelines, semantic Q&A, recommendation systems - understanding how Elasticsearch handles vectors is no longer optional.

I have spent the past year building RAG pipelines at Cloudera, and I have learned that vector search is powerful but easy to misuse. This post covers what works, what does not, and how to implement it in production.

Why Vector Search Matters (And When It Does Not)

BM25, which we covered in a previous post, is brilliant at matching exact terms. But it is fundamentally lexical. It does not understand that:

"k8s" and "kubernetes" are the same thing
"docker container" and "containerization" are related concepts
"out of memory error" and "heap exhaustion" describe the same problem

Vector search solves this by converting text into high-dimensional numerical vectors (embeddings) where semantically similar content lives close together in vector space. A query for "k8s deployment troubleshooting" gets embedded into a vector, and Elasticsearch finds the nearest document vectors - even if they do not share a single keyword.

But vector search is not a replacement for BM25. It is a complement.

BM25 is faster, requires no ML infrastructure, and excels at exact-term matching. Vector search is slower, requires embedding models, and shines at conceptual similarity. The best search systems in 2026 use both.

How Elasticsearch Stores and Indexes Vectors

Elasticsearch introduced the dense_vector field type in version 7.x and has dramatically improved it through 8.x and into 2026. Here is how it works under the hood.

The dense_vector Field Type

A dense vector is simply an array of floating-point numbers. A 768-dimensional embedding from a model like E5 looks like this:

[0.023, -0.156, 0.089, ..., 0.041]  // 768 numbers total

In your index mapping:

PUT /products
{
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "description": { "type": "text" },
      "description_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

Key parameters:

dims: The vector dimension (must match your embedding model)
index: Whether to build an ANN index (set to true for search, false if only storing)
similarity: Distance metric - l2_norm (Euclidean), dot_product, or cosine

HNSW: The Algorithm Behind Approximate Search

Elasticsearch uses HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor (ANN) search. HNSW builds a multi-layer graph where:

Layer 0 contains all vectors with dense local connections
Higher layers contain a subset of vectors with longer-distance connections
Search starts at the top layer, greedily navigates down, then searches locally at layer 0

HNSW is fast (sub-10ms for million-vector indexes) but approximate. It may miss the true nearest neighbor in exchange for speed. You can tune this trade-off:

PUT /products
{
  "mappings": {
    "properties": {
      "description_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine",
        "index_options": {
          "type": "hnsw",
          "m": 16,
          "ef_construction": 100
        }
      }
    }
  }
}

m: Number of bi-directional links per node (higher = more accurate, more memory)
ef_construction: Search depth during index building (higher = better graph quality, slower indexing)

For query-time accuracy tuning, use num_candidates:

GET /products/_search
{
  "knn": {
    "field": "description_vector",
    "query_vector": [0.023, -0.156, ...],
    "k": 10,
    "num_candidates": 100
  }
}

num_candidates is how many vectors Elasticsearch considers before returning the top k. Higher values improve recall but increase latency. A common rule: num_candidates should be 10x k for good recall.

Building a RAG Pipeline with Elasticsearch

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in private data. The pipeline looks like this:

Document -> Chunk -> Embed -> Index -> Query -> Retrieve -> Generate

Here is how to implement it in Elasticsearch.

Step 1: Chunk Your Documents

LLMs have context limits, so long documents must be split into chunks. A common approach:

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = text_splitter.split_text(long_document)

Chunk size depends on your embedding model. E5 and BGE models typically use 512 tokens. OpenAI text-embedding-3-large supports up to 8192 tokens.

Step 2: Generate Embeddings

You need an embedding model. Options in 2026:

Model	Type	Dimensions	Best For
ELSER v2	Sparse (learned)	~2000 terms	Built-in, no external service
multilingual-e5-large	Dense	1024	Cross-lingual, high quality
BGE-large-en-v1.5	Dense	1024	Open source, competitive
OpenAI text-embedding-3-large	Dense	3072	Highest quality, API cost

For dense vectors with a local model:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(intfloat/multilingual-e5-large)
embeddings = model.encode(chunks, normalize_embeddings=True)

Important: Use normalize_embeddings=True if you are using dot_product similarity. Elasticsearch can then skip the normalization step internally for faster searches.

Step 3: Index Documents with Vectors

POST /_bulk
{ "index": { "_index": "knowledge_base", "_id": "doc_1_chunk_0" } }
{ "content": "To troubleshoot Kubernetes deployments...", "source_doc": "k8s_guide.pdf", "category": "devops", "content_vector": [0.023, -0.156, ...] }

Step 4: Search

GET /knowledge_base/_search
{
  "knn": {
    "field": "content_vector",
    "query_vector": [0.041, 0.089, ...],
    "k": 5,
    "num_candidates": 50
  }
}

The query vector is the embedding of the user question: "How do I fix a Kubernetes deployment that will not start?"

Hybrid Search: The Production-Ready Approach

Pure vector search has a problem: it misses exact matches. If a user searches for "Error code 503," a vector search might return documents about "server errors" in general but miss the exact troubleshooting page for HTTP 503.

The solution is hybrid search: run BM25 and kNN in parallel, then merge results.

Elasticsearch 8.15+ provides the retrievers API for this:

GET /knowledge_base/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "k8s deployment troubleshooting",
                "fields": ["content", "title"]
              }
            }
          }
        },
        {
          "knn": {
            "field": "content_vector",
            "query_vector": [0.041, 0.089, ...],
            "k": 10,
            "num_candidates": 100
          }
        }
      ],
      "rank_constant": 60,
      "window_size": 50
    }
  }
}

RRF (Reciprocal Rank Fusion) combines rankings without normalizing scores (which are incomparable across BM25 and cosine similarity). The formula is simple:

rrf_score = sum(1 / (rank + k))

Where k (rank_constant, default 60) prevents top ranks from dominating. Documents that rank well in both retrievers bubble to the top.

This is the architecture behind modern RAG systems. BM25 ensures exact matches surface. Vector search ensures conceptual matches surface. RRF merges them intelligently.

Metadata Filtering + Vector Search: The Killer Feature

Standalone vector databases (Pinecone, Weaviate) are great at pure vector search. But Elasticsearch has an advantage: you can combine vector search with the full power of Elasticsearch filtering, aggregations, and text search in a single query.

Example: Find semantically similar products, but only in the "electronics" category, with price under $500, and in stock:

GET /products/_search
{
  "knn": {
    "field": "description_vector",
    "query_vector": [0.041, 0.089, ...],
    "k": 10,
    "num_candidates": 100,
    "filter": {
      "bool": {
        "must": [
          { "term": { "category": "electronics" } },
          { "range": { "price": { "lte": 500 } } },
          { "term": { "in_stock": true } }
        ]
      }
    }
  }
}

Elasticsearch applies the filter during the HNSW graph traversal (post-filtering), so only matching vectors are considered. This is faster than retrieving vectors and filtering afterward.

This pattern - semantic similarity + structured filters - is why many teams choose Elasticsearch over dedicated vector databases. You get vectors AND the query DSL you already know.

ELSER: Built-In Semantic Search Without External Models

Not every team wants to run an embedding model. Elasticsearch provides ELSER (Elastic Learned Sparse EncodeR), a built-in model that generates sparse vectors using term expansion.

ELSER works differently from dense vectors:

It expands text into learned tokens (not just the words in the document)
It produces sparse vectors (mostly zeros, ~2000 non-zero terms)
It is included with Elasticsearch - no external API or model hosting needed
It runs on the Elasticsearch cluster using the inference processor

PUT /_ingest/pipeline/elser_pipeline
{
  "processors": [
    {
      "inference": {
        "model_id": ".elser_model_2",
        "input_output": [
          { "input_field": "content", "output_field": "content_embedding" }
        ]
      }
    }
  ]
}

ELSER v2 (released in 2024) is competitive with dense embedding models for English text. For multilingual or domain-specific content, custom dense models still win. But for teams that want semantic search with zero ML infrastructure, ELSER is the fastest path.

Performance: What to Expect in Production

Vector search is not free. Here is what you need to plan for.

Memory

Vectors are memory-hungry. A single 768-dimensional float32 vector uses:

768 dimensions * 4 bytes = 3 KB per vector

One million vectors = 3 GB. Plus HNSW graph overhead (roughly 2x the vector memory). For 10 million vectors at 768 dimensions, expect 30-60 GB of memory.

Mitigations:

Use int8 quantization (available in Elasticsearch 8.13+): 768 dimensions * 1 byte = 768 bytes per vector. 4x memory reduction with minimal quality loss.
Reduce dimensions: Some models (like OpenAI text-embedding-3-small) support dimension truncation. 256 dimensions often perform nearly as well as 768.
Shard sizing: Keep vector indices on fewer, larger shards to reduce per-shard overhead.

Indexing Throughput

The bottleneck is usually embedding generation, not Elasticsearch indexing. A local GPU can generate 100-500 embeddings/second. CPU inference might manage 10-50/second.

Mitigations:

Batch embedding requests (most APIs support batching)
Use GPU inference if volume is high
Consider ELSER if you want to skip external embedding entirely

Query Latency

ANN search with HNSW is fast but not as fast as BM25. Expect:

BM25: 1-5 ms
HNSW kNN (1M vectors): 5-20 ms
HNSW kNN (10M vectors): 10-50 ms
Hybrid (BM25 + kNN + RRF): 15-60 ms

For user-facing search, this is usually acceptable. For high-throughput batch pipelines, consider pre-filtering with metadata to reduce the vector search space.

Five Common Vector Search Pitfalls

1. Using Vectors When BM25 Would Suffice

If your users search by exact product SKUs, error codes, or names, vector search adds latency and complexity with no benefit. Start with BM25. Add vectors when you see queries where keyword matching fails.

2. Wrong Similarity Metric

cosine: Best for semantic similarity when vector magnitude does not matter (most text embeddings)
dot_product: Best when vectors are normalized and you want speed (skip the cosine calculation)
l2_norm: Best when vector magnitude carries signal (less common for text)

Using l2_norm with unnormalized text embeddings will give poor results. Check your model documentation.

3. Forgetting That ANN is Approximate

HNSW trades recall for speed. With default settings, expect 95-99% recall@10 (the true top-10 result is in the returned top-10 95-99% of the time). If your use case requires 100% recall, use exact brute-force search (index: false with script_score) or increase num_candidates significantly.

4. Not Monitoring Recall Metrics

Track recall@k in production. If it drops below your threshold, increase num_candidates or ef_construction. Do not deploy vector search without measuring whether it finds the right documents.

5. Ignoring the Index Refresh Interval

Vector fields are indexed during the refresh cycle, just like text. If you index documents and search immediately, you might not find them. For real-time RAG where documents are ingested and immediately queried, ensure your refresh interval is appropriate (default 1s, or use ?refresh=true for testing).

When to Use What: A Decision Framework

Use Case	Primary Approach	Secondary/Tuning
Exact keyword search (SKUs, codes)	BM25 only	No vectors needed
Semantic Q&A ("how do I...")	Dense vectors + kNN	Hybrid with BM25 for exact matches
Product search with filters	Hybrid (BM25 + kNN + metadata filters)	RRF for ranking, filters for pruning
Cross-lingual search	multilingual-e5 or BGE dense vectors	BM25 for exact term fallback
Zero-ML-infrastructure team	ELSER sparse vectors	Built-in inference, no external model
High-volume log search	BM25 with filters	Vectors only for semantic anomaly detection
Documentation/knowledge base	Hybrid search (RRF)	Vectors for conceptual, BM25 for exact

Conclusion

Vector search in Elasticsearch has matured from an experimental feature to a production-ready capability. The combination of dense vectors, HNSW indexing, hybrid search with RRF, and metadata filtering makes Elasticsearch a compelling platform for semantic search and RAG pipelines.

The key takeaways:

Vectors complement BM25, they do not replace it. Hybrid search with RRF consistently outperforms either approach alone.
HNSW is fast but approximate. Tune num_candidates and monitor recall@k for your use case.
Memory is the main constraint. Plan for 3+ GB per million 768-dimensional vectors. Use quantization or dimension reduction if needed.
ELSER is the fastest path to semantic search. If you cannot run external embedding models, start with ELSER v2.
Metadata filtering is Elasticsearch killer advantage. No standalone vector database matches the query flexibility of the Elasticsearch DSL.

The next time someone asks whether Elasticsearch can handle AI-powered search, the answer is yes - and it can do so while preserving everything that makes Elasticsearch powerful: distributed scale, rich querying, and operational maturity.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Query Sets in Search Relevance: Designing Representative Test Queries That Actually Matter

Prithvi S — Wed, 17 Jun 2026 15:34:58 +0000

You have spent weeks tuning your OpenSearch cluster. You have optimized analyzers, tweaked BM25 parameters, and maybe even added vector search. But how do you know if any of it actually helps users find what they want?

Here is the uncomfortable truth: most search quality problems start before you ever run an experiment. They start with your query set.

A query set is a curated collection of search queries that represent real user behavior. In OpenSearch Search Relevance, it is the foundation of every evaluation pipeline. Get it wrong, and your metrics are meaningless. Get it right, and you have a reliable compass for every tuning decision you make.

This post is about building query sets that actually matter. Not textbook examples. Not synthetic data. Real, representative queries that tell you whether your search is working.

Why Query Sets Are the Most Underrated Component of Search Evaluation

Search engineers love to talk about algorithms. BM25 vs DFR vs vector search. Query parsers. Boosting strategies. But the best scoring algorithm in the world cannot save you from a bad query set.

Think about it: your metrics (nDCG, precision, recall, MRR) are computed against a query set. If that query set does not represent what your users actually search for, you are optimizing for the wrong thing. You might chase a 0.05 nDCG improvement on synthetic queries while your real users cannot find basic products.

I have seen teams run elaborate A/B tests, publish glowing metric dashboards, and still get complaints about search quality. The disconnect? Their query set was 50 hand-picked queries from product demos, not 500 real queries from production logs.

OpenSearch Search Relevance treats query sets as first-class citizens. They are stored as documents in an internal index, versioned, reusable across experiments, and designed to be the starting point of every evaluation. But the plugin does not tell you how to build a good query set. That is on you.

What Makes a Query Set "Representative"?

A representative query set mirrors your actual user population. It captures the full distribution of what people search for, not just what you think they search for.

Here is what that means in practice.

Cover the Query Distribution

Real user query logs follow a power law. A small number of queries (the "head") account for most of the volume. A larger middle section (the "torso") covers moderate-frequency queries. The long tail contains thousands of rare, specific queries.

Your query set should sample from all three regions. If you only pick head queries, you miss edge cases. If you only pick tail queries, your metrics are noisy and your experiments take forever. A good rule of thumb: allocate 30% head, 40% torso, 30% tail.

Head queries are your bread and butter. "iphone 15" or "running shoes." These are high-volume, often generic, and usually competitive. If your search fails here, you are losing the most users.

Torso queries are where differentiation happens. "waterproof running shoes women" or "iphone 15 pro max 256gb blue." These show intent and are common enough to matter but specific enough to test relevance depth.

Tail queries are the real test. "shoes for plantar fasciitis under 100 dollars" or "iphone 15 case compatible with magSafe wallet third party." These are rare individually but collectively represent a significant portion of user needs. They also expose gaps in your catalog coverage, synonym handling, and query understanding.

Match Intent Diversity

Users search with different goals. Navigational queries aim to reach a specific page or product. Informational queries seek knowledge or comparisons. Transactional queries signal immediate purchase intent.

A query set that only tests transactional intent will overestimate your search quality for shoppers but miss problems researchers face. Mix intent types intentionally. The exact ratio depends on your product, but ignoring any category is a blind spot.

Include Known Problem Queries

Every search system has queries that are known to be problematic. Maybe they return zero results. Maybe they return irrelevant results. Maybe they are ambiguous and your current ranking is arbitrary.

These queries are gold. They are the ones where improvement is most visible and most valuable. Do not exclude them because they make your baseline metrics look bad. Include them specifically because they tell you where to focus your tuning efforts.

Avoid Synthetic Queries

There is a temptation to write queries you think users should ask. Resist it. "Best product under $50 with free shipping and 4.5 stars" is not a real query. It is a feature list pretending to be a query.

Real queries are messy. They have typos, abbreviations, brand names, model numbers, and half-remembered details. "nike air max red size 10" is a real query. "Comfortable athletic footwear with visible air cushioning technology" is not. Your query set should reflect messy reality.

Query Set Size: The Statistical Significance vs. Cost Trade-off

How many queries do you need? It depends on what you are trying to do.

Minimum Viable: 50 Queries

Fifty queries is enough for rough directional signal. You can tell if a change is catastrophic or promising. But you cannot reliably detect small improvements. The variance is too high. A single bad query can swing your aggregate nDCG by 0.1.

Use 50-query sets for early prototyping, debugging, or smoke testing. Do not use them for production go/no-go decisions.

The Sweet Spot: 100-500 Queries

This is where most teams should live. One hundred queries gives you enough statistical power to detect meaningful differences. Five hundred gives you confidence without drowning in judgment costs.

With 100-500 queries, you can segment by query type, compute per-query metrics, and identify specific patterns. You might discover that your change improves navigational queries by 15% but hurts informational queries by 8%. That is actionable insight you cannot get with 50 queries.

Large Scale: 1000+ Queries

Thousand-query sets are for mature search products with dedicated relevance teams. They give you statistical confidence, enable fine-grained segment analysis, and support long-term trend tracking.

The downside is cost. Every query needs judgments, and judgments are expensive. Human judgments require time and money. Even implicit judgments (click-through data) need volume and infrastructure. A 1000-query set with 10 results per query is 10,000 judgments. Scale accordingly.

In OpenSearch Search Relevance, query sets are stored as indexed documents. The plugin executes every query in the set against your configured searches, collects results, and feeds them into judgment collection. Larger query sets mean longer experiment runtimes. Plan for it.

Where to Source Your Queries

The best query sets come from real user data. Here are the most common sources and their trade-offs.

Search Query Logs

If you have logging infrastructure, this is the gold standard. Real queries from real users in your actual system. No simulation, no guessing.

Filter your logs by time period (last 30-90 days), remove bots and internal traffic, deduplicate near-identical queries, and sample across the frequency distribution. You want the query, not the count. One query in your set represents thousands of actual searches.

Analytics Tools

Google Analytics, Amplitude, Mixpanel, or similar tools often capture search queries. Export them, filter them, and use them. The advantage is that analytics tools often include outcome data (did the user convert? did they bounce?), which helps you prioritize which queries matter most.

Customer Support Tickets

Support tickets and help desk queries reveal where search is failing. "I searched for X but could not find Y" is a perfect query for your set. These queries are often high-intent, high-frustration, and underrepresented in normal logs because users give up rather than file tickets.

Competitor Analysis

For new products without query logs, analyze what users search for on competitor sites or in related forums. Reddit, Quora, and industry-specific communities are rich sources of how people actually talk about your domain. This is less precise than your own logs but better than synthetic queries.

Manual Curation

Some manual curation is always necessary. You need to fill gaps, add edge cases, and ensure coverage. But manual curation should complement data-driven sampling, not replace it. Think of it as seasoning, not the main ingredient.

Query Set Maintenance: Fighting Query Drift

User behavior changes. Seasonal trends, product launches, marketing campaigns, and cultural shifts all alter what people search for. A query set from January is not representative in December.

Set a schedule for query set updates. Monthly reviews for fast-moving products. Quarterly for stable domains. At minimum, annually. When you update, compare the old and new distributions. If the head queries have shifted significantly, your metrics from old experiments may not generalize.

In OpenSearch Search Relevance, query sets are reusable and versioned. You can keep old query sets for historical comparison while running new experiments on updated sets. This is powerful for long-term trend analysis. You can answer questions like: "Has our search quality improved for holiday queries year over year?"

The Relationship Between Query Sets and Experiments

Once you have a query set, it becomes the input to an experiment. Here is how the flow works in OpenSearch Search Relevance.

You create an experiment and link it to a query set. The plugin iterates through each query in the set, executes it against your configured search (or multiple searches for comparison), and collects the results. These results are then judged (by humans or implicit signals) and metrics are computed.

The query set size directly impacts experiment runtime. A 500-query set with 2 search configurations being compared means 1000 query executions. If each query hits a large index with complex aggregations, this can take significant time. The plugin handles this automatically, but you should plan your experiment windows accordingly.

Query sets also determine the granularity of your insights. A well-structured query set lets you segment results by query type, frequency, or intent. You might discover that your new analyzer improves tail queries by 20% but does nothing for head queries. That is a valuable, specific insight you would miss with a poorly designed query set.

Practical Tips for OpenSearch Search Relevance Users

If you are using the OpenSearch Search Relevance plugin, here are specific tips for query set management.

Store metadata with your query sets. The plugin supports name and description fields. Use them. "Holiday 2025 product queries, 250 queries, sourced from Nov-Dec logs" is more useful than "Query Set 1."

Start small and grow. Begin with a 50-query smoke test to validate your experiment setup. Once you trust the pipeline, expand to 200-500 queries for decision-making.

Segment your query sets. If your product has distinct categories, create category-specific query sets. "Electronics queries" and "Clothing queries" should be separate if your search configurations differ between them.

Link query sets to business metrics. The best query sets are tied to business outcomes. If you know that 20% of your revenue comes from "brand name + product" queries, weight those queries heavily in your set. Your search tuning should prioritize what drives value.

Review query sets before every major experiment. Do not reuse a query set blindly. Check if the queries are still relevant, if new problem queries have emerged, and if your product catalog has changed. A query for a discontinued product is wasted effort.

Common Mistakes to Avoid

Over-reliance on synthetic queries. I have said this before, but it bears repeating. Synthetic queries give you synthetic metrics. Real users do not search like product managers.

Ignoring query frequency. All queries are not equal. A query that happens once a month is not as important as a query that happens a thousand times a day. Your query set should reflect this, either through sampling weights or through explicit frequency-based allocation.

Static query sets. A query set is not a monument. It is a living document. If you have not updated yours in six months, it is probably stale.

Too few judgments per query. A query set with 1000 queries but only 1 judgment per query is less useful than a 100-query set with 10 judgments per query. Judgment depth matters as much as query breadth.

Chasing aggregate metrics blindly. A 0.02 nDCG improvement across 500 queries sounds small. But if it is driven by a 0.15 improvement in your top 20 revenue-driving queries, it is a massive win. Segment your analysis.

Conclusion

Query sets are the foundation of search quality evaluation. Everything else - analyzers, scoring algorithms, experiments, metrics - rests on this foundation. A cracked foundation means shaky conclusions.

In OpenSearch Search Relevance, query sets are first-class objects. The plugin gives you the tools to manage, version, and reuse them. But it cannot tell you what queries to include. That requires understanding your users, your data, and your business.

The best query sets are sampled from reality, maintained over time, and tied to business outcomes. They are not perfect, but they are representative. And representative is enough to make better decisions than guessing.

If you have not reviewed your query set recently, do it now. Your metrics will thank you.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

DEV Community: Prithvi S

Segment Replication vs Document Replication: The OpenSearch 2.x Architecture Choice

The Baseline: How Replication Works in OpenSearch

Document Replication: Replaying Every Operation

Why this model is expensive

Segment Replication: Copying the Finished Product

Why this is a big deal

The Trade-offs Nobody Tells You About

1. Replication lag is coarser-grained

2. Network bandwidth moves from small and frequent to large and bursty

3. Recovery semantics change

4. It is the strategic direction of the project

Choosing Between Them: A Practical Framework

Trying It Out

The Bigger Picture

Lucene’s MMapDirectory: Zero‑Copy I/O and OS Page Cache

Introduction

1. The Directory Abstraction

2. How MMapDirectory Works

2.1 Memory‑Mapped Files

2.2 madvise Hints

2.3 Chunked Mapping

3. Benchmark: FS vs NIO vs MMap

4. When to Use Each Implementation

5. Production Tips

6. Code Example – Swapping in MMapDirectory

7. Pitfalls & Troubleshooting

8. Conclusion

The Inner Workings of OpenSearch: From Query to Fetch, Plugins to Performance

Search Execution: Query Phase vs Fetch Phase

Why the Two‑Phase Model?

Concurrent Segment Search and Caching

Concurrent Segment Search

Request Cache vs Query Cache

Tuning Tips

Extending OpenSearch with Plugins

Core Extension Points

Minimal Plugin Example (SearchPlugin)

Plugin Development Workflow

Data Lifecycle: Refresh, Flush, and Translog

Tuning Refresh and Flush

Practical Performance Tweaks

Conclusion

The Complete Polaris Security Stack: From Request to Credential

Why Traditional Approaches Fall Short

The Security Pipeline: A Request's Journey

Layer 1: Authentication - Who Are You?

Layer 2: Principal Roles - What Is Your Identity?

Layer 3: Catalog Roles and Privileges - What Can You Do?

Layer 4: Credential Vending - Scoped, Temporary Access

The Numbers Behind Credential Vending

OPA Integration: Externalizing Authorization

Production Security Hardening

TLS Everywhere

Persistence Security

Storage IAM Configuration

Credential Cache Tuning

Monitoring and Alerting

Why This Matters for Data Engineering Teams

The Bottom Line

How OpenSearch 3.0’s Concurrent Segment Search Changes Everything

The Problem Before 3.0

What Is Concurrent Segment Search?

Why It Works

Trade‑offs

How the Decision Engine Works

Configuring CSS

Basic Settings

Example: Tuning for a 16‑core Search Node

OpenSearch Search Pipelines: Transforming Results Without Changing Your Clients

Why Search Pipelines Exist

Pipeline Architecture: Three Types of Processors

1. Request Processors

2. Response Processors

3. Search Ext Processors

How Pipelines Are Defined and Applied

Building a Custom Processor: The Java Side

1. The Processor Class

2. The Factory

3. Plugin Registration

1. The `Directory` Abstraction

2. How `MMapDirectory` Works

2.2 `madvise` Hints

6. Code Example – Swapping in `MMapDirectory`