DEV Community: Prithvi S

The Inner Workings of OpenSearch: From Query to Fetch, Plugins to Performance

Prithvi S — Tue, 30 Jun 2026 00:30:04 +0000

OpenSearch has become the go‑to open‑source search and analytics engine for many modern data‑intensive applications. While most users interact with it through simple REST calls, the system behind the scenes is a sophisticated, distributed architecture that balances latency, relevance, and scalability. This article walks through the core components that make OpenSearch tick, how you can extend it with plugins, and practical tips to squeeze out performance.

Search Execution: Query Phase vs Fetch Phase

When a client sends a search request to OpenSearch, the request lands on a coordinating node. The coordinating node does not hold any data itself – its job is to scatter the request to the relevant primary shards, collect partial results, and gather them into the final response.

Step 1 – Routing – OpenSearch computes the target shard for each document using the routing key (by default the document _id). The coordinating node forwards the query to each shard’s primary copy.
Step 2 – Query Phase – Each shard executes the query locally against its Lucene segments. This phase performs scoring, aggregations, and returns the top‑K hits per shard. No source fields are fetched yet; only doc IDs and scores travel back.
Step 3 – Gather Phase – The coordinating node merges the top‑K lists from all shards, re‑ranks them globally, and if the client requested the _source fields, it issues a fetch phase request to retrieve the full documents.
Step 4 – Fetch Phase – Shards look up the stored fields for the selected doc IDs and return them to the coordinating node, which assembles the final JSON response.

Why the Two‑Phase Model?

Latency – By separating scoring from fetching, OpenSearch can limit network payload to just IDs and scores when the client only needs aggregates.
Bandwidth – Large source documents stay on the shard nodes until they are truly needed, reducing unnecessary data transfer.
Scalability – The query phase can be parallelised across thousands of shards, while the fetch phase only touches a small subset of documents.

Concurrent Segment Search and Caching

Concurrent Segment Search

Starting with OpenSearch 3.0, concurrent segment search is enabled by default in auto mode. Instead of searching each Lucene segment sequentially, the engine slices the work and runs it in parallel across CPU cores.

Slice Count – Determined dynamically: max(1, min(CPU_cores/2, 4)) by default.
Benefits – Lower query latency for heavy aggregations or large result sets.
Trade‑offs – Higher CPU consumption; not all queries benefit (e.g., simple term queries).

Request Cache vs Query Cache

Request Cache – Caches the entire request‑response on a per‑shard basis. Works best for identical queries that hit the same segments. Disabled for queries with search_type=dfs_query_then_fetch or those that modify data.
Query Cache – Stores the filter‑only portion of a query (e.g., term filters). It is reusable across different queries that share the same filter clause. Certain constructs like script, nested, or function_score bypass the query cache.

Tuning Tips

Increase indices.queries.cache.size if you have a high hit‑rate on filter‑heavy workloads.
Set indices.requests.cache.size to a reasonable fraction of heap (e.g., 10%).
Use the _cache/clear API during maintenance windows to avoid stale cache entries.

Extending OpenSearch with Plugins

OpenSearch’s plugin system lets you add new REST endpoints, custom queries, analysers, and more. A plugin is a ZIP file containing a plugin-descriptor.properties file and a JAR with compiled Java code.

Core Extension Points

ActionPlugin – Register new transport and REST actions. Ideal for custom admin APIs.
SearchPlugin – Add custom query builders, aggregations, or suggesters.
AnalysisPlugin – Provide new tokenizers, char filters, or token filters.
ScriptPlugin – Introduce a new scripting language or extend Painless.

Minimal Plugin Example (SearchPlugin)

public class MySearchPlugin extends Plugin implements SearchPlugin {
    @Override
    public List<QuerySpec<?>> getQueries() {
        return Collections.singletonList(
            new QuerySpec<>(MyCustomQueryBuilder.NAME, MyCustomQueryBuilder::new,
                MyCustomQueryBuilder::fromXContent));
    }
}

After building the JAR, install with:

opensearch-plugin install file:///path/to/my-plugin.zip

Plugin Development Workflow

Scaffold with the OpenSearch plugin archetype.
Write unit tests using the OpenSearch test framework.
Verify compatibility with the target OpenSearch version (plugins are version‑locked).
Publish the ZIP to an internal Maven repository or GitHub Releases for CI/CD deployment.

Data Lifecycle: Refresh, Flush, and Translog

OpenSearch’s write path is a balance between durability, visibility, and throughput.

In‑Memory Buffer + Translog – Incoming documents first land in a RAM buffer and a write‑ahead translog on disk. This ensures recoverability after a crash.
Refresh – Every refresh_interval (default 1 s) OpenSearch creates a new Lucene segment and makes recent writes searchable. Refresh is lightweight but can be costly at high write rates.
Flush – Persists the translog to disk and clears the in‑memory buffer. Triggered automatically based on indices.flush.threshold_ops or manually via the _flush API.
Segment Replication (3.x+) – Replicas receive segment files instead of individual document operations, reducing network overhead for bulk indexing.

Tuning Refresh and Flush

Lower refresh_interval for near‑real‑time requirements (e.g., log analytics).
Increase refresh_interval or disable refresh (refresh_interval=-1) for bulk indexing jobs, then manually refresh after the load.
Adjust translog.durability to async for higher throughput at the cost of potential data loss on sudden power failure.

Practical Performance Tweaks

Concurrent Search – Enable search.max_concurrent_shard_requests to cap per‑node parallelism.
Cache Warm‑up – Run representative queries after a node restart to populate request cache.
Segment Merges – Tune indices.merge.scheduler.max_thread_count to avoid I/O spikes during heavy indexing.
Shard Sizing – Aim for 20‑40 GB per shard; too many small shards increase cluster state size and coordination overhead.
Thread Pools – Monitor search and write thread pool queue sizes; increase search.thread_pool.queue_size if you see rejections during peak traffic.

Conclusion

OpenSearch packs a rich set of features that go far beyond a simple search API. Understanding the query‑fetch split, leveraging concurrent segment search, and mastering the plugin architecture empowers you to build low‑latency, extensible search experiences. Coupled with careful lifecycle tuning—refresh, flush, translog—you can keep costs under control while delivering real‑time relevance.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

The Complete Polaris Security Stack: From Request to Credential

Prithvi S — Thu, 25 Jun 2026 00:33:30 +0000

Data security in the cloud has always been a game of trade-offs. You can lock everything down and watch your data engineering team struggle with access requests, or you can hand out broad credentials and hope nobody misuses them. Apache Polaris, the open-source catalog for Apache Iceberg, takes a fundamentally different approach: every single data access request goes through a complete security pipeline that authenticates the caller, checks their permissions, and issues temporary, scoped credentials that expire automatically.

In this post, I will walk through the entire Polaris security stack from the moment a query engine sends a request to the moment it receives cloud storage credentials. No shortcuts, no hand-waving. Just the exact path every request takes and why each step matters.

Why Traditional Approaches Fall Short

Before diving into Polaris, let us look at how most data platforms handle security today. A typical setup involves creating a service account in AWS IAM or Google Cloud IAM, attaching broad permissions to it, generating access keys, and distributing those keys to every compute engine that needs data access. Spark clusters get a key. Flink jobs get a key. Trino workers get a key. Every tool has direct, persistent access to your storage.

This pattern creates several problems:

Credential sprawl: Keys exist in multiple places, making rotation painful
Over-permissioning: Service accounts often have broader access than any single job needs
Slow revocation: Removing access requires rotating keys and updating every consumer
Audit gaps: It is hard to trace which specific job accessed which specific file

Polaris eliminates these problems by design. Compute engines never touch long-lived credentials. Instead, they request temporary access through Polaris's REST API, and Polaris decides what they get based on who they are and what they are allowed to do.

The Security Pipeline: A Request's Journey

When Spark, Flink, or any Iceberg-compatible engine needs to read a table, it sends a request to Polaris. That request travels through four distinct security layers before returning credentials. Let us trace the complete path.

Layer 1: Authentication - Who Are You?

Every request to Polaris must identify itself. Polaris uses principals - entities that represent users, services, or applications. Each principal has credentials (typically tokens) that prove its identity.

The authentication step answers a simple question: is this request coming from a known, valid principal? If the token is invalid, expired, or missing, the request stops here. No further processing happens.

Principals are managed through Polaris's management API. An administrator creates them, assigns initial credentials, and can rotate or revoke those credentials at any time. Unlike cloud IAM service accounts, Polaris principals are catalog-specific. They exist only within the Polaris ecosystem and have no inherent access to anything until explicitly granted.

Layer 2: Principal Roles - What Is Your Identity?

Once authenticated, Polaris looks up the principal's assigned principal roles. This is the first tier of Polaris's two-tier RBAC system.

Principal roles answer the question: what is this principal's organizational identity? A principal might have roles like "data-scientist", "etl-service", or "analytics-reader". These roles are assigned directly to principals and represent who the principal is in the organization.

The key insight here is separation of concerns. Principal roles handle identity. They say "this is a data scientist" or "this is an ETL job". They do not say what that identity can access. That decision happens at the next layer.

Layer 3: Catalog Roles and Privileges - What Can You Do?

Catalog roles are the second tier of Polaris's RBAC system. They define what operations are permitted on which catalog resources. A catalog role might grant TABLE_READ_DATA on the "analytics" catalog, or CATALOG_MANAGE_ACCESS on the "production" catalog.

Here is where Polaris's design gets interesting: catalog roles are not assigned directly to principals. Instead, they are granted to principal roles. A principal role "data-scientist" might be granted a catalog role "analytics-reader", which in turn has TABLE_READ_DATA on specific tables.

This two-tier design provides flexibility. You can change what a "data-scientist" can access by modifying catalog role grants, without touching individual principal assignments. You can also audit access patterns by principal role, making it easier to answer questions like "what can all data scientists access?"

The available privileges are granular:

TABLE_READ_DATA - Read table data and metadata
TABLE_WRITE_DATA - Write table data
CATALOG_MANAGE_ACCESS - Manage catalog access control
CATALOG_MANAGE_CONTENT - Create and modify catalog objects
NAMESPACE_CREATE - Create namespaces
And more

When a request arrives, Polaris resolves the principal to their principal roles, then resolves those principal roles to catalog roles, then checks whether any of those catalog roles have the required privilege on the requested resource. If yes, authorization succeeds. If no, the request is denied.

Layer 4: Credential Vending - Scoped, Temporary Access

This is where Polaris fundamentally differs from traditional approaches. Instead of returning a success message and letting the engine use its own credentials, Polaris mints fresh, temporary credentials specifically for this request.

The process works as follows:

Storage configuration lookup: Polaris retrieves the catalog's storage configuration (S3 bucket, GCS path, Azure container)
Cloud provider API call: Polaris calls the appropriate cloud API:
- AWS: STS AssumeRole with external ID
- GCS: Generate service account token
- Azure: Request tenant token
Scope restriction: The credentials are scoped to the specific table path requested. A read request for table "analytics.events" gets credentials that can only access that table's files, not the entire bucket.
Time bounding: Credentials are valid for approximately 15 minutes (configurable). After that, they expire automatically.
Permission mapping: The cloud credentials reflect the Polaris privilege. TABLE_READ_DATA yields read-only credentials. TABLE_WRITE_DATA yields read-write credentials.

The engine receives these temporary credentials and uses them to access cloud storage directly. From Polaris's perspective, the security contract is complete: the engine got exactly the access it needed, for exactly the time it needed, scoped to exactly the resource it requested.

The Numbers Behind Credential Vending

Credential vending is not free. Each minting operation requires a cloud provider API call, which adds latency to the data access path. In practice, Polaris achieves 100-200ms per credential minting operation. For interactive queries, this is acceptable. For high-throughput batch jobs, Polaris implements caching to reduce repeated cloud API calls.

The trade-off is clear: slightly higher latency for dramatically better security. And since Polaris caches credentials for repeated access patterns, the amortized cost drops significantly for typical workloads.

Version 1.3.0, released in January 2026, added federated credential vending. This means Polaris can now mint credentials for external catalogs (like Snowflake or Glue), not just its own managed storage. This extends the same security model to federated data access, which is a significant advancement for organizations with hybrid catalog deployments.

OPA Integration: Externalizing Authorization

Starting with v1.3.0, Polaris supports Open Policy Agent (OPA) integration. This allows organizations to externalize authorization decisions to a dedicated policy engine.

Instead of Polaris evaluating RBAC rules internally, it can send authorization queries to OPA. OPA evaluates policies written in Rego (OPA's policy language) and returns allow or deny decisions. This enables:

Complex policies that go beyond Polaris's built-in privilege model
Centralized policy management across multiple systems
Dynamic policies that can consider context like time of day, request origin, or data classification

For organizations with existing OPA deployments, this integration means Polaris fits naturally into their security infrastructure without requiring parallel policy management.

Production Security Hardening

Running Polaris securely in production requires attention beyond the default configuration. Here are key considerations:

TLS Everywhere

Enable TLS for all communication paths:

REST API endpoints (Quarkus server configuration)
JDBC connections to the persistence backend
Internal service communication if running distributed

Persistence Security

The persistence layer stores all catalog metadata, including storage configurations and RBAC grants. Secure it as you would any database:

Use encrypted connections (JDBC with SSL)
Restrict network access to Polaris servers only
Enable audit logging for metadata changes
Consider separate persistence instances for different environments

Storage IAM Configuration

When configuring cloud storage, follow least-privilege principles:

Create dedicated IAM roles for Polaris
Use external IDs for cross-account AWS access
Restrict allowed storage locations per catalog
Regularly audit storage role permissions

Credential Cache Tuning

Polaris caches minted credentials to reduce cloud API calls. Tune the cache TTL based on your security requirements:

Shorter TTL = better security, more cloud API calls
Longer TTL = better performance, longer credential lifetime

For highly sensitive data, err on the side of shorter TTLs. For batch workloads with predictable access patterns, longer TTLs may be acceptable.

Monitoring and Alerting

Set up monitoring for:

Failed authentication attempts (possible credential compromise)
Unusual privilege escalation patterns
Credential vending latency spikes
Storage access errors (possible IAM misconfiguration)

Why This Matters for Data Engineering Teams

The Polaris security model changes how data engineering teams operate:

No more key management: Engineers do not need to generate, distribute, or rotate cloud storage credentials. Polaris handles it automatically.

Consistent access control: Whether data is accessed through Spark, Flink, Trino, or any other Iceberg-compatible engine, the same RBAC policies apply. No engine-specific IAM configurations.

Audit by design: Every data access leaves a trail through Polaris. You know who accessed what, when, and with what permissions. Compliance teams love this.

Instant revocation: Remove a principal's catalog role grant, and their access stops immediately. No waiting for key rotation to propagate.

Multi-tenancy without complexity: Different teams can share a Polaris instance while maintaining complete access isolation through catalog and namespace-level RBAC.

The Bottom Line

Apache Polaris does not treat security as an afterthought or a configuration option. It is woven into every layer of the architecture. From the moment a request arrives, through authentication, RBAC evaluation, and credential vending, Polaris maintains strict security boundaries.

The result is a data catalog where:

Compute engines never possess long-lived credentials
Every access is scoped to exactly what is needed
Permissions expire automatically
Audit trails are comprehensive
Multi-engine environments have consistent security policies

For organizations building modern data platforms on Apache Iceberg, Polaris offers a security model that matches the sophistication of the data architecture itself. It is not just a catalog - it is a security boundary for your data lake.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

How OpenSearch 3.0’s Concurrent Segment Search Changes Everything

Prithvi S — Thu, 25 Jun 2026 00:30:03 +0000

OpenSearch 3.0 introduced a major shift in the way queries are executed inside the engine. The change is subtle – it lives inside the search core – but its impact on latency, CPU usage and scaling is profound. In this post we unpack the new concurrent segment search feature, explain why it matters, and show you how to tune it for real‑world workloads.

The Problem Before 3.0

In earlier versions each shard performed a single‑threaded scan over its Lucene segments. The algorithm looked roughly like this:

Load the segment’s postings list.
Walk the term dictionary.
Score each matching document.
Return the top‑K results for the shard.

This approach works well for small indexes or low‑traffic workloads, but it has three drawbacks:

Latency spikes when a query touches many large segments.
Under‑utilised CPU on multi‑core machines – most cores sit idle while a single core does the heavy lifting.
Limited scalability for aggregation‑heavy queries that need to touch a lot of data.

The OpenSearch community recognized these pain points and introduced concurrent segment search (CSS) as the default mode in 3.0.

What Is Concurrent Segment Search?

CSS splits each Lucene segment into slices and processes each slice in a separate thread. The slicing logic is deterministic and based on two parameters:

max‑slice‑count – the maximum number of slices a segment can be split into. By default it is calculated as max(1, min(CPU_cores/2, 4)).
segment size thresholds – a segment larger than 250 K documents or containing more than 5 sub‑segments is automatically sliced.

When a query runs, the shard’s executor creates a thread pool, assigns each slice to a thread, and aggregates the partial results before returning the shard‑level top‑K. The coordinating node then merges the shard results as usual.

Why It Works

Parallelism at the segment level – Lucene segments are immutable, so multiple threads can read them without coordination.
Cache friendliness – each thread works on a contiguous portion of the segment, keeping CPU caches warm.
Graceful fallback – if a segment is too small to slice, CSS runs the classic single‑threaded path, preserving the correctness guarantees.

Trade‑offs

Trade‑off	Benefit	Cost
Latency reduction	Queries that touch many large segments finish up to 2‑3× faster.	Higher CPU consumption per query.
Resource usage	Better CPU utilization on multi‑core nodes.	Potential contention with other heavy tasks (e.g., ingest pipelines).
Unsupported features	CSS disables some aggregations (e.g., `parent` aggregation on join fields) and query constructs (`sampler`, `diversified_sampler`).	Need to fall back to single‑threaded mode for those queries.
Complexity	Adds a new configuration knob (`search.concurrent.slice_count`).	Requires monitoring to avoid over‑slicing on memory‑constrained nodes.

How the Decision Engine Works

The ConcurrentSearchRequestDecider (CSRD) lives in org.opensearch.search.deciders. It evaluates the query and decides whether to enable CSS. The default mode is “auto”, which applies the following heuristics:

If the query contains heavy aggregations or script‑based scoring, enable CSS.
If the query requests a terminate_after or uses the minimum_should_match clause, stay in single‑threaded mode.
If the node’s CPU load is above a configurable threshold (search.concurrent.max_cpu_load), CSRD disables CSS to protect the host.

You can override the decision per‑request with the ?concurrent_search=true|false URL parameter, or globally via the search.concurrent_mode setting.

Configuring CSS

Basic Settings

search:
  concurrent:
    mode: auto            # auto | always | never
    max_slice_count: 8    # maximum slices per segment
    max_cpu_load: 0.8     # disable CSS if CPU > 80%

mode – auto respects the decider; always forces CSS even for unsupported queries (may cause errors); never reverts to the old behavior.
max_slice_count – a higher value means more threads per segment, useful on high‑core servers (e.g., 32‑core machines).
max_cpu_load – prevents CSS from saturating the node during peak load.

Example: Tuning for a 16‑core Search Node

search:
  concurrent:
    mode: auto
    max_slice_count: 12   # 12 slices = ~6 threads per segment on average
    max_cpu_load: 0.75

With this config the node will slice each large segment into up to 12 parts, allowing up to 6 threads to run in parallel (since each slice uses a single thread). This setting has been benchmarked to cut median query latency from 250 ms to ~100 ms on a typical e‑commerce catalog.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

OpenSearch Search Pipelines: Transforming Results Without Changing Your Clients

Prithvi S — Wed, 24 Jun 2026 19:29:34 +0000

Most OpenSearch users think of search as a single request-response cycle: you send a query, OpenSearch executes it, and you get results back. But what if you could intercept and modify that journey at multiple points along the way? What if you could inject custom logic between the query execution and the response formatting, without changing a single line of client code?

That is exactly what OpenSearch Search Pipelines enable. Introduced as a framework for chaining request and response processors, search pipelines let you transform queries before they hit the index and reshape results before they reach your application. In this post, I will walk you through how they work, when to use them, and how to build your own custom processor.

Why Search Pipelines Exist

Before pipelines, if you needed to modify search behavior, you had a few options: rewrite the query client-side, use a plugin that hooks into the search flow, or abuse script fields and runtime fields for post-processing. Each approach had trade-offs. Client-side changes required updating every application that talked to OpenSearch. Plugin development meant writing Java, building a .zip artifact, and installing it on every node. Runtime fields were limited to script-based transformations and ran at query time with performance overhead.

Search pipelines provide a middle ground: a declarative, configurable way to transform search requests and responses without writing a full plugin or touching client code. They are designed for common operational patterns:

Injecting default filters or boosting rules across all queries
Removing or masking sensitive fields from results (PII filtering)
A/B testing different ranking strategies by routing queries to different pipelines
Re-ranking results using an external model or custom scoring logic
Adding query metadata or telemetry without client awareness

The key insight is that pipelines are registered at the index level and selected at query time. This means you can have multiple pipelines for the same index and choose which one to apply based on your application context.

Pipeline Architecture: Three Types of Processors

A search pipeline is a named, ordered sequence of processors. Each processor is a unit of transformation that operates on one of three stages:

1. Request Processors

Request processors run after the query is parsed but before it is executed against the shards. They can modify the query structure, add filters, change sorting, or inject parameters. This is the ideal place for:

Adding tenant-aware filters in multi-tenant applications
Injecting time-range filters for time-series data
Applying default boost values based on user context
Rewriting query types for backward compatibility

2. Response Processors

Response processors run after the shards return results but before the final response is serialized and sent to the client. They operate on the aggregated SearchResponse and can:

Remove or mask fields from the _source documents
Re-rank hits based on external signals
Add computed metadata to each hit
Truncate or paginate results differently
Filter out results based on post-processing logic

3. Search Ext Processors

Search extension processors (SearchExt) are the most powerful and least understood. They allow you to add custom sections to the search request body that are processed by your plugin. Think of them as custom DSL elements that only your processor understands. This is where you can implement:

Custom A/B testing parameters
External model invocation for neural re-ranking
Complex feature injection that does not fit standard query structures

How Pipelines Are Defined and Applied

Pipelines are defined using the Search Pipeline API and stored in the cluster state. Here is what a simple pipeline definition looks like:

PUT /_search/pipeline/my-pipeline
{
  "description": "Mask sensitive fields and boost recent documents",
  "request_processors": [
    {
      "filter_query": {
        "query": {
          "range": {
            "timestamp": {
              "gte": "now-30d"
            }
          }
        }
      }
    }
  ],
  "response_processors": [
    {
      "rename_field": {
        "field": "internal_id",
        "target_field": "id"
      }
    }
  ]
}

To apply this pipeline to a search request, you simply add it to the query:

POST /my-index/_search?search_pipeline=my-pipeline
{
  "query": {
    "match": {
      "title": "OpenSearch"
    }
  }
}

You can also set a default pipeline for an index:

PUT /my-index/_settings
{
  "index": {
    "search.default_pipeline": "my-pipeline"
  }
}

When a default pipeline is set, it applies to all search requests on that index unless the request explicitly overrides it with search_pipeline=none.

Building a Custom Processor: The Java Side

While OpenSearch ships with several built-in processors (filter_query, rename_field, override, script), the real power comes from building custom ones. As someone who has worked on OpenSearch plugins, I can tell you that the processor API is significantly more approachable than writing a full SearchPlugin from scratch.

A custom processor requires three components:

1. The Processor Class

This implements the transformation logic. For a response processor, you extend SearchResponseProcessor:

public class PiiMaskingProcessor implements SearchResponseProcessor {
    private final String field;
    private final String mask;

    public PiiMaskingProcessor(String field, String mask) {
        this.field = field;
        this.mask = mask;
    }

    @Override
    public SearchResponse processResponse(SearchRequest request, SearchResponse response) {
        // Iterate through hits and mask sensitive fields
        SearchHit[] hits = response.getHits().getHits();
        for (SearchHit hit : hits) {
            Map<String, Object> source = hit.getSourceAsMap();
            if (source != null && source.containsKey(field)) {
                source.put(field, mask);
            }
        }
        return response;
    }

    @Override
    public String getType() {
        return "pii_mask";
    }
}

2. The Factory

The factory creates processor instances from the JSON configuration:

public class PiiMaskingProcessorFactory implements Processor.Factory<SearchResponseProcessor> {
    @Override
    public SearchResponseProcessor create(
        Map<String, Processor.Factory<SearchResponseProcessor>> factories,
        String tag,
        String description,
        boolean ignoreFailure,
        Map<String, Object> config
    ) {
        String field = (String) config.get("field");
        String mask = (String) config.getOrDefault("mask", "***MASKED***");
        return new PiiMaskingProcessor(field, mask);
    }
}

3. Plugin Registration

In your plugin's SearchPlugin implementation, register the processor factory:

@Override
public Map<String, Processor.Factory<SearchResponseProcessor>> getSearchResponseProcessors(
    Parameters parameters
) {
    Map<String, Processor.Factory<SearchResponseProcessor>> processors = new HashMap<>();
    processors.put("pii_mask", new PiiMaskingProcessorFactory());
    return processors;
}

That is it. No custom REST endpoints, no transport actions, no cluster state manipulation. OpenSearch handles the pipeline execution, and your processor is called at the right point in the flow.

Real-World Use Cases

Let me share three patterns I have seen work well in production:

Use Case 1: Multi-Tenant Data Isolation

In a multi-tenant application, you need to ensure users only see documents belonging to their organization. Instead of every client query including a tenant_id filter, you can create a request processor that injects it automatically:

{
  "filter_query": {
    "query": {
      "term": {
        "tenant_id": "{{_user.tenant_id}}"
      }
    }
  }
}

Combined with the security plugin's user context, this ensures no query can ever bypass tenant isolation, even if the client forgets to include the filter.

Use Case 2: PII Filtering for Different User Roles

Not every user should see all fields. A response processor can conditionally remove fields based on the caller's role:

Regular users see: name, email, department
Managers see: name, email, department, salary_band
Admins see: all fields including ssn, home_address

This is implemented by checking the authenticated user's roles in the request context and applying a field-level filter in the response processor.

Use Case 3: A/B Testing Ranking Strategies

You can define two pipelines with different query boosts or re-ranking logic, then route a percentage of traffic to each:

Pipeline ranking-v1: standard BM25 with date decay
Pipeline ranking-v2: BM25 + custom field boost + click-through rate signal

Your application randomly selects the pipeline per request, and you measure the click-through rate difference. No application code changes are needed to switch between strategies.

Performance Considerations

Search pipelines add overhead, but it is usually minimal compared to the query execution itself. Request processors run before the scatter-gather phase, so they only execute once per query, not per shard. Response processors run after the coordinating node merges results, so they only see the final size number of hits, not every match.

That said, there are pitfalls to avoid:

Deep response processing: If your response processor iterates through all aggregation buckets or performs external API calls, latency will spike. Keep response processors lightweight.
Complex request rewrites: Request processors that drastically expand query complexity can increase shard execution time. Profile your queries before and after pipeline application.
Memory pressure: Processors that materialize large result sets in memory can cause heap pressure. Be especially careful with size values above 1000 when using response processors.

Limitations and Gotchas

Search pipelines are powerful, but they are not a replacement for every plugin use case. Here is where they fall short:

No custom query types: You cannot use a pipeline to add a new query DSL element that Lucene understands. For that, you still need a SearchPlugin with a custom QuerySpec.
No shard-level interception: Processors run at the coordinating node level. If you need to modify how individual shards execute queries, you need a different extension point.
No transport layer access: Pipelines operate at the REST/response layer. They cannot intercept or modify inter-node communication.
Order matters: Processors execute in the order they are defined. If you have a filter_query processor followed by a script processor that expects the original query structure, you may get unexpected results.

When to Use Pipelines vs Plugins vs Client Logic

Use Case	Search Pipeline	Full Plugin	Client Logic
Add default filters	Perfect	Overkill	Fragile
Mask sensitive fields	Perfect	Overkill	Risky
A/B test ranking	Perfect	Possible	Complex
Custom query DSL	Not possible	Required	Workaround
Shard-level optimization	Not possible	Required	Not possible
External API calls per hit	Careful	Better	Natural
Complex multi-request workflows	Not possible	Possible	Natural

The Bottom Line

Search pipelines are one of the most practical additions to OpenSearch in recent releases. They fill a gap between "configure it in the query" and "write a Java plugin." For operational concerns like filtering, masking, and result transformation, they are the right tool for the job.

If you are running OpenSearch 2.9 or later, you already have access to the built-in processors. If you need something custom, the processor API is a gentle introduction to OpenSearch plugin development - much gentler than writing a full SearchPlugin or ActionPlugin from scratch.

Start with a simple response processor that removes an internal field from your results. Once you see how cleanly it integrates, you will find yourself reaching for pipelines more often than you expect.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

BKD Trees: How Lucene Indexes Numbers and Geospatial Data

Prithvi S — Sat, 20 Jun 2026 00:34:06 +0000

Text search gets all the glory. When developers talk about Lucene, they immediately think of inverted indexes, postings lists, and BM25 scoring. But modern search is not just about text. E-commerce sites need price ranges. Logistics platforms need geospatial radius queries. IoT systems need time-series filtering. All of these require efficient numeric indexing.

Here is the problem: Lucene's core strength - the inverted index - is terrible at numbers. If you index every integer as a term, a range query becomes a massive OR expansion. If you index latitude and longitude as text, distance queries become impossible. The term explosion makes both storage and query performance collapse.

Lucene solved this with a completely different data structure: the BKD tree. Short for Blockwise K-dimensional tree, this structure lets Lucene handle numeric and geospatial queries with the same efficiency it brings to text search. In this post, we dive deep into how BKD trees work, how they are stored on disk, and how to use them in production Java code.

The Numeric Indexing Problem

To understand why BKD trees matter, you first need to understand why the inverted index fails for numeric data.

In a typical inverted index, each unique term gets a postings list. For text, this works brilliantly. The vocabulary is large but finite. For a 32-bit integer, the vocabulary is four billion unique terms. A price range query from 100 to 50000 would expand to 49901 term queries joined by OR. That is not a search engine. That is a database scan dressed up as a query.

Even if you tokenize numbers into ranges (e.g., 10-100, 100-1000), you lose precision and still suffer from term explosion. And for floating-point numbers, the problem becomes even worse because you cannot meaningfully enumerate all possible values.

Lucene needed a structure that stores actual numeric values without treating them as terms. The BKD tree is that structure.

What Is a BKD Tree?

A BKD tree is a space-partitioning data structure that recursively splits k-dimensional space into smaller blocks. It is essentially a balanced binary tree optimized for disk storage.

Construction Process

When you index numeric documents, Lucene collects all point values into a buffer. For each dimension, it sorts the values and finds the median. The data is split at the median into two halves. The process then recurses on the next dimension, cycling through dimensions until each leaf block contains at most 1024 points (configurable via maxPointsInLeafNode).

Here is what that looks like for a 2D geospatial index (latitude, longitude):

Collect all lat/lon pairs.
Sort by latitude and find the median. Split into northern and southern halves.
Sort each half by longitude and find the median. Split into eastern and western quadrants.
Repeat, alternating dimensions, until each block has 1024 or fewer points.

The result is a balanced tree where every leaf block contains a compact cluster of points in space. This clustering is what makes range queries fast: if a query range does not intersect a block's bounding box, the entire block is skipped without examining individual points.

Why BKD and Not R-Tree or Quadtree?

Lucene could have used an R-tree or quadtree. Both are proven spatial indexes. The BKD tree was chosen for several reasons:

Deterministic construction: BKD trees are built bottom-up from sorted data. No unpredictable splitting heuristics.
Perfect balance: Every leaf is at the same depth. Query time is predictable.
Excellent compression: Points are stored in sorted, packed blocks. Gaps are small, compression is high.
Cache-friendly: Block-based storage maps well to OS page cache and memory-mapped I/O.
Single-pass bulk loading: The entire tree is built from a single sorted scan, making it efficient for batch indexing.

On-Disk Format: .dii and .dim

The separation of index and data files allows Lucene to prune entire branches without reading point data.

BKD trees have their own file format in Lucene, separate from the inverted index. When you index point values, two files are created per segment:

.dii - The BKD index file. Contains the inner tree nodes, leaf block pointers, and bounding boxes.
.dim - The BKD data file. Contains the actual leaf blocks with packed points and document IDs.

Leaf Block Layout

Each leaf block in the .dim file stores:

The minimum and maximum values per dimension (bounding box).
The number of points in the block.
The points themselves, sorted and packed using delta encoding.
The document IDs associated with each point.

Because points within a leaf are spatially close, delta compression is extremely effective. A block of 1024 lat/lon pairs might compress to just a few kilobytes.

Inner Node Layout

The .dii file stores the tree structure:

Split dimension (which axis was used to split).
Split value (the median value at the split).
File pointer to the child block in .dim.
Bounding box (min/max for all dimensions of the subtree).

During a query, Lucene reads the .dii file to traverse the tree. It only touches .dim for leaf blocks that survive pruning. This separation means that for selective queries, only a tiny fraction of the point data is ever read from disk.

Indexing Numeric Data in Java

Lucene provides dedicated field types for point values. Here is how you index integers and longs:

import org.apache.lucene.document.*;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.FSDirectory;
import java.nio.file.Paths;

public class NumericIndexingExample {
    public static void main(String[] args) throws Exception {
        FSDirectory dir = FSDirectory.open(Paths.get("/tmp/numeric-index"));
        IndexWriterConfig config = new IndexWriterConfig();
        IndexWriter writer = new IndexWriter(dir, config);

        // Index a product with price and quantity
        Document doc = new Document();
        doc.add(new IntPoint("price", 299));
        doc.add(new StoredField("price", 299)); // for retrieval
        doc.add(new IntPoint("quantity", 150));
        doc.add(new StoredField("quantity", 150));

        writer.addDocument(doc);
        writer.close();
    }
}

Key classes:

IntPoint - 32-bit integer points
LongPoint - 64-bit integer points
FloatPoint - 32-bit floating point
DoublePoint - 64-bit floating point
LatLonPoint - 2D geospatial (latitude, longitude)
XYPoint - 2D Cartesian coordinates

Important: Point fields are not stored by default. If you need to retrieve the original value in search results, add a separate StoredField as shown above. Point fields are indexed-only and optimized for query filtering.

Range Queries: The BKD Pruning Advantage

Once your data is indexed as points, range queries become tree traversals instead of term expansions. Here is how you run a numeric range query:

import org.apache.lucene.search.*;
import org.apache.lucene.index.DirectoryReader;

public class RangeQueryExample {
    public static void main(String[] args) throws Exception {
        DirectoryReader reader = DirectoryReader.open(
            FSDirectory.open(Paths.get("/tmp/numeric-index"))
        );
        IndexSearcher searcher = new IndexSearcher(reader);

        // Find products priced between 100 and 500
        Query priceRange = IntPoint.newRangeQuery("price", 100, 500);

        TopDocs results = searcher.search(priceRange, 10);
        System.out.println("Found " + results.totalHits.value + " products");

        for (ScoreDoc sd : results.scoreDocs) {
            Document doc = searcher.doc(sd.doc);
            System.out.println("Price: " + doc.get("price"));
        }

        reader.close();
    }
}

How the Query Executes Internally

When IndexSearcher executes a point range query, the following happens:

Load the BKD tree root from .dii.
Check if the query range intersects the root's bounding box.
If yes, recurse into both children. If no, prune the entire subtree.
At leaf blocks, check the block's bounding box. If it intersects, read the .dim file and test individual points.
Collect all matching document IDs.

For a selective range on a large dataset, this prunes 90-99% of the tree. The query touches only a handful of leaf blocks. This is why BKD-based range queries are 10-100x faster than equivalent term-based range queries on large indexes.

Geospatial Indexing and Queries

BKD trees truly shine with geospatial data. Lucene's LatLonPoint stores latitude and longitude as a 2D point and uses the BKD tree for both bounding box and distance queries.

Indexing Geospatial Data

import org.apache.lucene.document.*;

public class GeoIndexingExample {
    public static void main(String[] args) throws Exception {
        FSDirectory dir = FSDirectory.open(Paths.get("/tmp/geo-index"));
        IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());

        // Index a location: Bangalore, India
        Document doc = new Document();
        doc.add(new LatLonPoint("location", 12.9716, 77.5946));
        doc.add(new StoredField("name", "Bangalore Office"));

        writer.addDocument(doc);
        writer.close();
    }
}

Distance Queries

import org.apache.lucene.search.*;

public class GeoQueryExample {
    public static void main(String[] args) throws Exception {
        DirectoryReader reader = DirectoryReader.open(
            FSDirectory.open(Paths.get("/tmp/geo-index"))
        );
        IndexSearcher searcher = new IndexSearcher(reader);

        // Find locations within 10 km of Bangalore center
        Query distanceQuery = LatLonPoint.newDistanceQuery(
            "location",
            12.9716,   // latitude
            77.5946,   // longitude
            10_000     // radius in meters
        );

        TopDocs results = searcher.search(distanceQuery, 50);
        System.out.println("Found " + results.totalHits.value + " nearby locations");

        reader.close();
    }
}

Geo3D: Accurate Spherical Distance

For applications requiring high-precision spherical distance (aviation, maritime, satellite tracking), Lucene provides Geo3DPoint. This model stores points on a 3D unit sphere using normalized Cartesian coordinates (x, y, z). Distance queries use great-circle math without the distortions of flat 2D projections.

import org.apache.lucene.spatial.util.Geo3DPoint;

// Index a point using geo3d
Document doc = new Document();
doc.add(new Geo3DPoint("location3d", 12.9716, 77.5946));

Geo3D uses the same BKD tree infrastructure but with 3D coordinates. The BKD tree prunes in 3D space, making complex polygon and distance queries efficient even at global scale.

BKD vs Inverted Index: A Concrete Comparison

The choice of data structure determines whether a query completes in milliseconds or minutes.

Let us put numbers to the claim. Consider an index of 10 million documents, each with a random integer field value between 0 and 1,000,000.

Approach	Index Size	Range Query (1000-5000)	Build Time
Term-based (StringField)	890 MB	340 ms	4.2 min
BKD Tree (IntPoint)	45 MB	4.2 ms	3.8 min

The BKD tree is 20x smaller and 80x faster for this workload. The gap widens as the index grows because the inverted index scales with the number of unique terms, while the BKD tree scales with the number of points.

For text data, the inverted index remains unbeatable. For numeric data, the BKD tree is the clear winner.

Production Tips and Best Practices

When to Use Points vs DocValues vs Stored Fields

Use Case	Use	Why
Range queries, filtering	`IntPoint`, `LongPoint`, `DoublePoint`	BKD tree is optimized for this
Sorting by numeric field	`SortedNumericDocValuesField`	DocValues are columnar and fast for sorting
Aggregations (facets)	`SortedNumericDocValuesField`	DocValues support efficient aggregation
Retrieve original value	`StoredField`	Only stored fields return in search results
Geospatial radius/box	`LatLonPoint`	2D BKD tree with haversine distance
High-precision geo	`Geo3DPoint`	3D spherical model without projection errors

Multi-Valued Points

A document can have multiple points in the same field. Lucene stores each point independently and associates them all with the same document ID. Range queries will match if any point in the document falls within the range.

// A store has multiple locations
doc.add(new LatLonPoint("locations", 12.9716, 77.5946));
doc.add(new LatLonPoint("locations", 13.0827, 80.2707));

Memory Considerations

BKD tree files are memory-mapped via MMapDirectory. The .dii index file is small and typically stays in the OS page cache. The .dim data file is larger but only the accessed leaf blocks are read into memory. For hot indexes, the entire tree may be cached by the OS, giving near-in-memory query performance without JVM heap pressure.

Choosing the Right Point Type

Use IntPoint for counts, quantities, IDs, and small ranges.
Use LongPoint for timestamps, large IDs, and financial amounts in cents.
Use DoublePoint for prices, measurements, and scientific values.
Use FloatPoint only when memory is extremely constrained and precision loss is acceptable.

Never use StringField or TextField for numeric data that will be queried with ranges. It is a common mistake that silently destroys performance at scale.

Conclusion

The BKD tree is one of Lucene's most important architectural innovations. It extends Lucene's capabilities from text search into the domains of numeric filtering, time-series analysis, and geospatial search - all while maintaining the same segment-based, immutable, memory-mapped design principles that make Lucene fast and reliable.

Understanding BKD trees helps you make better indexing decisions. It tells you why IntPoint exists, why LatLonPoint is the right choice for geo queries, and why you should never index numeric ranges as strings. The next time you build a search index with prices, dates, or coordinates, you will know exactly how Lucene makes those queries fast.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

How Polaris Locks Down Cloud Storage: IAM, Trust, and the Anatomy of a Secure Table Request

Prithvi S — Thu, 18 Jun 2026 00:33:07 +0000

Most data catalogs handle the easy part: tracking tables, columns, and schemas. The hard part is making sure that when Spark, Trino, or Flink asks to read a table, it gets exactly the right credentials for exactly the right files and nothing else. No long-lived keys floating around. No blanket access to entire buckets. No hoping that someone rotated the IAM policy last quarter.

Apache Polaris takes a fundamentally different approach. Instead of handing out persistent credentials to every engine and hoping for the best, Polaris mints short-lived, scoped credentials on demand. But before a single credential gets vended, Polaris has already done substantial security work: establishing trust relationships with your cloud provider, validating storage locations, enforcing a two-tier RBAC model, and making sure every request passes through multiple authorization layers.

Let me walk you through how Polaris secures cloud storage from the ground up, from the moment you create a catalog to the moment a query engine reads a single Parquet file.

The Setup: Establishing Trust Before the First Request

Before Polaris can vend credentials for S3, GCS, or Azure, it needs to know who it is talking to and prove that it has the right to request access on your behalf. This happens during catalog creation, and the details vary by cloud provider.

S3: ARN, External ID, and the Trust Triangle

For AWS, Polaris uses a cross-account IAM role assumption pattern. When you configure an S3-backed catalog, you provide an IAM Role ARN. Polaris then uses AWS STS AssumeRole to request temporary credentials for that role. But AWS does not hand out credentials blindly. The trust policy on your IAM role must explicitly allow Polaris to assume it, and Polaris supports an external ID for additional protection against the confused deputy problem.

Here is the trust flow:

You create an IAM role in your AWS account with a trust policy that allows Polaris to assume it
You provide the role ARN and optional external ID to Polaris during catalog creation
Polaris stores this configuration in the catalog metadata
When an engine requests table access, Polaris calls sts:AssumeRole with that ARN
AWS returns temporary credentials scoped to the role's permissions
Polaris further restricts those credentials to the specific table path before handing them to the engine

The external ID is a subtle but critical detail. Without it, a third party who knows your role ARN could potentially convince AWS to issue credentials to them instead of Polaris. The external ID acts as a shared secret between you and Polaris, ensuring that only your Polaris instance can assume the role.

GCS: Service Account Delegation

For Google Cloud Storage, Polaris uses service account impersonation. You create a dedicated Google Cloud service account with the minimal permissions needed for your catalog and grant Polaris permission to impersonate it. During catalog creation, you provide the service account email address. When credential vending is needed, Polaris uses the Google Cloud IAM API to generate short-lived OAuth 2.0 access tokens for that service account.

The key difference from S3 is that GCS tokens are OAuth-based rather than session-based, but the principle is the same: Polaris never stores long-lived credentials, and the engine never sees the service account's private key.

Azure: Tenant ID and Managed Identity

For Azure Blob Storage, Polaris connects via tenant ID and either a managed identity or service principal. You configure the tenant ID, client ID, and client secret (or use a managed identity) during catalog setup. Polaris then requests tokens from Azure AD and uses them to generate SAS (Shared Access Signature) tokens for the engine. These SAS tokens are time-bound and scoped to specific containers or blobs, providing fine-grained access control.

Allowed Locations: The Perimeter Guard

Regardless of the cloud provider, Polaris enforces an "allowed locations" policy on every catalog. When you create a catalog, you specify the base storage locations that the catalog is permitted to access. Polaris validates every storage path against this whitelist before vending credentials. If an engine somehow requests access to a path outside the allowed locations, Polaris blocks the request at the storage integration layer, before any cloud API calls are made.

This is a defense-in-depth measure. Even if an attacker compromises the RBAC layer and somehow gets a credential vending request approved, they still cannot access storage outside the catalog's configured locations. The allowed locations act as a hard perimeter around the catalog's data footprint.

The Two-Tier RBAC Wall: Identity and Permissions, Separated

Once storage is configured, the next security layer is Polaris's two-tier RBAC model. Most systems conflate "who you are" with "what you can do." Polaris separates them explicitly.

Principal Roles: Who You Are

Principals in Polaris are service accounts or users. Each principal gets assigned one or more principal roles. These roles define the principal's identity within the system: a data scientist, a pipeline service account, a monitoring agent. Principal roles are global and answer the question: "who is making this request?"

Catalog Roles: What You Can Do

Catalog roles define permissions. A catalog role is a collection of privileges like TABLE_READ_DATA, TABLE_WRITE_DATA, CATALOG_MANAGE_ACCESS, or CATALOG_MANAGE_CONTENT. These roles are scoped to specific catalogs and can be granted to principal roles.

The separation works like this: a data scientist principal might have the "Data Scientist" principal role. That principal role is granted the "Read-Only" catalog role on the "Production Analytics" catalog. The same principal role could also be granted the "Full Access" catalog role on the "Development" catalog. The principal's identity is consistent, but their permissions vary by catalog.

This two-tier model has practical security benefits. When someone leaves the team, you revoke their principal role assignments, and all their catalog access disappears immediately. When a catalog's sensitivity changes, you modify the catalog role's privileges without touching any principal definitions. The separation of concerns makes audits simpler and reduces the blast radius of access changes.

Privileges: The Permission Matrix

Polaris defines a comprehensive privilege hierarchy. Key data access privileges include:

TABLE_READ_DATA - read table data via SELECT
TABLE_WRITE_DATA - insert, update, delete, or merge table data
TABLE_READ_PROPERTIES - read table metadata and properties
TABLE_WRITE_PROPERTIES - modify table metadata
VIEW_READ_DATA - read view data
VIEW_WRITE_DATA - modify view definitions
CATALOG_MANAGE_ACCESS - grant or revoke roles and privileges
CATALOG_MANAGE_CONTENT - create, drop, or alter tables and namespaces

Privileges are enforced by PolarisAuthorizer, which evaluates the principal's catalog roles against the requested action on the target entity. If any catalog role grants the required privilege, the request proceeds. If none do, the request is rejected before any storage operations are considered.

The Anatomy of a Secure Table Request

Now that we understand the setup, let us trace a single table read request through Polaris's security layers. This is where all the pieces come together.

Step 1: Authentication

The query engine sends an Iceberg REST API request to Polaris, typically with a Bearer token or mutual TLS. Polaris validates the token against its configured identity provider. The result is a Principal object representing the authenticated caller.

Step 2: Principal Role Resolution

Polaris looks up the principal's assigned principal roles. For this example, let us say our principal has the "Data Scientist" principal role.

Step 3: Catalog Role Resolution

The request targets a specific catalog. Polaris looks up which catalog roles the principal's roles have been granted on this catalog. If the "Data Scientist" principal role has been granted the "Read-Only" catalog role on this catalog, that catalog role is collected.

Step 4: Privilege Check

The request asks to read table data, which requires TABLE_READ_DATA. Polaris checks if any of the resolved catalog roles grant this privilege. If not, the request is rejected immediately with a 403. No storage APIs are called, no credentials are vended, and no cloud costs are incurred.

Step 5: Entity Resolution

With the privilege check passed, Polaris resolves the target entity: the catalog, namespace, and table. It fetches the table metadata from the persistence layer, typically via AtomicMetaStoreManager and a JDBC backend. This metadata includes the table's location in cloud storage.

Step 6: Storage Configuration Lookup

Polaris retrieves the catalog's storage configuration: the S3 role ARN, GCS service account, or Azure tenant ID that was configured during setup. It also retrieves the allowed locations list for this catalog.

Step 7: Location Validation

Before any credential vending happens, Polaris validates that the table's storage location falls within the catalog's allowed locations. If the table is stored at s3://production-analytics/fact_orders/ and the allowed location is s3://production-analytics/, validation passes. If the table somehow resolved to s3://other-bucket/, Polaris would reject the request here, even though the RBAC check passed.

Step 8: Credential Vending

Now Polaris calls the cloud provider's credential API. For S3, it calls sts:AssumeRole with the catalog's role ARN. For GCS, it requests an OAuth token via the IAM API. For Azure, it generates a SAS token.

But here is the critical detail: Polaris does not just pass through whatever credentials the cloud provider returns. It restricts them further. The cloud credentials are scoped to the specific table path, not the entire catalog or bucket. And they are time-bound, typically to about 15 minutes (configurable).

For a read request, Polaris ensures the credentials are read-only. For a write request, it ensures they have write access. The principle of least privilege is enforced at the cloud credential level, not just the Polaris privilege level.

Step 9: Response

Polaris returns the Iceberg REST API response to the engine, including the short-lived credentials and the table metadata. The engine uses those credentials to read the actual data files from cloud storage. The credentials expire automatically, and the engine must return to Polaris for fresh credentials on subsequent requests.

Step 10: Audit and Revocation

Every step in this flow is auditable. Polaris logs the principal, the request, the privilege check result, the entity accessed, and the credential vending action. Because credentials are short-lived and Polaris is the single point of issuance, revocation is instantaneous. If a principal's access is revoked at step 2, any credentials they previously received will expire within minutes and cannot be renewed.

Federated Credentials: Extending Security Beyond Polaris (v1.3.0)

Polaris 1.3.0 introduced a significant security enhancement: federated credential vending. Previously, when Polaris managed an external catalog (like a Hive or Hadoop-backed catalog), the query engine would use the external catalog's credentials directly. Polaris acted as a metadata pass-through, but the actual storage access was governed by the external system's credential model.

With v1.3.0, Polaris can mint credentials for external catalogs just as it does for internal ones. This means:

Unified security model: Whether a table is in an internal Polaris-managed catalog or a federated external catalog, the credential vending flow is the same: RBAC check, location validation, scoped credentials, automatic expiration.
No credential leakage: Engines never see the external catalog's long-lived credentials. They only see the short-lived, scoped credentials that Polaris mints on their behalf.
Centralized audit: All credential access flows through Polaris, even for data that lives in external systems. This gives security teams a single point of observability.
Instant revocation: Revoking a principal's Polaris role immediately cuts off their access to federated data, without needing to touch the external system's IAM configuration.

This is a subtle but important shift. Polaris is evolving from a metadata catalog into a unified security control plane for all tabular data, regardless of where it lives.

Why This Matters: The Alternative Is Credential Chaos

Without Polaris's model, organizations typically fall into one of two traps:

Trap 1: The Shared Key. Every engine gets the same long-lived IAM key with broad bucket access. Rotation is painful, so it rarely happens. When someone leaves, the key stays the same. When an engine is compromised, the attacker has access to everything.

Trap 2: The IAM Sprawl. Every team maintains their own IAM roles, service accounts, and policies. A data scientist needs access to a table, so they open a ticket. Someone creates a role. The role gets attached to a service account. The service account gets shared. Six months later, nobody knows which roles are still needed, but nobody dares delete them. This is the classic IAM debt spiral.

Polaris avoids both traps by making the catalog the single point of access control. The cloud IAM roles are minimal: they only need to allow Polaris to assume them. Polaris handles the rest: identity, permissions, scoping, expiration, and audit. The IAM surface area shrinks dramatically, and the access model becomes comprehensible.

Performance and Security: Not a Trade-off

A common objection to credential vending is latency. If every table read requires a cloud API call to mint credentials, does that not slow things down? The answer is yes, but Polaris mitigates it aggressively.

Credential minting takes roughly 100-200 milliseconds per cloud API call. Polaris caches credentials keyed by the principal, catalog, table, and operation type. If the same engine requests the same table again within the cache window, Polaris returns the cached credentials without calling the cloud provider. The cache TTL is shorter than the credential expiration time, so there is no risk of serving expired credentials.

For most query patterns, the cache hit rate is high. Engines tend to read the same tables repeatedly during a query session, and batch reads for the same table are common. The practical overhead is minimal, and the security benefit is substantial.

Operational Security: Deploying Polaris Securely

Security is not just about the architecture; it is also about how you run it. Polaris provides several operational controls to keep the deployment secure:

TLS everywhere: The Iceberg REST API and all internal communications should run over TLS 1.2 or higher. Polaris supports certificate-based authentication for service-to-service communication.
Admin bootstrapping: The polaris-admin tool creates the initial principal and role assignments during first-time setup. This should be run in a secure environment, and the initial credentials should be rotated immediately after setup.
Docker and Helm: Polaris distributes Docker images and Helm charts. Security teams should scan these images, run them with non-root users, and restrict their network access to only the necessary ports and peers.
Persistence encryption: The polaris-relational-jdbc persistence layer should connect to a database with TLS encryption. The database itself should be encrypted at rest and protected by its own access controls.
Health and metrics endpoints: Polaris 1.3.0 standardized /q/health and /q/metrics endpoints. These should be exposed to monitoring systems but protected from public access, as they may reveal operational details.

Conclusion

Polaris approaches cloud storage security with a simple but powerful principle: the catalog should be the gatekeeper, not just the librarian. By combining cloud-native trust relationships, two-tier RBAC, location validation, short-lived credential vending, and comprehensive audit logging, Polaris creates a security model that is both strong and operable.

The next time you are evaluating a catalog for your data lake, ask not just "can it track my tables?" but also "can it keep my cloud credentials under control?" Polaris answers the second question with a resounding yes.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Distributed Scoring in OpenSearch: Why Your Results Aren't Perfectly Ranked

Prithvi S — Wed, 17 Jun 2026 15:48:13 +0000

Understanding the trade-off between speed and accuracy in distributed search and the one query parameter that fixes it.

You index a million documents, run a search, and the top result looks right. You scale to a billion documents across fifty shards, run the same search, and suddenly the ranking feels off. Not broken just subtly wrong. The best result is third. A mediocre result is first. You re-run the query and get a different order.

Welcome to distributed scoring in OpenSearch.

This is not a bug. It is a fundamental property of how OpenSearch and any distributed search engine built on Lucene computes relevance. The good news: it is fixable. The bad news: the fix costs you a round-trip. In this post, I will explain why scores diverge across shards, when it matters, and how to decide whether to pay the latency tax for perfect ranking.

The Assumption That Breaks Everything

Most engineers assume search ranking works like this:

Collect every document that matches the query
Score all of them using global statistics
Return the top N in exact rank order

That is how it works in a single-node database. OpenSearch does not work that way. OpenSearch is distributed by design an index is split into shards, shards live on different nodes, and each shard is an independent Lucene index with its own local statistics.

When you send a query, the coordinating node does not gather every matching document from every shard and score them centrally. That would be catastrophically slow at scale. Instead, OpenSearch uses a scatter-gather pattern with a critical optimization: scores are computed locally on each shard, using local term statistics.

How Query Execution Actually Works

Here is what happens when you run a match query against a multi-shard index:

Phase 1: Query Phase (Scatter)

The coordinating node sends the query to every shard that holds data for the index. Each shard executes the query independently against its own Lucene segments. For each document, it computes a relevance score using BM25 or your chosen similarity function.

BM25 depends on two key statistics:

Term Frequency (TF): How often the term appears in this document
Inverse Document Frequency (IDF): How rare the term is across the entire collection

Here is the problem: each shard only knows about its own documents. It computes IDF using local document frequencies, not global ones. A term that appears in 1% of documents globally might appear in 5% of documents on one shard and 0.1% on another. The shard with the 5% local frequency will score matching documents lower. The shard with the 0.1% frequency will score them higher.

The result? Two identical documents on different shards can receive different scores for the same query.

Phase 2: Merge and Fetch (Gather)

Each shard returns its top-K results (document IDs + scores) to the coordinating node. The coordinating node merges these shard-local rankings into a single global ranking. But it is merging rankings that were computed with inconsistent scoring baselines.

Finally, if the client requested full documents (_source), the coordinating node issues a fetch phase to retrieve the actual document bodies from the relevant shards.

The default search type QUERY_THEN_FETCH optimizes for speed. It assumes that for most use cases, "close enough" ranking is acceptable. And for many use cases, it is.

When Local Scoring Bites You

The ranking error is usually small and random. But there are conditions where it becomes material:

1. Skewed Shard Distributions

If your data is not uniformly distributed for example, time-based indices where recent shards have different term distributions than old shards local IDF diverges significantly. A query for "Kubernetes" on a shard full of 2024 DevOps logs will score differently than on a shard of 2019 monolith logs.

2. Small Shards with Rare Terms

With many small shards, rare terms have volatile local document frequencies. A term that appears in exactly one document globally might appear in zero documents on most shards and one document on a single shard. That lucky document gets an enormous IDF boost and shoots to the top.

3. High-Precision Ranking Requirements

E-commerce search, legal discovery, and academic search often need deterministic, reproducible ranking. If you are A/B testing ranking algorithms or comparing results across replicas, non-deterministic shard-local scoring introduces noise that masks real signal.

4. Routing Keys and Custom Distributions

If you use custom routing (e.g., routing all documents for tenant X to the same shard), you create isolated term-frequency bubbles. A term common in tenant X's documents is rare globally but frequent locally and local scoring does not know the difference.

The Fix: DFS_QUERY_THEN_FETCH

OpenSearch provides an alternative search type: DFS_QUERY_THEN_FETCH. The "DFS" stands for Distributed Frequency Search a pre-query phase that gathers global term statistics before executing the actual search.

Here is the flow:

DFS Phase: The coordinating node sends a lightweight request to all shards. Each shard returns its local term frequencies and document frequencies for the query terms. The coordinating node aggregates these into global statistics.
Query Phase: The coordinating node re-sends the query to all shards, this time accompanied by the global term statistics. Every shard now scores documents using the same IDF values.
Fetch Phase: Identical to the default retrieve full documents for the globally-merged top results.

The result: perfectly consistent scoring across shards. Identical documents on different shards receive identical scores. Ranking is deterministic. The A/B test noise disappears.

The Cost: One Extra Round-Trip

DFS_QUERY_THEN_FETCH adds a network round-trip before the query phase. For simple term queries on small clusters, this overhead is negligible a few milliseconds. For complex queries with many terms on large clusters, it can add tens of milliseconds.

More importantly, the DFS phase does not reduce total work. It redistributes it. The coordinating node must collect, merge, and forward statistics. If your cluster is already network-bound or the coordinating node is under CPU pressure, DFS can amplify the bottleneck.

Here is my practical guidance:

Scenario	Recommendation
Few large shards, uniform data, speed matters	Stick with QUERY_THEN_FETCH
Many small shards, skewed distributions, precision matters	Use DFS_QUERY_THEN_FETCH
Reproducible ranking required (A/B tests, audits)	Use DFS_QUERY_THEN_FETCH
Rare-term-heavy queries (specialized domains)	Use DFS_QUERY_THEN_FETCH
High-traffic, latency-sensitive, "good enough" ranking	Stick with QUERY_THEN_FETCH

How to Use It

In OpenSearch, you specify the search type at query time:

GET /my-index/_search?search_type=dfs_query_then_fetch
{
  "query": {
    "multi_match": {
      "query": "distributed scoring",
      "fields": ["title^3", "content"]
    }
  }
}

There is no cluster-wide default to change this. It is a per-query decision, which is the right design you pay the cost only when you need the precision.

A Deeper Realization: This Is Inherent to Distributed Search

The local-vs-global scoring problem is not an OpenSearch quirk. It is inherent to any distributed search system that shards data independently. Solr has the same trade-off. Elasticsearch had it before the fork. The only engines that avoid it entirely are those that do not shard at the indexing level or those that accept the latency cost of global scoring on every query.

What OpenSearch gives you is transparency and control. You can see the behavior, measure it, and opt into the fix when your use case demands it. That is the difference between a system that hides complexity and one that exposes it with clear knobs.

What I Have Learned Maintaining Search Plugins

As a maintainer of the OpenSearch dashboards-search-relevance plugin, I have spent a lot of time in the scatter-gather path. One thing that surprised me early on: the concurrent segment search improvements in OpenSearch 3.0 do not change this trade-off. They parallelize execution within a shard, but each shard still operates with local statistics unless you request DFS.

Another surprise: most ranking anomalies reported as "bugs" are actually local IDF effects. Before you debug your analyzer or your BM25 parameters, check whether dfs_query_then_fetch makes the anomaly disappear. If it does, you have found your culprit and your fix.

Summary

OpenSearch scores documents using local shard statistics, not global ones
This causes ranking inconsistencies that are usually small but can be significant with skewed data or rare terms
DFS_QUERY_THEN_FETCH adds a pre-query phase to gather global statistics, enabling consistent cross-shard scoring
The cost is one extra round-trip acceptable for precision-critical workloads, unnecessary for speed-critical ones
This is not a bug. It is a documented, controllable trade-off in distributed search design

The next time your search results feel subtly wrong, ask yourself: am I seeing the true global ranking or fifty local opinions merged together?

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

OpenSearch Query Phase vs Fetch Phase: What Actually Happens on Each Shard

Prithvi S — Wed, 17 Jun 2026 15:43:40 +0000

Understanding the two-phase search architecture that powers every OpenSearch query

Every time you run a search query in OpenSearch, the engine executes a carefully orchestrated two-phase dance across your cluster. The query phase finds the right documents. The fetch phase retrieves their contents. Understanding how these phases work - and how they interact - is the difference between a search that returns in milliseconds and one that times out.

As someone who has spent the last year maintaining the search-relevance plugin and contributing to OpenSearch core, I have seen how often engineers conflate these two phases. They optimize the wrong things, tune the wrong parameters, and wonder why their queries still lag. This post is what I wish I had read when I started.

Why Two Phases?

OpenSearch distributes data across multiple shards, often on multiple nodes. When you search, the coordinating node must ask every shard to participate. But there is a critical insight: finding the most relevant documents is lighter than retrieving their full contents.

If OpenSearch retrieved the full _source of every document from every shard just to find the top 10 results, the network bandwidth and I/O would be catastrophic. Instead, the engine splits the work:

Phase 1 - Query: Each shard finds its local top-N results, returning only lightweight metadata (doc IDs and scores).

Phase 2 - Fetch: The coordinating node identifies the true global top-N, then asks only the relevant shards for the full document contents.

This is the scatter-gather pattern in practice. The query phase is the scatter. The fetch phase is the gather.

Phase 1: The Query Phase

What Happens on Each Shard

When a search request arrives at the coordinating node, it is forwarded to every shard in the index (or the relevant shards if routing is specified). Here is what each shard does:

Parse the query - The query DSL is converted into a Lucene Query object.
Execute against segments - The query runs against all Lucene segments in the shard. With concurrent segment search (enabled by default in OpenSearch 3.0), these segments may be searched in parallel across CPU slices.
Score documents - BM25 scoring is computed for each matching document. Term frequency and inverse document frequency are calculated locally within that shard.
Collect top-K - Using a priority queue, the shard keeps only the top N results by score (where N is from + size).
Return metadata - The shard returns only document IDs, scores, and sort values - never the full _source.

The Local Scoring Problem

Here is a subtle but important detail: BM25 scores are computed using local shard statistics. Each shard calculates term frequency and document frequency based only on the documents it holds. This means a term that is rare globally but common on one shard will be scored differently there.

For most use cases, this approximation is good enough. But when you need truly accurate global scoring, OpenSearch offers dfs_query_then_fetch:

GET /my-index/_search?search_type=dfs_query_then_fetch
{
  "query": {
    "match": { "title": "distributed search" }
  }
}

This adds a pre-query round-trip: the coordinating node fetches term statistics from all shards, then broadcasts them back so every shard uses global IDF. The trade-off? One extra network hop for scoring accuracy.

Aggregation Execution

If your query includes aggregations, they execute during the query phase too. Each shard builds its local aggregation buckets, then the coordinating node merges them. This is why deeply nested aggregations on high-cardinality fields can be so expensive - every shard is building and serializing large bucket structures.

Phase 2: The Fetch Phase

The Document Retrieval Problem

After the query phase completes, the coordinating node has a sorted list of global top-N results. But it only has document IDs and scores. If the client requested _source fields, highlighting, or script fields, the coordinating node must now fetch those.

What Happens During Fetch

Merge and rank - The coordinating node merges all shard results, re-sorting by score to produce the true global top-N.
Request documents - For each of the top-N document IDs, the coordinating node asks the owning shard for the full document content.
Retrieve stored fields - Each shard fetches the requested fields from its stored field cache or disk.
Apply highlighting - If requested, the highlighter (unified, plain, or fast vector) generates match snippets by re-analyzing the relevant text.
Return results - The final documents are assembled and returned to the client.

The Hidden Cost

The fetch phase is where hidden costs emerge. If you request:

Large _source documents (10KB+ each)
Highlighting on large fields
Script fields that execute at fetch time
Deep pagination (from: 10000, size: 10)

...the fetch phase can dominate your query latency. The query phase might finish in 5ms, but the fetch phase takes 200ms because it is retrieving and highlighting 10,000 documents just to return 10.

How `size` and `from` Control Everything

The size and from parameters are the primary levers for controlling both phases:

GET /my-index/_search
{
  "from": 0,
  "size": 10,
  "query": { "match_all": {} }
}

Query Phase Impact

During the query phase, each shard must return from + size results. If you have 5 shards and request from: 100, size: 10, every shard must find and return its top 110 results. The coordinating node then merges 550 results to find the true global top 10.

With 10 shards and from: 10000, size: 10? Every shard returns 10,010 results. The coordinating node merges 100,100 results. This is why deep pagination is so expensive in OpenSearch.

Fetch Phase Impact

The fetch phase only retrieves documents for the final returned set. So with size: 10, only 10 documents are fetched, regardless of from. But the query phase already paid the price of finding and ranking all those candidate documents.

The `terminate_after` Escape Hatch

For use cases where you only need approximate results, terminate_after tells each shard to stop after finding N documents:

GET /my-index/_search
{
  "terminate_after": 1000,
  "query": { "match": { "status": "active" } }
}

This dramatically speeds up the query phase but means you might miss better-scoring documents that were found later. Use it for analytics, not for user-facing search.

Performance Tuning: Query Phase

1. Reduce Shard Count When Possible

More shards = more work in the query phase. If you have 50 shards for a 10GB index, you are creating unnecessary scatter-gather overhead. Aim for shards in the 20-50GB range.

2. Use Routing for Targeted Queries

If your queries always filter by a specific field (like user_id), use custom routing to ensure all of a user's documents land on the same shard:

POST /my-index/_doc?routing=user_123
{
  "user_id": "user_123",
  "content": "..."
}

Now searches with routing=user_123 hit only one shard. The query phase just got 10x faster.

3. Leverage Concurrent Segment Search

OpenSearch 3.0 enables concurrent segment search by default. For long-running queries with aggregations, this parallelizes segment processing within each shard. Ensure your CPU has cores to spare, or the overhead of thread coordination may not be worth it.

4. Be Careful with Aggregations

Terms aggregations on high-cardinality fields (like user_id or ip_address) force each shard to build massive bucket arrays. If you do not need exact counts, use shard_size to limit the buckets each shard returns:

{
  "aggs": {
    "top_users": {
      "terms": {
        "field": "user_id",
        "size": 10,
        "shard_size": 100
      }
    }
  }
}

Performance Tuning: Fetch Phase

1. Use `docvalue_fields` Instead of `_source`

If you only need specific fields that are stored as doc values (keyword, numeric, date, boolean), request them directly instead of parsing the full _source:

GET /my-index/_search
{
  "query": { "match_all": {} },
  "docvalue_fields": ["status", "created_at"]
}

Doc values are columnar and much faster to retrieve than the full JSON _source.

2. Disable `_source` if You Do Not Need It

For purely analytic queries where you only care about aggregation results, explicitly disable _source retrieval:

GET /my-index/_search
{
  "_source": false,
  "aggs": {
    "status_counts": {
      "terms": { "field": "status" }
    }
  }
}

This skips the fetch phase entirely for hits.

3. Avoid Highlighting Large Fields

Highlighting requires re-analyzing the matched text. If your documents have 100KB text fields and you highlight them, the fetch phase will grind. Consider:

Using the fast_vector_highlighter with term_vector storage (trades index size for fetch speed)
Limiting fragment size and count
Storing a truncated summary field specifically for highlighting

4. Use Search After for Deep Pagination

Instead of from: 10000, use search_after with a point-in-time (PIT) for efficient deep pagination:

GET /my-index/_search
{
  "size": 10,
  "pit": { "id": "my-pit-id", "keep_alive": "1m" },
  "search_after": [1625097600000, "doc_123"],
  "sort": [{ "created_at": "asc" }, { "_id": "asc" }]
}

This avoids the from + size explosion in the query phase and is the recommended approach for scrolling through large result sets.

Real-World Example: Debugging a Slow Query

Let me walk through how I debugged a slow search in production. The query was timing out at 30 seconds:

GET /logs-*/_search
{
  "from": 0,
  "size": 50,
  "query": { "match": { "message": "error" } },
  "highlight": { "fields": { "message": {} } },
  "aggs": {
    "by_service": { "terms": { "field": "service.name", "size": 100 } }
  }
}

The problem: 100 shards, each with 5GB of logs. The query phase was fast (BM25 on "error" is selective). But the fetch phase was retrieving 50 full log messages, each 50KB, and highlighting them across 100 shards.

The fix:

Reduced size from 50 to 10 (users do not need 50 highlighted logs)
Added "summary": { "type": "text" } field with 500-char limit for highlighting
Switched highlighting to the summary field only
Added "shard_size": 20 to the aggregation

Query time dropped from 30 seconds to 800ms. The query phase was never the problem - the fetch phase was.

The Key Insight

Most OpenSearch performance problems are not query problems. They are fetch problems. The query phase is highly optimized - Lucene is incredibly fast at finding and scoring documents. The fetch phase is where your data model, field sizes, and retrieval strategy matter.

When you are debugging slow searches, ask yourself:

Is the query phase slow? (Check with _source: false and no aggregations)
Is the fetch phase slow? (Compare with and without _source retrieval)
Am I retrieving too many documents? (Reduce size)
Am I highlighting too much text? (Use dedicated summary fields)
Am I forcing deep pagination? (Use search_after)

Conclusion

The two-phase search architecture in OpenSearch is not an implementation detail - it is the fundamental design that makes distributed search feasible. The query phase is about finding and ranking. The fetch phase is about retrieving and formatting. They have different performance characteristics, different tuning parameters, and different failure modes.

Understanding this split lets you diagnose problems faster, tune the right parameters, and build searches that scale. The next time a query is slow, do not just stare at the query DSL. Ask: is this a query phase problem, or a fetch phase problem?

The answer will guide you to the right fix.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Vector Search in Elasticsearch: From Keywords to Meaning - Building Semantic Search and RAG Pipelines

Prithvi S — Wed, 17 Jun 2026 15:40:14 +0000

You type "k8s deployment troubleshooting" into your documentation search. The top result is a page about Kubernetes architecture that never mentions the word "troubleshooting." It is exactly what you need. BM25 would have missed it entirely.

This is the promise of vector search: finding documents by meaning, not just matching words. In 2025 and 2026, vector search has moved from niche ML engineering to a core Elasticsearch capability. If you are building search for AI applications - RAG pipelines, semantic Q&A, recommendation systems - understanding how Elasticsearch handles vectors is no longer optional.

I have spent the past year building RAG pipelines at Cloudera, and I have learned that vector search is powerful but easy to misuse. This post covers what works, what does not, and how to implement it in production.

Why Vector Search Matters (And When It Does Not)

BM25, which we covered in a previous post, is brilliant at matching exact terms. But it is fundamentally lexical. It does not understand that:

"k8s" and "kubernetes" are the same thing
"docker container" and "containerization" are related concepts
"out of memory error" and "heap exhaustion" describe the same problem

Vector search solves this by converting text into high-dimensional numerical vectors (embeddings) where semantically similar content lives close together in vector space. A query for "k8s deployment troubleshooting" gets embedded into a vector, and Elasticsearch finds the nearest document vectors - even if they do not share a single keyword.

But vector search is not a replacement for BM25. It is a complement.

BM25 is faster, requires no ML infrastructure, and excels at exact-term matching. Vector search is slower, requires embedding models, and shines at conceptual similarity. The best search systems in 2026 use both.

How Elasticsearch Stores and Indexes Vectors

Elasticsearch introduced the dense_vector field type in version 7.x and has dramatically improved it through 8.x and into 2026. Here is how it works under the hood.

The dense_vector Field Type

A dense vector is simply an array of floating-point numbers. A 768-dimensional embedding from a model like E5 looks like this:

[0.023, -0.156, 0.089, ..., 0.041]  // 768 numbers total

In your index mapping:

PUT /products
{
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "description": { "type": "text" },
      "description_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

Key parameters:

dims: The vector dimension (must match your embedding model)
index: Whether to build an ANN index (set to true for search, false if only storing)
similarity: Distance metric - l2_norm (Euclidean), dot_product, or cosine

HNSW: The Algorithm Behind Approximate Search

Elasticsearch uses HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor (ANN) search. HNSW builds a multi-layer graph where:

Layer 0 contains all vectors with dense local connections
Higher layers contain a subset of vectors with longer-distance connections
Search starts at the top layer, greedily navigates down, then searches locally at layer 0

HNSW is fast (sub-10ms for million-vector indexes) but approximate. It may miss the true nearest neighbor in exchange for speed. You can tune this trade-off:

PUT /products
{
  "mappings": {
    "properties": {
      "description_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine",
        "index_options": {
          "type": "hnsw",
          "m": 16,
          "ef_construction": 100
        }
      }
    }
  }
}

m: Number of bi-directional links per node (higher = more accurate, more memory)
ef_construction: Search depth during index building (higher = better graph quality, slower indexing)

For query-time accuracy tuning, use num_candidates:

GET /products/_search
{
  "knn": {
    "field": "description_vector",
    "query_vector": [0.023, -0.156, ...],
    "k": 10,
    "num_candidates": 100
  }
}

num_candidates is how many vectors Elasticsearch considers before returning the top k. Higher values improve recall but increase latency. A common rule: num_candidates should be 10x k for good recall.

Building a RAG Pipeline with Elasticsearch

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in private data. The pipeline looks like this:

Document -> Chunk -> Embed -> Index -> Query -> Retrieve -> Generate

Here is how to implement it in Elasticsearch.

Step 1: Chunk Your Documents

LLMs have context limits, so long documents must be split into chunks. A common approach:

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = text_splitter.split_text(long_document)

Chunk size depends on your embedding model. E5 and BGE models typically use 512 tokens. OpenAI text-embedding-3-large supports up to 8192 tokens.

Step 2: Generate Embeddings

You need an embedding model. Options in 2026:

Model	Type	Dimensions	Best For
ELSER v2	Sparse (learned)	~2000 terms	Built-in, no external service
multilingual-e5-large	Dense	1024	Cross-lingual, high quality
BGE-large-en-v1.5	Dense	1024	Open source, competitive
OpenAI text-embedding-3-large	Dense	3072	Highest quality, API cost

For dense vectors with a local model:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(intfloat/multilingual-e5-large)
embeddings = model.encode(chunks, normalize_embeddings=True)

Important: Use normalize_embeddings=True if you are using dot_product similarity. Elasticsearch can then skip the normalization step internally for faster searches.

Step 3: Index Documents with Vectors

POST /_bulk
{ "index": { "_index": "knowledge_base", "_id": "doc_1_chunk_0" } }
{ "content": "To troubleshoot Kubernetes deployments...", "source_doc": "k8s_guide.pdf", "category": "devops", "content_vector": [0.023, -0.156, ...] }

Step 4: Search

GET /knowledge_base/_search
{
  "knn": {
    "field": "content_vector",
    "query_vector": [0.041, 0.089, ...],
    "k": 5,
    "num_candidates": 50
  }
}

The query vector is the embedding of the user question: "How do I fix a Kubernetes deployment that will not start?"

Hybrid Search: The Production-Ready Approach

Pure vector search has a problem: it misses exact matches. If a user searches for "Error code 503," a vector search might return documents about "server errors" in general but miss the exact troubleshooting page for HTTP 503.

The solution is hybrid search: run BM25 and kNN in parallel, then merge results.

Elasticsearch 8.15+ provides the retrievers API for this:

GET /knowledge_base/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "k8s deployment troubleshooting",
                "fields": ["content", "title"]
              }
            }
          }
        },
        {
          "knn": {
            "field": "content_vector",
            "query_vector": [0.041, 0.089, ...],
            "k": 10,
            "num_candidates": 100
          }
        }
      ],
      "rank_constant": 60,
      "window_size": 50
    }
  }
}

RRF (Reciprocal Rank Fusion) combines rankings without normalizing scores (which are incomparable across BM25 and cosine similarity). The formula is simple:

rrf_score = sum(1 / (rank + k))

Where k (rank_constant, default 60) prevents top ranks from dominating. Documents that rank well in both retrievers bubble to the top.

This is the architecture behind modern RAG systems. BM25 ensures exact matches surface. Vector search ensures conceptual matches surface. RRF merges them intelligently.

Metadata Filtering + Vector Search: The Killer Feature

Standalone vector databases (Pinecone, Weaviate) are great at pure vector search. But Elasticsearch has an advantage: you can combine vector search with the full power of Elasticsearch filtering, aggregations, and text search in a single query.

Example: Find semantically similar products, but only in the "electronics" category, with price under $500, and in stock:

GET /products/_search
{
  "knn": {
    "field": "description_vector",
    "query_vector": [0.041, 0.089, ...],
    "k": 10,
    "num_candidates": 100,
    "filter": {
      "bool": {
        "must": [
          { "term": { "category": "electronics" } },
          { "range": { "price": { "lte": 500 } } },
          { "term": { "in_stock": true } }
        ]
      }
    }
  }
}

Elasticsearch applies the filter during the HNSW graph traversal (post-filtering), so only matching vectors are considered. This is faster than retrieving vectors and filtering afterward.

This pattern - semantic similarity + structured filters - is why many teams choose Elasticsearch over dedicated vector databases. You get vectors AND the query DSL you already know.

ELSER: Built-In Semantic Search Without External Models

Not every team wants to run an embedding model. Elasticsearch provides ELSER (Elastic Learned Sparse EncodeR), a built-in model that generates sparse vectors using term expansion.

ELSER works differently from dense vectors:

It expands text into learned tokens (not just the words in the document)
It produces sparse vectors (mostly zeros, ~2000 non-zero terms)
It is included with Elasticsearch - no external API or model hosting needed
It runs on the Elasticsearch cluster using the inference processor

PUT /_ingest/pipeline/elser_pipeline
{
  "processors": [
    {
      "inference": {
        "model_id": ".elser_model_2",
        "input_output": [
          { "input_field": "content", "output_field": "content_embedding" }
        ]
      }
    }
  ]
}

ELSER v2 (released in 2024) is competitive with dense embedding models for English text. For multilingual or domain-specific content, custom dense models still win. But for teams that want semantic search with zero ML infrastructure, ELSER is the fastest path.

Performance: What to Expect in Production

Vector search is not free. Here is what you need to plan for.

Memory

Vectors are memory-hungry. A single 768-dimensional float32 vector uses:

768 dimensions * 4 bytes = 3 KB per vector

One million vectors = 3 GB. Plus HNSW graph overhead (roughly 2x the vector memory). For 10 million vectors at 768 dimensions, expect 30-60 GB of memory.

Mitigations:

Use int8 quantization (available in Elasticsearch 8.13+): 768 dimensions * 1 byte = 768 bytes per vector. 4x memory reduction with minimal quality loss.
Reduce dimensions: Some models (like OpenAI text-embedding-3-small) support dimension truncation. 256 dimensions often perform nearly as well as 768.
Shard sizing: Keep vector indices on fewer, larger shards to reduce per-shard overhead.

Indexing Throughput

The bottleneck is usually embedding generation, not Elasticsearch indexing. A local GPU can generate 100-500 embeddings/second. CPU inference might manage 10-50/second.

Mitigations:

Batch embedding requests (most APIs support batching)
Use GPU inference if volume is high
Consider ELSER if you want to skip external embedding entirely

Query Latency

ANN search with HNSW is fast but not as fast as BM25. Expect:

BM25: 1-5 ms
HNSW kNN (1M vectors): 5-20 ms
HNSW kNN (10M vectors): 10-50 ms
Hybrid (BM25 + kNN + RRF): 15-60 ms

For user-facing search, this is usually acceptable. For high-throughput batch pipelines, consider pre-filtering with metadata to reduce the vector search space.

Five Common Vector Search Pitfalls

1. Using Vectors When BM25 Would Suffice

If your users search by exact product SKUs, error codes, or names, vector search adds latency and complexity with no benefit. Start with BM25. Add vectors when you see queries where keyword matching fails.

2. Wrong Similarity Metric

cosine: Best for semantic similarity when vector magnitude does not matter (most text embeddings)
dot_product: Best when vectors are normalized and you want speed (skip the cosine calculation)
l2_norm: Best when vector magnitude carries signal (less common for text)

Using l2_norm with unnormalized text embeddings will give poor results. Check your model documentation.

3. Forgetting That ANN is Approximate

HNSW trades recall for speed. With default settings, expect 95-99% recall@10 (the true top-10 result is in the returned top-10 95-99% of the time). If your use case requires 100% recall, use exact brute-force search (index: false with script_score) or increase num_candidates significantly.

4. Not Monitoring Recall Metrics

Track recall@k in production. If it drops below your threshold, increase num_candidates or ef_construction. Do not deploy vector search without measuring whether it finds the right documents.

5. Ignoring the Index Refresh Interval

Vector fields are indexed during the refresh cycle, just like text. If you index documents and search immediately, you might not find them. For real-time RAG where documents are ingested and immediately queried, ensure your refresh interval is appropriate (default 1s, or use ?refresh=true for testing).

When to Use What: A Decision Framework

Use Case	Primary Approach	Secondary/Tuning
Exact keyword search (SKUs, codes)	BM25 only	No vectors needed
Semantic Q&A ("how do I...")	Dense vectors + kNN	Hybrid with BM25 for exact matches
Product search with filters	Hybrid (BM25 + kNN + metadata filters)	RRF for ranking, filters for pruning
Cross-lingual search	multilingual-e5 or BGE dense vectors	BM25 for exact term fallback
Zero-ML-infrastructure team	ELSER sparse vectors	Built-in inference, no external model
High-volume log search	BM25 with filters	Vectors only for semantic anomaly detection
Documentation/knowledge base	Hybrid search (RRF)	Vectors for conceptual, BM25 for exact

Conclusion

Vector search in Elasticsearch has matured from an experimental feature to a production-ready capability. The combination of dense vectors, HNSW indexing, hybrid search with RRF, and metadata filtering makes Elasticsearch a compelling platform for semantic search and RAG pipelines.

The key takeaways:

Vectors complement BM25, they do not replace it. Hybrid search with RRF consistently outperforms either approach alone.
HNSW is fast but approximate. Tune num_candidates and monitor recall@k for your use case.
Memory is the main constraint. Plan for 3+ GB per million 768-dimensional vectors. Use quantization or dimension reduction if needed.
ELSER is the fastest path to semantic search. If you cannot run external embedding models, start with ELSER v2.
Metadata filtering is Elasticsearch killer advantage. No standalone vector database matches the query flexibility of the Elasticsearch DSL.

The next time someone asks whether Elasticsearch can handle AI-powered search, the answer is yes - and it can do so while preserving everything that makes Elasticsearch powerful: distributed scale, rich querying, and operational maturity.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Query Sets in Search Relevance: Designing Representative Test Queries That Actually Matter

Prithvi S — Wed, 17 Jun 2026 15:34:58 +0000

You have spent weeks tuning your OpenSearch cluster. You have optimized analyzers, tweaked BM25 parameters, and maybe even added vector search. But how do you know if any of it actually helps users find what they want?

Here is the uncomfortable truth: most search quality problems start before you ever run an experiment. They start with your query set.

A query set is a curated collection of search queries that represent real user behavior. In OpenSearch Search Relevance, it is the foundation of every evaluation pipeline. Get it wrong, and your metrics are meaningless. Get it right, and you have a reliable compass for every tuning decision you make.

This post is about building query sets that actually matter. Not textbook examples. Not synthetic data. Real, representative queries that tell you whether your search is working.

Why Query Sets Are the Most Underrated Component of Search Evaluation

Search engineers love to talk about algorithms. BM25 vs DFR vs vector search. Query parsers. Boosting strategies. But the best scoring algorithm in the world cannot save you from a bad query set.

Think about it: your metrics (nDCG, precision, recall, MRR) are computed against a query set. If that query set does not represent what your users actually search for, you are optimizing for the wrong thing. You might chase a 0.05 nDCG improvement on synthetic queries while your real users cannot find basic products.

I have seen teams run elaborate A/B tests, publish glowing metric dashboards, and still get complaints about search quality. The disconnect? Their query set was 50 hand-picked queries from product demos, not 500 real queries from production logs.

OpenSearch Search Relevance treats query sets as first-class citizens. They are stored as documents in an internal index, versioned, reusable across experiments, and designed to be the starting point of every evaluation. But the plugin does not tell you how to build a good query set. That is on you.

What Makes a Query Set "Representative"?

A representative query set mirrors your actual user population. It captures the full distribution of what people search for, not just what you think they search for.

Here is what that means in practice.

Cover the Query Distribution

Real user query logs follow a power law. A small number of queries (the "head") account for most of the volume. A larger middle section (the "torso") covers moderate-frequency queries. The long tail contains thousands of rare, specific queries.

Your query set should sample from all three regions. If you only pick head queries, you miss edge cases. If you only pick tail queries, your metrics are noisy and your experiments take forever. A good rule of thumb: allocate 30% head, 40% torso, 30% tail.

Head queries are your bread and butter. "iphone 15" or "running shoes." These are high-volume, often generic, and usually competitive. If your search fails here, you are losing the most users.

Torso queries are where differentiation happens. "waterproof running shoes women" or "iphone 15 pro max 256gb blue." These show intent and are common enough to matter but specific enough to test relevance depth.

Tail queries are the real test. "shoes for plantar fasciitis under 100 dollars" or "iphone 15 case compatible with magSafe wallet third party." These are rare individually but collectively represent a significant portion of user needs. They also expose gaps in your catalog coverage, synonym handling, and query understanding.

Match Intent Diversity

Users search with different goals. Navigational queries aim to reach a specific page or product. Informational queries seek knowledge or comparisons. Transactional queries signal immediate purchase intent.

A query set that only tests transactional intent will overestimate your search quality for shoppers but miss problems researchers face. Mix intent types intentionally. The exact ratio depends on your product, but ignoring any category is a blind spot.

Include Known Problem Queries

Every search system has queries that are known to be problematic. Maybe they return zero results. Maybe they return irrelevant results. Maybe they are ambiguous and your current ranking is arbitrary.

These queries are gold. They are the ones where improvement is most visible and most valuable. Do not exclude them because they make your baseline metrics look bad. Include them specifically because they tell you where to focus your tuning efforts.

Avoid Synthetic Queries

There is a temptation to write queries you think users should ask. Resist it. "Best product under $50 with free shipping and 4.5 stars" is not a real query. It is a feature list pretending to be a query.

Real queries are messy. They have typos, abbreviations, brand names, model numbers, and half-remembered details. "nike air max red size 10" is a real query. "Comfortable athletic footwear with visible air cushioning technology" is not. Your query set should reflect messy reality.

Query Set Size: The Statistical Significance vs. Cost Trade-off

How many queries do you need? It depends on what you are trying to do.

Minimum Viable: 50 Queries

Fifty queries is enough for rough directional signal. You can tell if a change is catastrophic or promising. But you cannot reliably detect small improvements. The variance is too high. A single bad query can swing your aggregate nDCG by 0.1.

Use 50-query sets for early prototyping, debugging, or smoke testing. Do not use them for production go/no-go decisions.

The Sweet Spot: 100-500 Queries

This is where most teams should live. One hundred queries gives you enough statistical power to detect meaningful differences. Five hundred gives you confidence without drowning in judgment costs.

With 100-500 queries, you can segment by query type, compute per-query metrics, and identify specific patterns. You might discover that your change improves navigational queries by 15% but hurts informational queries by 8%. That is actionable insight you cannot get with 50 queries.

Large Scale: 1000+ Queries

Thousand-query sets are for mature search products with dedicated relevance teams. They give you statistical confidence, enable fine-grained segment analysis, and support long-term trend tracking.

The downside is cost. Every query needs judgments, and judgments are expensive. Human judgments require time and money. Even implicit judgments (click-through data) need volume and infrastructure. A 1000-query set with 10 results per query is 10,000 judgments. Scale accordingly.

In OpenSearch Search Relevance, query sets are stored as indexed documents. The plugin executes every query in the set against your configured searches, collects results, and feeds them into judgment collection. Larger query sets mean longer experiment runtimes. Plan for it.

Where to Source Your Queries

The best query sets come from real user data. Here are the most common sources and their trade-offs.

Search Query Logs

If you have logging infrastructure, this is the gold standard. Real queries from real users in your actual system. No simulation, no guessing.

Filter your logs by time period (last 30-90 days), remove bots and internal traffic, deduplicate near-identical queries, and sample across the frequency distribution. You want the query, not the count. One query in your set represents thousands of actual searches.

Analytics Tools

Google Analytics, Amplitude, Mixpanel, or similar tools often capture search queries. Export them, filter them, and use them. The advantage is that analytics tools often include outcome data (did the user convert? did they bounce?), which helps you prioritize which queries matter most.

Customer Support Tickets

Support tickets and help desk queries reveal where search is failing. "I searched for X but could not find Y" is a perfect query for your set. These queries are often high-intent, high-frustration, and underrepresented in normal logs because users give up rather than file tickets.

Competitor Analysis

For new products without query logs, analyze what users search for on competitor sites or in related forums. Reddit, Quora, and industry-specific communities are rich sources of how people actually talk about your domain. This is less precise than your own logs but better than synthetic queries.

Manual Curation

Some manual curation is always necessary. You need to fill gaps, add edge cases, and ensure coverage. But manual curation should complement data-driven sampling, not replace it. Think of it as seasoning, not the main ingredient.

Query Set Maintenance: Fighting Query Drift

User behavior changes. Seasonal trends, product launches, marketing campaigns, and cultural shifts all alter what people search for. A query set from January is not representative in December.

Set a schedule for query set updates. Monthly reviews for fast-moving products. Quarterly for stable domains. At minimum, annually. When you update, compare the old and new distributions. If the head queries have shifted significantly, your metrics from old experiments may not generalize.

In OpenSearch Search Relevance, query sets are reusable and versioned. You can keep old query sets for historical comparison while running new experiments on updated sets. This is powerful for long-term trend analysis. You can answer questions like: "Has our search quality improved for holiday queries year over year?"

The Relationship Between Query Sets and Experiments

Once you have a query set, it becomes the input to an experiment. Here is how the flow works in OpenSearch Search Relevance.

You create an experiment and link it to a query set. The plugin iterates through each query in the set, executes it against your configured search (or multiple searches for comparison), and collects the results. These results are then judged (by humans or implicit signals) and metrics are computed.

The query set size directly impacts experiment runtime. A 500-query set with 2 search configurations being compared means 1000 query executions. If each query hits a large index with complex aggregations, this can take significant time. The plugin handles this automatically, but you should plan your experiment windows accordingly.

Query sets also determine the granularity of your insights. A well-structured query set lets you segment results by query type, frequency, or intent. You might discover that your new analyzer improves tail queries by 20% but does nothing for head queries. That is a valuable, specific insight you would miss with a poorly designed query set.

Practical Tips for OpenSearch Search Relevance Users

If you are using the OpenSearch Search Relevance plugin, here are specific tips for query set management.

Store metadata with your query sets. The plugin supports name and description fields. Use them. "Holiday 2025 product queries, 250 queries, sourced from Nov-Dec logs" is more useful than "Query Set 1."

Start small and grow. Begin with a 50-query smoke test to validate your experiment setup. Once you trust the pipeline, expand to 200-500 queries for decision-making.

Segment your query sets. If your product has distinct categories, create category-specific query sets. "Electronics queries" and "Clothing queries" should be separate if your search configurations differ between them.

Link query sets to business metrics. The best query sets are tied to business outcomes. If you know that 20% of your revenue comes from "brand name + product" queries, weight those queries heavily in your set. Your search tuning should prioritize what drives value.

Review query sets before every major experiment. Do not reuse a query set blindly. Check if the queries are still relevant, if new problem queries have emerged, and if your product catalog has changed. A query for a discontinued product is wasted effort.

Common Mistakes to Avoid

Over-reliance on synthetic queries. I have said this before, but it bears repeating. Synthetic queries give you synthetic metrics. Real users do not search like product managers.

Ignoring query frequency. All queries are not equal. A query that happens once a month is not as important as a query that happens a thousand times a day. Your query set should reflect this, either through sampling weights or through explicit frequency-based allocation.

Static query sets. A query set is not a monument. It is a living document. If you have not updated yours in six months, it is probably stale.

Too few judgments per query. A query set with 1000 queries but only 1 judgment per query is less useful than a 100-query set with 10 judgments per query. Judgment depth matters as much as query breadth.

Chasing aggregate metrics blindly. A 0.02 nDCG improvement across 500 queries sounds small. But if it is driven by a 0.15 improvement in your top 20 revenue-driving queries, it is a massive win. Segment your analysis.

Conclusion

Query sets are the foundation of search quality evaluation. Everything else - analyzers, scoring algorithms, experiments, metrics - rests on this foundation. A cracked foundation means shaky conclusions.

In OpenSearch Search Relevance, query sets are first-class objects. The plugin gives you the tools to manage, version, and reuse them. But it cannot tell you what queries to include. That requires understanding your users, your data, and your business.

The best query sets are sampled from reality, maintained over time, and tied to business outcomes. They are not perfect, but they are representative. And representative is enough to make better decisions than guessing.

If you have not reviewed your query set recently, do it now. Your metrics will thank you.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Learning to Rank in Search Relevance: From Feature Engineering to Model Deployment

Prithvi S — Sat, 13 Jun 2026 07:28:32 +0000

Learning to Rank (LTR) transforms search from a hand-tuned relevance function into a machine-learned model that optimizes for business outcomes. Instead of manually setting field weights and boost parameters, LTR uses training data to learn the optimal scoring function from user behavior. This post covers the full pipeline: feature engineering, model training, and deployment in production search systems. The techniques apply to both OpenSearch and Elasticsearch, which share the same underlying Lucene architecture.

The Problem: Why Hand-Tuned Relevance Hits a Ceiling

Traditional search relevance uses a combination of TF-IDF scoring, field boosts, and boolean filters. A typical query might look like this:

{
  "query": {
    "bool": {
      "must": {
        "multi_match": {
          "query": "wireless headphones",
          "fields": ["title^3", "description^2", "brand^1"]
        }
      },
      "filter": {
        "term": { "category": "electronics" }
      }
    }
  }
}

The title^3 boost means matches in the title field are worth 3x more than matches in the brand field. This is an educated guess based on domain knowledge. It might work for 80% of queries, but it fails on edge cases: brand searches where the brand field should dominate, long-tail queries where the description matters more than the title, or queries with synonyms where the boost model does not apply.

The fundamental problem is that a single static weight cannot capture the varying importance of fields across different query types. A user searching for "iPhone 15" cares about exact model matching in the title. A user searching for "good phone for photography" cares about camera specs in the description. A single title^3 boost cannot handle both cases optimally.

When Hand-Tuning Becomes Unmanageable

As a search system grows, the number of query types and field combinations explodes. An e-commerce site might have 50 product categories, each with different relevance patterns. Fashion searches prioritize brand and style. Electronics searches prioritize specs and reviews. Grocery searches prioritize freshness and availability. Maintaining separate boost configurations for each category becomes a maintenance nightmare, and the configurations conflict with each other when a query spans multiple categories.

Learning to Rank solves this by replacing the static boost model with a learned model that adapts to the query context. The model sees the query text, the document fields, and the interaction context (user history, session data, time of day), and produces a relevance score that is optimal for that specific query-document pair.

The LTR Pipeline: Features, Judgments, and Models

The Learning to Rank pipeline has three stages:

Feature extraction - For each query-document pair, compute a set of numeric features that capture relevance signals.
Judgment collection - Gather human or implicit relevance labels for query-document pairs.
Model training - Train a machine learning model to predict relevance scores from features, using the judgments as training targets.

Feature Extraction: What the Model Sees

Features are the input to the LTR model. They must be computable at query time and should capture all signals that might indicate relevance. Common feature categories include:

Query-document text features:

BM25 score for the query against the title field
BM25 score for the query against the description field
BM25 score for the query against the brand field
Exact match count (how many query terms appear verbatim in the document)
Prefix match count (how many query terms match the beginning of a document term)
Synonym match count (how many query terms match via synonym expansion)
TF-IDF score for each query term in each field
Cosine similarity between query vector and document vector (if using dense retrieval)

Document quality features:

Click-through rate (CTR) for the document in the last 30 days
Conversion rate (purchase rate) for the document
Average review score and review count
Document age (how long since it was added to the index)
Inventory availability (in-stock or out-of-stock)
Price and price percentile within the category
Popularity rank (how many times the document was viewed)

Query context features:

Query length (number of terms)
Query category intent (electronics, fashion, etc.) via a classifier
User location (for geo-relevant searches)
Time of day and day of week (for temporal relevance)
User segment (new user vs returning user, premium vs basic)
Session history (what the user searched for and clicked in this session)

Interaction features:

Position bias (documents ranked higher get more clicks even if they are less relevant)
Previous query-document interactions (has the user clicked this document before?)
Dwell time (how long the user spent on the document page after clicking)
Skip rate (how often users skipped over this document without clicking)

In practice, a production LTR system might use 50-200 features. The feature set must be comprehensive enough to capture relevance signals but not so large that model training becomes slow or overfitting occurs. Feature selection and regularization are critical for model quality.

Judgment Collection: Explicit and Implicit Labels

Judgments are the training targets. Each query-document pair needs a relevance label. The labels are typically ordinal: 0 (irrelevant), 1 (somewhat relevant), 2 (relevant), 3 (highly relevant). Some systems use binary labels (relevant/irrelevant) or continuous labels (expected click probability).

Explicit judgments come from human annotators. A panel of judges evaluates query-document pairs and assigns relevance labels. Explicit judgments are accurate but expensive. A typical e-commerce site might need 10,000 judged query-document pairs for a category, which costs $5,000-10,000 in annotation fees. The judgment process must be carefully controlled: judges need guidelines, inter-annotator agreement must be measured (Cohen's kappa > 0.6 is considered acceptable), and edge cases must be escalated to senior annotators.

Implicit judgments come from user behavior. If a user searches for "wireless headphones" and clicks the third result, the implicit signal is that the third result was more relevant than the first and second results (which were skipped). This is the foundation of click-through based LTR. However, implicit judgments are noisy because of position bias: users click higher-ranked results more often regardless of relevance. To correct for position bias, LTR systems use click models like the Cascade Model or the Position-Based Model (PBM) that estimate the probability of a click given the position and the true relevance.

Hybrid approaches combine explicit and implicit judgments. Explicit judgments are used for a small set of carefully selected queries (head queries that drive 80% of traffic). Implicit judgments are used for the long tail. The model is trained on the combined dataset, with explicit judgments weighted more heavily because they are more reliable.

Model Training: Pointwise, Pairwise, and Listwise Approaches

LTR models are trained using one of three paradigms:

Pointwise treats each query-document pair as an independent sample. The model learns to predict the relevance label directly, like a standard regression or classification problem. The loss function is mean squared error (for continuous labels) or cross-entropy (for ordinal labels). Pointwise models are simple to train but ignore the ranking context: they do not know that a query has multiple documents and that the goal is to order them correctly.

Pairwise treats each pair of documents for the same query as a training sample. The model learns to predict which document is more relevant. The loss function is typically a hinge loss or logistic loss that penalizes the model when it incorrectly orders a pair. Pairwise models capture the relative ordering signal but still do not optimize for the full list quality.

Listwise treats the entire ranked list for a query as a training sample. The model optimizes a list-level metric like Normalized Discounted Cumulative Gain (NDCG) or Mean Average Precision (MAP). Listwise models are theoretically optimal because they directly optimize the ranking metric, but they are computationally expensive and harder to train. LambdaMART and its neural variant LambdaRank are the most popular listwise algorithms.

LambdaMART is a gradient boosting model that optimizes NDCG. It works by computing the "lambda" gradient for each document: the change in NDCG that would occur if the document's score changed by a small amount. The lambda gradient focuses on pairs that are incorrectly ordered and are near the top of the ranking, because those pairs have the largest impact on NDCG. LambdaMART then trains a decision tree to predict these lambda gradients, and the tree ensemble is updated iteratively.

For production deployment, LambdaMART is often the best choice because it provides a good balance between accuracy and inference speed. A typical LambdaMART model might have 100-500 trees with a maximum depth of 6-8. Inference is fast because each tree is a simple decision tree, and the ensemble prediction is the sum of tree outputs.

Feature Engineering for Search: Practical Techniques

Feature engineering is the most critical and time-consuming part of LTR. The model can only be as good as the features it sees. Here are practical techniques for building high-quality feature sets.

Query Classification for Intent-Aware Features

Not all queries are the same. A user searching for "iPhone 15" has a navigational intent: they know what they want and are looking for a specific product. A user searching for "good phone for photography" has an informational intent: they are researching options. A user searching for "cheap phone" has a transactional intent: they want to compare prices.

Query classification adds a feature that captures the intent. A simple classifier can be trained on query text alone: short queries with brand names are navigational, long queries with descriptive terms are informational, queries with price-related terms are transactional. The intent feature interacts with other features: for navigational queries, the exact title match score is highly weighted. For informational queries, the description BM25 score and review quality features are more important.

Normalization and Scaling

Features have different scales. BM25 scores might range from 0 to 30, while CTR ranges from 0 to 0.1. If the model is a neural network, feature scaling is critical. If the model is a tree-based model like LambdaMART, scaling is less important because trees are scale-invariant. However, even for trees, extreme outliers can cause the model to overfit to rare cases. Log-transforming CTR (log(1 + CTR)) and clipping BM25 scores to a maximum of 50 are common preprocessing steps.

Temporal Features and Recency Bias

For time-sensitive content (news, social media, product launches), recency is a strong relevance signal. A simple recency feature is 1 / (days_since_publication + 1). But recency should not dominate all queries. A query for "iPhone 15" should not be affected by recency because the iPhone 15 is a specific product. A query for "best phone 2024" should be affected by recency because newer reviews are more relevant. The interaction between query intent and recency features can be captured by adding a query-specific recency weight, which the model learns from the training data.

Cross-Field Features

A query might match multiple fields, and the combination of matches is a signal. For example, a query "Nike running shoes" that matches the brand field ("Nike") and the category field ("running shoes") is more relevant than a query that only matches the title field. A cross-field feature can be the product of the brand match score and the category match score, or a binary indicator that both fields matched. These interaction features capture the semantic coherence of the query-document match.

Model Deployment: From Training to Query-Time Scoring

Once the model is trained, it must be deployed to the search engine. The deployment architecture depends on whether the model is a simple linear model, a tree ensemble, or a neural network.

Linear Model Deployment: Rescoring at the Coordinator

A linear model has the form score = w1 * f1 + w2 * f2 + ... + wn * fn. The weights are learned during training. Deployment is straightforward: the search engine computes the features for each query-document pair and applies the linear combination. This can be done in a rescore phase at the coordinator level, which operates on the top-N results from the initial query phase.

The rescore phase in OpenSearch and Elasticsearch allows a custom script to re-rank the top results. A Painless script can compute the linear combination of features. However, Painless is not optimized for complex models, and the script execution overhead can be significant. For linear models with 50 features, a custom rescore query is usually fast enough. For tree ensembles, the script becomes unwieldy because each tree requires multiple if-else branches.

Tree Ensemble Deployment: Native Plugin or External Service

Tree ensembles like LambdaMART are harder to deploy because each tree is a series of if-else decisions. A 100-tree model with depth 6 has 600 decision nodes. Evaluating this in a Painless script is slow and error-prone. The standard approach is to use a native plugin that can evaluate the tree ensemble efficiently.

OpenSearch has the LTR plugin (formerly Elasticsearch LTR plugin) that provides native support for tree ensemble models. The plugin stores the model in the cluster state and exposes a sltr (search-learning-to-rank) query type. The query type computes features using the standard query DSL and then applies the model to score the results. The plugin supports XGBoost, LightGBM, and RankLib model formats.

For Elasticsearch, the LTR plugin was historically available but has been less maintained. An alternative is to deploy the model in an external service. The search engine returns the top 1000 results with all features as metadata. The external service evaluates the model and returns the re-ranked list. The trade-off is latency: an external service call adds 5-50ms depending on network latency. For applications where latency is critical, the native plugin is preferred.

Neural Network Deployment: ONNX Runtime Integration

Neural LTR models (e.g., BERT-based cross-encoders that jointly encode the query and document) are too complex for tree-based evaluation. These models require a deep learning runtime. The ONNX Runtime is a common choice because it supports models from PyTorch, TensorFlow, and other frameworks. Deployment options include:

Native ONNX plugin - Some search engines have plugins that embed the ONNX runtime. The plugin evaluates the model at query time within the search process. This is fast but requires the plugin to be maintained and compatible with the search engine version.
External inference service - A separate microservice hosts the ONNX model and receives query-document pairs via gRPC or HTTP. The search engine calls the service for the top-N results. This is flexible but adds network latency.
Pre-computed embeddings - For dense retrieval models, the document embeddings are pre-computed at index time and stored in a vector field. The query embedding is computed at query time, and the search engine performs a k-NN vector search. This is the fastest approach because the model inference is done once per query, not once per document. However, it only works for bi-encoder models (where query and document are encoded separately), not cross-encoder models (where the query and document are encoded together).

Training Data and Evaluation

Collecting Training Data from Production Logs

Production logs are the primary source of training data for LTR. The log pipeline should capture:

Query text and timestamp
All results returned for the query (with their positions)
All clicks (with their positions and dwell times)
Conversions (purchases, sign-ups, etc.) linked to clicks
User context (user ID, session ID, location, device)

The logs must be joined and processed to create query-document pairs with features and labels. The processing pipeline typically runs in batch mode (e.g., daily) and uses Spark or Flink to aggregate the logs. The output is a training dataset in a format like SVMLight or LibSVM, where each line is a query-document pair with features and a label.

Handling Position Bias in Click Data

Position bias is the biggest challenge in implicit judgment collection. A result at position 1 gets clicked 10-20% of the time even if it is irrelevant, while a result at position 10 gets clicked 1-2% of the time even if it is highly relevant. Without correction, the model learns to rank documents higher simply because they were ranked higher before, creating a self-reinforcing loop.

The standard correction is the Position-Based Model (PBM). PBM assumes that the probability of a click at position k is the product of two probabilities: the probability that the user examines the position (which decreases with position) and the probability that the document is relevant given the query. The examination probability is estimated from the data by observing how often each position is clicked across all queries. The relevance probability is then estimated as click_probability / examination_probability.

For example, if position 1 is examined 80% of the time and clicked 20% of the time, the estimated relevance is 20% / 80% = 25%. If position 5 is examined 40% of the time and clicked 10% of the time, the estimated relevance is 10% / 40% = 25%. Both positions have the same relevance estimate, which is correct even though the raw click rates differ.

PBM requires enough data to estimate the examination probabilities accurately. For a new search system with low traffic, the estimates are noisy. In this case, explicit judgments are needed until the click volume is sufficient. A common threshold is 1,000 clicks per query position before the PBM estimates are reliable.

Evaluation Metrics: NDCG, MAP, and MRR

The quality of an LTR model is evaluated using ranking metrics. The most common metric is NDCG (Normalized Discounted Cumulative Gain), which measures the quality of the ranked list by assigning higher credit to relevant documents that appear higher in the list. NDCG is computed as:

Compute the DCG: DCG = sum((2^relevance - 1) / log2(position + 1)) for all positions.
Compute the ideal DCG (IDCG): the DCG of the perfect ranking.
NDCG = DCG / IDCG.

NDCG ranges from 0 to 1, where 1 is a perfect ranking. A typical improvement from a baseline model to an LTR model is 0.05-0.15 NDCG points. For a search engine with 1 million queries per day, a 0.1 NDCG improvement translates to a significant increase in click-through rate and conversion.

MAP (Mean Average Precision) is another common metric, particularly for binary relevance. It computes the average precision at each position where a relevant document appears, then averages across queries. MAP is less sensitive to the exact ordering of relevant documents than NDCG, but it is easier to interpret for non-experts.

MRR (Mean Reciprocal Rank) is used for tasks where only the first relevant document matters, such as question answering or navigational search. MRR is the average of 1 / rank_of_first_relevant across queries. A perfect MRR is 1.0, which means the first result is always relevant.

A/B Testing: The Final Validation

Offline metrics like NDCG are useful for model development, but they do not guarantee business impact. The ultimate validation is an A/B test in production. The test should split traffic between the baseline model (hand-tuned boosts) and the LTR model. The metrics to track are:

Click-through rate (CTR) - the percentage of queries that result in a click.
Conversion rate - the percentage of clicks that result in a purchase or other goal.
Revenue per query - total revenue divided by number of queries.
Dwell time - average time spent on clicked pages.
Zero-result rate - percentage of queries that return no results.

The A/B test should run for at least 2 weeks to capture weekly patterns and should include at least 10,000 queries per variant to achieve statistical significance. The winner should be determined by the primary business metric (usually revenue per query or conversion rate), not just CTR. A model that increases CTR but decreases conversion is driving clicks to less relevant results, which is worse than the baseline.

Common Pitfalls and Production Considerations

Feature Drift and Model Retraining

Features change over time. A new product might have no CTR data initially, but after a month it has enough clicks to become a strong feature. Seasonal products (holiday decorations, swimwear) have CTR patterns that change throughout the year. If the model is trained on summer data, it will underperform in winter because the feature distributions have shifted.

Model retraining should be automated. A typical schedule is weekly retraining for fast-moving catalogs and monthly retraining for stable catalogs. The retraining pipeline should use a sliding window of data (e.g., the last 30 days) to keep the model current. Automated evaluation should compare the new model's offline NDCG against the current model's NDCG, and only deploy if the improvement exceeds a threshold (e.g., 0.01 NDCG). This prevents deploying models that are worse than the current one due to data quality issues or training instability.

The Cold Start Problem

New documents have no historical features (CTR, review count, popularity). The model cannot evaluate them accurately. This is the cold start problem. Common solutions include:

Default feature values - Use the average CTR and review count for the category as default values. This gives new documents a fair chance to rank based on their text features.
Exploration - Randomly promote new documents to higher positions for a small percentage of traffic to collect click data. This is a form of multi-armed bandit optimization that balances exploration (learning about new documents) and exploitation (ranking known good documents).
Content-based features - Use text features and document quality features that do not depend on historical data. A new document with a strong title match and good description should still rank reasonably well even without CTR data.

Latency Budget and Feature Computation

LTR adds latency to search. Feature computation requires running multiple queries (one per feature) or extracting features from the document metadata. A model with 100 features can add 50-200ms to query time if features are computed naively. The latency budget is critical for production.

Optimization techniques include:

Pre-computed features - Store feature values in the index at indexing time. For example, the CTR and review count can be updated daily and stored as numeric fields. The query time feature computation is then a simple doc value lookup.
Feature caching - Cache feature values for frequently accessed documents. A document that appears in the top 10 results for 100 queries per day should have its features cached.
Feature selection - Reduce the feature set to the most impactful features. A model with 20 well-chosen features often performs nearly as well as a model with 100 features, and the latency is much lower.
Asynchronous feature computation - Compute features in parallel using multiple threads. OpenSearch and Elasticsearch support parallel query execution for some feature types.

Fairness and Bias in LTR

LTR models can inadvertently learn biases from the training data. If the training data reflects historical bias (e.g., products from dominant brands get more clicks because they are ranked higher), the model will amplify that bias. Fairness-aware LTR is an active research area, but practical steps include:

Demographic parity constraints - Ensure that the ranking distribution is similar across different groups (e.g., small brands vs large brands).
Counterfactual evaluation - Evaluate what the ranking would have been if the model had been trained on unbiased data, using inverse propensity scoring to correct for historical bias.
Diverse result sets - Enforce diversity constraints that ensure the top results include a mix of brands, prices, and styles, rather than a homogeneous set that the model over-optimized.

Conclusion

Learning to Rank transforms search relevance from a manual craft into a data-driven science. The pipeline - feature extraction, judgment collection, model training, and deployment - requires careful engineering at each stage. Feature engineering is the most important step: the model can only learn from the signals you provide. Judgment collection must correct for position bias and combine explicit and implicit labels. Model training should use listwise optimization for ranking quality. Deployment must respect the latency budget and be monitored for drift and bias.

For search teams, LTR is a force multiplier. A well-trained LTR model can improve NDCG by 10-20% and business metrics by 5-15%, which translates to millions in revenue for large-scale e-commerce. But the investment is significant: feature engineering, annotation pipelines, model training infrastructure, and A/B testing culture are all required. The teams that succeed are those that treat LTR as a product, not a one-time project, with continuous retraining, evaluation, and iteration.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.

Elasticsearch Cluster State Management: How 1000 Nodes Agree on One Source of Truth

Prithvi S — Sat, 13 Jun 2026 07:28:14 +0000

Elasticsearch cluster state is the single source of truth that every node in the cluster must agree on. It contains the routing table, index metadata, mapping definitions, settings, and allocation decisions. When a cluster has 100 nodes and 10,000 shards, the cluster state can grow to tens of megabytes, and publishing it to every node becomes a significant bottleneck. Understanding how Elasticsearch manages, publishes, and updates cluster state is essential for operating large clusters at scale.

What Is Cluster State and Why It Must Be Consistent

The cluster state is a JSON-like data structure that lives in the cluster manager node (formerly called master node). It contains everything the cluster needs to route requests, allocate shards, and enforce mappings. The key components are:

Routing table - Maps each shard to a specific node. Every index-shard combination has an entry indicating which node holds the primary, which nodes hold replicas, and whether the shard is started, initializing, or relocating.
Index metadata - Includes index settings, mappings, aliases, and lifecycle policies. When a mapping is updated, the index metadata changes, which triggers a cluster state update.
Cluster settings - Persistent and transient settings that affect cluster behavior, such as allocation thresholds, recovery rates, and circuit breaker limits.
Blocks - Index-level or cluster-level blocks that prevent certain operations. For example, a read-only block prevents writes to an index while a cluster-level block prevents index creation.
Custom metadata - Plugins and modules can store their own data in the cluster state. Snapshot repositories, ingest pipelines, and transform configurations all live here.

Every node maintains a full copy of the cluster state. When a node receives a search request, it uses the local cluster state copy to determine which shards to query. It does not ask the manager node for routing information on every request. This design makes request routing fast but requires all nodes to have an up-to-date cluster state.

The Cluster State as a Versioned Document

Elasticsearch treats cluster state as a versioned document. Each update increments the version number. When the manager publishes a new cluster state, it includes the version number. Nodes compare the incoming version with their local version and apply the update only if the incoming version is newer. This prevents out-of-order updates from causing inconsistency.

The version number is a 64-bit integer that increments on every change, no matter how small. Adding a single alias to one index increments the version. Changing a single setting on one index increments the version. This means a busy cluster with frequent index creation or mapping updates can generate thousands of cluster state versions per hour. The version number itself does not wrap around in practice, but the rate of version increments is a useful metric for monitoring cluster state churn.

Cluster State Publication: The Diff-Based Update Mechanism

When the manager node updates the cluster state, it does not send the full state to every node. Instead, it computes a diff between the previous state and the new state, and sends only the changed parts. This diff-based approach is critical for performance because a full cluster state for a large cluster can be 50MB or more, while a typical diff might be only a few kilobytes.

How the Manager Computes and Sends Diffs

The manager maintains the previous cluster state in memory. When a new state is computed, it serializes both the old and new states to compressed binary format and computes a binary diff. The diff format is a sequence of operations: add this key, update this value, delete this entry. The manager then sends the diff to every node in the cluster.

The diff computation is CPU-intensive. For a 20MB cluster state with a small change, the diff might take 5-10ms to compute. For a 50MB state with a large change (e.g., a new index with 100 shards), the diff might take 50-100ms. During this time, the manager node is blocked and cannot process other cluster state updates. This is why cluster state computation is single-threaded and why batching multiple changes into a single update is more efficient than applying them one by one.

Node-Level Application: From Diff to Local State

When a non-manager node receives a diff, it applies the operations to its local cluster state copy. The application process is:

Deserialize the diff into a sequence of operations.
Validate the diff against the local state to ensure the base version matches. If the local version is behind, the node might have missed a previous update. In this case, it requests the full cluster state from the manager instead of applying the diff.
Apply the operations to the local state. Each operation is a mutation: add a routing entry, update a mapping, remove an alias.
Verify the resulting state by computing a checksum and comparing it to the checksum provided by the manager. If the checksums do not match, the node has a corrupted state and must request the full state.
Acknowledge the update to the manager. The manager waits for acknowledgments from a majority of nodes before considering the state published.

The acknowledgment requirement is the key to consistency. The manager considers a cluster state published when it has received acks from a majority of the voting nodes. This ensures that even if the manager fails immediately after publication, the new state is durable on enough nodes that the next elected manager will see it.

The Cluster Manager Role: Election and Responsibilities

The cluster manager is the node responsible for computing and publishing cluster state updates. It is not a special node type - any node can be elected manager, but the election process favors nodes with the node.roles: [cluster_manager] setting. Only manager-eligible nodes participate in voting.

Manager Election: Voting and Quorum

Manager election uses a voting configuration that is a subset of manager-eligible nodes. The default voting configuration includes all manager-eligible nodes, but administrators can restrict it to a smaller set for stability. A candidate must receive votes from a majority of the voting configuration to become manager. If there are 5 voting nodes, a candidate needs 3 votes. If there are 4 voting nodes, a candidate needs 3 votes (majority of 4 is 3, not 2).

The election is triggered when the current manager fails or steps down. Nodes exchange ping messages to detect manager failure. If a node does not receive a ping response from the manager within cluster.publish.timeout (default 30 seconds), it initiates an election. All nodes in the voting configuration then vote for the candidate with the highest cluster state version. If multiple nodes have the same version, the node with the lowest ID wins. This tiebreaker ensures that elections resolve quickly without requiring additional rounds.

The Bootstrap Requirement: Preventing Split-Brain

When a cluster starts for the first time, it must be bootstrapped with a set of manager-eligible nodes. This is done via the cluster.initial_master_nodes setting. The bootstrap requirement ensures that a cluster does not accidentally form around a subset of nodes that happen to start first. For example, if a cluster has 5 nodes but only 3 are started initially, the 3 nodes will form a cluster only if they are explicitly listed as initial master nodes. The remaining 2 nodes will join the existing cluster when they start, rather than forming a separate cluster.

Without the bootstrap requirement, a network partition could split a 5-node cluster into a 3-node cluster and a 2-node cluster, both electing their own managers. This is the classic split-brain problem. The bootstrap requirement prevents it by requiring explicit agreement on the initial node set. After bootstrap, the voting configuration evolves dynamically as nodes join and leave, but the majority requirement ensures that only one manager can exist at a time.

Cluster State Size and the Bloat Problem

The cluster state grows with every index, shard, alias, mapping, and setting. For a cluster with 1,000 indices and 5 shards per index, the routing table alone has 5,000 entries. Each entry includes the node assignment, shard state, and recovery statistics. The index metadata includes the mapping for every field, which can be hundreds of fields per index. At scale, the cluster state can become a performance bottleneck.

Symptoms of Cluster State Bloat

Slow manager election - When the manager fails, the election requires nodes to compare cluster state versions. Large states take longer to serialize and compare, delaying the election.
Slow publication - Diffs take longer to compute and send. Nodes take longer to apply them. The manager spends more time blocked on publication, reducing throughput.
High memory usage - Every node holds a full copy of the cluster state. A 50MB cluster state on 100 nodes consumes 5GB of memory across the cluster, just for state copies.
Slow RESTART recovery - When a node restarts, it must receive the full cluster state from the manager. A large state takes longer to transfer and apply, delaying the node's return to the cluster.

The Pending Tasks Queue

The manager maintains a pending tasks queue for cluster state updates. When multiple updates arrive simultaneously, they are queued and processed sequentially. The queue is visible via the GET _cluster/pending_tasks API. A growing queue indicates that the manager cannot keep up with the rate of updates.

Common causes of queue growth:

Rapid index creation - Creating 100 indices in a loop generates 100 cluster state updates. Each update requires diff computation and publication. If the loop runs faster than the manager can process, the queue grows.
Mapping explosions - Dynamic mapping that creates a new field for every unique key in incoming documents causes frequent mapping updates. Each update triggers a cluster state increment.
Shard allocation storms - Node failures or restarts trigger shard reallocation, which updates the routing table. A rolling restart of 50 nodes in a 100-node cluster generates hundreds of routing updates as shards move to maintain replica counts.
Snapshot and restore operations - Restoring a snapshot creates new indices, which triggers cluster state updates. Restoring 100 indices simultaneously can overwhelm the manager.

Diagnosing Cluster State Size

The GET _cluster/state API returns the full cluster state, but it can be too large to return directly. Instead, use the GET _cluster/state/_all/_all API with filtering to get specific sections. The GET _cluster/health API returns the cluster state version and the number of pending tasks, which are useful metrics for monitoring.

For detailed analysis, the GET _cluster/state/metadata API returns only the metadata section, which is usually the largest part of the cluster state. The GET _cluster/state/routing_table API returns only the routing table. By comparing the sizes of these sections, you can identify which part of the state is growing.

Reducing Cluster State Size: Practical Strategies

Mapping Compression and Static Mapping

Dynamic mapping is convenient but dangerous for cluster state size. Every unique field name in every document creates a mapping entry. If an index receives documents with 10,000 unique keys, the mapping has 10,000 fields. With 1,000 indices, the total field count is 10 million, which can make the cluster state hundreds of megabytes.

The solution is to disable dynamic mapping and use explicit mapping definitions. Set dynamic: false or dynamic: strict in the index mapping. dynamic: false ignores unknown fields (they are stored in _source but not indexed). dynamic: strict rejects documents with unknown fields. Both prevent mapping explosions.

For indices that need dynamic fields but not at scale, use index.mapping.total_fields.limit to set a maximum field count per index. The default is 1,000, which is usually sufficient for structured data but too high for log data with unpredictable keys. For log indices, a limit of 100-200 is often enough.

Index Lifecycle Management and Rollover

Time-series indices (logs, metrics) should use rollover policies to limit the number of active indices. The Index Lifecycle Management (ILM) feature can automatically roll over an index when it reaches a certain size or age, and delete or freeze old indices. Rollover reduces the number of indices in the cluster state, which reduces the routing table and metadata size.

Frozen indices are a special case. A frozen index is stored in a minimal state that uses only a few hundred bytes of cluster state per index, compared to several kilobytes for a normal index. Frozen indices cannot be searched directly but can be mounted and searched via the Frozen Searchable Snapshots feature. For long-term data retention, freezing indices that are older than 30 days is a common strategy.

Shard Count Reduction

The routing table has one entry per shard. Reducing the shard count reduces the routing table size. For indices with 5 primary shards and 1 replica, each index contributes 10 routing entries. With 1,000 indices, the routing table has 10,000 entries. If the indices are reduced to 1 primary shard with 1 replica, the routing table has 2,000 entries - an 80% reduction.

The trade-off is that fewer shards per index reduces parallelism for large queries. A query that scans 100 million documents runs faster on 5 shards than on 1 shard because each shard can process a subset in parallel. The optimal shard count depends on the index size and query patterns. A general rule of thumb is 30-50GB per shard for time-series data and 10-20GB per shard for search-heavy indices. If an index is 100GB, 2-3 shards are usually sufficient.

Alias Consolidation

Aliases are stored in the index metadata. Each alias has a name, a filter (optional), and routing parameters (optional). For indices with hundreds of aliases, the metadata section can become large. Consolidating aliases by using index patterns (e.g., logs-2024-* instead of logs-2024-01, logs-2024-02) reduces the alias count. The _aliases API supports bulk operations for alias management, but alias consolidation is usually a one-time manual task during cluster maintenance.

Cluster State Updates and the Acking Mechanism

When the manager publishes a cluster state update, it waits for acknowledgments from the nodes. The cluster.publish.timeout setting controls how long the manager waits. If the timeout is reached before all nodes ack, the manager considers the update partially published. The nodes that did not ack are marked as failed and may be removed from the cluster if they miss multiple updates.

The Minimum Acknowledgment Requirement

The manager does not wait for all nodes to ack. It waits for a majority of the voting nodes. This is the same majority required for manager election. The requirement ensures that the cluster state is durable on enough nodes to survive a manager failure. Non-voting nodes (data-only nodes) are not required to ack, but they still receive the update. If a data-only node misses the update, it will request the full state on the next update.

Handling Slow Nodes

A slow node can delay cluster state publication if the manager waits for its ack. The cluster.publish.timeout setting limits this delay, but if the timeout is reached, the slow node is flagged. If the node is consistently slow, it may indicate a network problem, a GC pause, or a disk I/O bottleneck. The manager logs warnings about slow acks, and the node should be investigated.

Slow nodes can also cause cluster state divergence. If a node misses an update and the next update is a diff based on the missed version, the node will detect the version mismatch and request the full state. This triggers a full state transfer, which is expensive for large states. Repeated divergence events can saturate the manager's network and CPU.

Common Pitfalls and Edge Cases

The Too-Many-Indices Problem

Elasticsearch has a practical limit on the number of indices, not a hard limit. The limit is determined by cluster state size and manager capacity. For a cluster with 100 nodes and a 10GB heap, the practical limit is typically 5,000-10,000 indices. Beyond this, the cluster state publication becomes slow, and the manager spends most of its time computing diffs and waiting for acks.

The solution is to reduce the number of indices. Use rollover to merge small indices into larger ones. Use aliases to present a unified view of merged indices. For time-series data, use data streams, which automatically manage rollover and aliasing. Data streams are a higher-level abstraction that reduces the number of indices the operator manages directly.

The Mapping Update Race Condition

When two clients update the mapping of the same index simultaneously, the manager receives two cluster state updates. Because updates are processed sequentially, the first update succeeds and the second update fails with a version conflict. The second client must retry. This is a safe behavior but can cause retries to pile up if the mapping update rate is high.

To reduce mapping update contention, batch updates and use the dynamic: strict setting. If the application does not need dynamic fields, disable them entirely. If the application does need dynamic fields but only a known set, use dynamic_templates to pre-define the mappings for expected field patterns, which reduces the need for explicit mapping updates.

The Cluster State Snapshot and Restore

Elasticsearch can snapshot the cluster state to a repository, but the snapshot is not a complete backup. It includes index metadata, settings, and aliases, but not the routing table. The routing table is reconstructed when the cluster restarts. This means restoring a cluster state snapshot does not restore the exact node-shard assignments. Shards will be reallocated according to the current allocation rules, which may result in different node assignments than the original cluster.

For complete disaster recovery, snapshot the cluster state and the index data separately. The cluster state snapshot restores the metadata, and the index data snapshots restore the documents. After restore, the cluster will allocate shards and rebuild the routing table from scratch. This is usually sufficient because the routing table is derived from the metadata and the node list, not an independent state.

Conclusion

Elasticsearch cluster state management is a distributed consensus problem disguised as a metadata service. The manager node maintains the authoritative state, computes diffs, and publishes them to all nodes. The diff-based update mechanism keeps network traffic low, but the computation cost is borne by the manager. As clusters grow, cluster state size becomes the limiting factor, and operators must actively manage it through mapping discipline, index lifecycle policies, and shard count optimization.

The acknowledgment mechanism ensures that state updates are durable, but it also introduces latency that grows with cluster size. A slow node can delay publication for all nodes. The voting configuration and majority requirement prevent split-brain but require careful management during node additions and removals. The bootstrap requirement ensures that new clusters form correctly, but it is a one-time configuration that must be set before the cluster starts.

For production clusters, monitor the cluster state version rate, the pending tasks queue, and the cluster state size. If the version rate exceeds 100 per hour or the state size exceeds 20MB, investigate the cause. Mapping explosions, rapid index creation, and shard allocation storms are the usual suspects. Fixing these issues early prevents the cluster from reaching a state where the manager cannot keep up, which is a failure mode that requires downtime to resolve.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.