Elasticsearch / OpenSearch for Data Engineers: Ingest Pipelines, Mappings & Analyzers

elasticsearch for data engineers is a different job from Elasticsearch for backend developers, and the interview probes the difference relentlessly — because the data engineer is the one who owns the shape of the index, the transform on the way in, and the scaling math, while the app team merely writes queries against whatever you built. A search or observability cluster is not a magic box you throw JSON at; it is an ETL sink with a strongly-typed schema (the mapping), a per-field text-processing stage (the analyzer), an in-cluster transform layer (ingest pipelines), and a horizontal-scaling substrate (shards and replicas) — and every one of those four surfaces is a place where a data engineer either prevents a production incident or causes one. Get the mapping wrong and you reindex a billion documents. Get the analyzer wrong and searches silently return nothing. Get the shard count wrong and the cluster goes red at 3 AM.

This guide is the data-engineering walkthrough you wished existed the first time an interviewer asked "explain the inverted index and why text and keyword are different types", or "your team is dumping raw NGINX logs into an index and the mapping keeps exploding — what do you do?", or "walk me through how you'd backfill 500 million documents through an ingest pipeline without melting the cluster." It covers the five things a data engineer must own: why the search cluster is your ETL sink and how the opensearch fork changed the licensing landscape, mappings and the inverted index (field types, dynamic vs explicit mapping, index templates), analyzers (tokenizers, token filters, the _analyze API, custom analyzers), ingest pipelines and the bulk api (processors, enrich, reindex, throughput), and shards and replicas (sizing, aggregations, and when not to reach for Elasticsearch at all). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the streaming practice library →, and sharpen the aggregation axis with the real-time analytics practice library →.

---

On this page

• Why data engineers own the search cluster
• Mappings and the inverted index
• Analyzers — tokenizers, filters, and the analyze API
• Ingest pipelines and the bulk API
• Shards, scaling and interview signals
• Cheat sheet — Elasticsearch / OpenSearch recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why data engineers own the search cluster

The index is an ETL sink with a schema — the mapping, the analyzer, and the ingest pipeline are all transform stages you own

The one-sentence invariant: an Elasticsearch or OpenSearch index is a strongly-typed ETL sink whose schema is the mapping, whose per-field text transform is the analyzer, whose in-cluster reshape stage is the ingest pipeline, and whose horizontal-scaling unit is the shard — so the data engineer who feeds it owns four separate design decisions that the app team consuming it never sees, and each decision is expensive-to-irreversible once documents are written. The application developer writes GET /orders/_search {...} and gets results; they never think about whether status is a keyword or a text field, whether the analyzer lowercased the token, whether the timestamp was parsed by an ingest date processor, or whether the index has 5 shards or 500. Those are the data engineer's decisions, and getting any of them wrong ships a bug that only surfaces at scale or during an audit.

What the data engineer actually owns.

• The mapping. The typed schema of the index — which fields exist, what type each is (text, keyword, long, date, boolean, nested), whether they are indexed, and how they are analyzed. Changing an existing field's type requires a reindex. This is a schema-migration problem exactly like a relational ALTER TABLE, except worse because you usually cannot alter in place.
• The analyzer. The pipeline that turns a text field's string into the terms stored in the inverted index. It decides what a query can ever match. The analyzer is a data-modeling decision, not a query-time tweak.
• The ingest pipeline. The in-cluster transform stage that runs before a document is indexed — parse a log line with grok, rename fields, convert strings to numbers, enrich with a lookup table, drop bad records. This is the "T" of ETL living inside the search cluster.
• Shards, replicas, and lifecycle. How the index is split across nodes for horizontal scale (primary shards), how it is made highly available (replica shards), and how it ages out (index lifecycle management — hot/warm/cold/delete). Getting the shard count wrong is the single most common cause of an unhealthy cluster.

The ES vs OpenSearch fork — the context every interviewer expects you to know.

• 2021 — the license split. Elastic relicensed Elasticsearch and Kibana from Apache 2.0 to a dual SSPL / Elastic License in early 2021, largely in response to AWS offering a managed Elasticsearch service. AWS responded by forking the last Apache-2.0 version and creating OpenSearch (and OpenSearch Dashboards from Kibana), now stewarded under the Linux Foundation's OpenSearch Software Foundation.
• 2024 — Elastic adds AGPL. Elastic added the AGPLv3 as a third license option for Elasticsearch, making it OSI-approved open source again — but the fork had already happened and OpenSearch had its own momentum, so both products now exist and diverge.
• What's the same. The core Lucene-based inverted index, the mapping/analyzer/ingest/shard model, the bulk API, the query DSL basics — everything in this guide applies to both. A data engineer who understands the model works on either.
• What diverges. Newer features drift apart: Elastic's ES|QL, some vector-search and ML features, security defaults, and licensing of specific plugins. For a data engineer, the operational model is 95% identical; the divergence is in advanced query languages and proprietary ML, not in mappings/analyzers/ingest/shards.

What interviewers listen for.

• Do you frame the index as "a typed ETL sink, not a query engine" — naming mapping, analyzer, ingest, and shards as the four things you own? — senior signal.
• Do you know the ES vs OpenSearch fork story (2021 relicense → AWS fork → 2024 AGPL) without reciting marketing? — senior signal.
• Do you say "changing a field type requires a reindex" the moment mappings come up? — required answer.
• Do you describe the inverted index as "term → posting list" rather than "it's like a database index"? — senior signal.
• Do you name at least one case where Elasticsearch is the wrong tool (source of truth, transactional writes, relational joins)? — senior signal.

Worked example — the four-surface ownership map

Detailed explanation. The most useful artifact for an Elasticsearch data-engineering interview is a memorised map of the four surfaces you own, what each does, and what the cost of getting it wrong is. Walk through building the map for a hypothetical events index that ingests application logs and clickstream events destined for both search (Kibana / Dashboards) and aggregation (dashboards, alerting).

• Source. JSON events from a Kafka topic — { "@timestamp", "level", "service", "message", "user_id", "latency_ms", "url" }.
• Consumers. Ops engineers searching message full-text; dashboards aggregating latency_ms by service; alerting on level = "ERROR" counts.
• Scale. ~50 GB/day, 30-day retention, ~1.5 TB hot.

Question. Map each of the four surfaces to the concrete decision it forces for the events index, and note the cost of getting each wrong.

Input.

Surface	Decision it forces	Cost of getting it wrong
Mapping	message = text, service/level = keyword, latency_ms = long, @timestamp = date	full reindex of the index
Analyzer	how message is tokenised (lowercase, stopwords, stemming)	searches silently miss / over-match
Ingest pipeline	parse url into components, convert types, drop bad docs	garbage or wrong-typed data in the index
Shards / ILM	shard count per daily index, replica count, rollover + retention	red cluster, wasted heap, or lost HA

Code.

PUT _index_template/events-template
{
  "index_patterns": ["events-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.default_pipeline": "events-ingest"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "level":      { "type": "keyword" },
        "service":    { "type": "keyword" },
        "message":    { "type": "text" },
        "user_id":    { "type": "keyword" },
        "latency_ms": { "type": "long" },
        "url":        { "type": "keyword" }
      }
    }
  }
}

Step-by-step explanation.

• The template binds all four surfaces at once. number_of_shards / number_of_replicas is the scaling surface; index.default_pipeline wires the ingest surface; mappings.properties is the schema surface; the per-field type (implicitly) selects the analyzer surface (a text field gets the standard analyzer unless overridden).
• level and service are keyword, not text. They are enum-like values you filter and aggregate on exactly, never full-text search. Making them text would break exact aggregation and waste an analyzer.
• message is text. It is the one field ops engineers search with free-text queries, so it must be analyzed into terms.
• latency_ms is long, not a string. Numeric type enables range queries and avg/percentiles aggregations. If the source sends it as a string, the ingest pipeline must convert it — otherwise the mapping either rejects it or (worse) dynamically maps it as text and the dashboards break.
• dynamic: "strict" means an unexpected field throws an error at index time instead of silently auto-mapping — the guardrail against mapping explosion covered in section 2.

Output.

Field	Type	Searchable	Aggregatable	Why
message	text	yes (full-text)	no	analyzed; terms only
level	keyword	yes (exact)	yes	enum; filter + terms agg
latency_ms	long	yes (range)	yes	numeric; avg/percentiles
@timestamp	date	yes (range)	yes	time buckets

Rule of thumb. Before you write a single document, write the index template. The mapping, analyzer selection, ingest pipeline, and shard count are all schema decisions that are cheap to make up front and expensive to change later. "We'll let it auto-map" is the phrase that precedes a reindex.

Worked example — what "ETL sink" means concretely

Detailed explanation. Framing the index as an ETL sink is not a metaphor — it maps one-to-one onto the extract/transform/load stages a data engineer already knows. Walk through the mapping for the events pipeline so the framing is concrete rather than hand-wavy.

• Extract. Read events off Kafka (or Filebeat, or a Spark job).
• Transform. The ingest pipeline (grok, convert, enrich) plus the analyzer (tokenisation) — both run inside the cluster, replacing part of what you'd otherwise do in Spark/Flink.
• Load. The bulk API writes documents into shards; the mapping enforces the schema.

Question. Lay out where each classic ETL stage lives when the sink is Elasticsearch, and what moves out of your Spark/Flink job into the cluster.

Input.

ETL stage	Traditional (Spark → warehouse)	Elasticsearch sink
Extract	read source	read source (unchanged)
Transform	Spark map/filter/join	ingest pipeline + analyzer (in-cluster)
Load	write Parquet / MERGE	bulk API into shards
Schema	table DDL	index mapping

Code.

# The "load" stage — a bulk write is the Elasticsearch equivalent of an INSERT batch
from elasticsearch import Elasticsearch, helpers

es = Elasticsearch("https://search-cluster:9200", api_key="...")

def event_actions(events):
    for e in events:
        yield {
            "_index": "events-2026.08.03",
            "_source": e,        # ingest pipeline (from index.default_pipeline) runs server-side
        }

# helpers.bulk batches actions into NDJSON bulk requests automatically
ok, errors = helpers.bulk(
    es,
    event_actions(stream_of_events),
    chunk_size=2000,           # docs per bulk request
    raise_on_error=False,      # collect per-item errors instead of aborting
)
print(f"indexed {ok} docs, {len(errors)} failures")

Step-by-step explanation.

• The helpers.bulk call is the "load" stage. It batches Python dicts into NDJSON bulk requests, sends them over one HTTP connection each, and returns success/error counts. This is your INSERT ... VALUES batch.
• The transform happens server-side. Because the index template set index.default_pipeline: events-ingest, every document runs through that pipeline inside the cluster before it lands in a shard. Work you'd otherwise do in Spark (parsing, type conversion, enrichment) moves into the ingest node.
• The analyzer is the other half of the transform. When the message field is written, the mapping's analyzer tokenises it into the terms stored in the inverted index. You never see this happen; it is transform-on-write.
• raise_on_error=False turns the bulk write into a partial-success operation: good documents index, bad ones come back in errors for a dead-letter queue. This is the Elasticsearch equivalent of a Spark job writing good rows and quarantining bad ones.
• The mapping is the schema contract. If a document violates it (wrong type, unknown field under strict), that single document fails while the rest of the batch succeeds — again, partial success, not all-or-nothing.

Output.

Concern	Where it lives	Data-engineer takeaway
Parsing / typing	ingest pipeline	moves out of Spark into the cluster
Tokenisation	analyzer (mapping)	transform-on-write, invisible
Batch write	bulk API / helpers	your INSERT batch
Bad records	per-item bulk errors	dead-letter, don't abort

Rule of thumb. Treat the index like a warehouse table with a strict DDL and an inline transform. The moment you internalise "mapping = schema, analyzer + ingest = transform, bulk = load," every Elasticsearch design question becomes a data-engineering question you already know how to answer.

Data engineering interview question on owning the search cluster

A senior interviewer often opens with: "Your team is standing up an Elasticsearch (or OpenSearch) cluster to back both product search and log analytics. The backend team wants to 'just POST JSON and let it auto-map.' As the data engineer, explain what you own, why auto-mapping is dangerous, and the minimal set of decisions you'd lock before the first document is written."

Solution Using an index template that locks mapping, analyzer, ingest, and shard decisions up front

// 1. Component template for shared settings (reusable across indices)
PUT _component_template/base-settings
{
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.refresh_interval": "5s",
      "index.mapping.total_fields.limit": 200
    }
  }
}

// 2. Index template that composes it and locks the schema
PUT _index_template/events-template
{
  "index_patterns": ["events-*"],
  "composed_of": ["base-settings"],
  "priority": 200,
  "template": {
    "settings": { "index.default_pipeline": "events-ingest" },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "level":      { "type": "keyword" },
        "service":    { "type": "keyword" },
        "message":    { "type": "text", "analyzer": "standard" },
        "user_id":    { "type": "keyword" },
        "latency_ms": { "type": "long" }
      }
    }
  }
}

// 3. Minimal ingest pipeline referenced by the template
PUT _ingest/pipeline/events-ingest
{
  "processors": [
    { "convert": { "field": "latency_ms", "type": "long", "ignore_missing": true } },
    { "set": { "field": "ingested_at", "value": "{{{_ingest.timestamp}}}" } }
  ]
}

Step-by-step trace.

Step	Decision	Why it is locked up front
Shards	1 primary / 1 replica per daily index	~50 GB/day → one 50 GB shard is in the sweet spot
Field limit	total_fields.limit = 200	caps mapping explosion from rogue auto-mapping
dynamic	strict	unknown fields error instead of silently mapping
latency_ms	long via ingest convert	numeric aggregations need a numeric type
Pipeline	index.default_pipeline	transform runs for every write, no client coordination

After applying the template, any index matching events-* is created with the locked schema, the ingest pipeline, and the right shard count — the backend team can still "just POST JSON," but now a document with an unexpected field or a stringy latency_ms fails loudly instead of quietly corrupting the mapping. The data engineer has encoded every decision as infrastructure, not tribal knowledge.

Output:

Behaviour	Without template (auto-map)	With template
New field foo: "bar"	silently mapped as text	rejected (strict)
latency_ms: "50" (string)	mapped as text; aggs break	converted to long by ingest
Shard count	1 (default) regardless of size	sized per daily volume
Mapping growth	unbounded	capped at 200 fields
Reproducibility	per-index drift	one versioned template

Why this works — concept by concept:

• Index template — a template binds an index pattern (events-*) to a schema, settings, and a default pipeline, so every new daily index is born correct. It is infrastructure-as-code for the index shape, eliminating per-index drift.
• Component template — reusable settings blocks (composed_of) let many index templates share one definition of shards/replicas/limits. Change the base once; every composing template inherits it.
• dynamic: strict — the single most important guardrail: an unmapped field throws at index time. This converts "silent mapping corruption" into "loud, catchable error" — the difference between a schema you control and one that controls you.
• index.default_pipeline — attaches the transform stage to the index itself, so the ingest pipeline runs regardless of which client wrote the document. The transform is a property of the sink, not of every producer.
• Cost — a few kilobytes of template JSON and five minutes of design. The eliminated cost is a multi-hour reindex of a corrupted index, plus the on-call incident when a stringy numeric silently disabled every dashboard aggregation. O(1) up-front design versus O(N-documents) remediation.

ETL
Topic — etl
ETL problems on building typed sinks and schemas

Practice →

Streaming Topic — streaming Streaming ingestion and write-path problems

Practice →

---

2. Mappings and the inverted index

mappings are the typed schema of the index, and the inverted index is why full-text search is fast — text vs keyword is the decision that binds every query

The mental model in one line: a mapping is the strongly-typed schema that tells Elasticsearch how to store and index every field, the inverted index is the term-to-document data structure that makes full-text search fast (it maps each term to the list of documents containing it, so a query looks up terms instead of scanning documents), and the single most consequential mapping decision is text vs keyword — text is analyzed into terms for full-text search and cannot be aggregated or sorted efficiently, while keyword is stored verbatim for exact-match filtering, sorting, and aggregation. Almost every "search returns nothing" and "aggregation is wrong" bug traces back to a text/keyword mistake in the mapping.

The inverted index — the data structure under everything.

• Forward index (what you'd naively build). Document → list of words. Answering "which documents contain 'error'?" means scanning every document. O(N) per query.
• Inverted index (what Lucene builds). Term → posting list (the sorted list of document IDs containing that term, plus positions and frequencies). Answering "which documents contain 'error'?" is a single dictionary lookup returning the posting list. This is why full-text search over billions of documents is fast.
• The term is the unit. The inverted index stores terms, not raw field values. A text field's string is run through an analyzer to produce terms; the terms are what get indexed. "The Quick Fox" might become the terms [quick, fox]. A query for "quick" matches; a query for "The Quick Fox" (as an exact phrase) matches only if positions line up.
• doc_values — the columnar sibling. Sorting, aggregating, and scripting need the values per document, which the inverted index is bad at. So Elasticsearch also builds doc_values: an on-disk columnar store (value → document is the wrong way round for aggs; doc_values is document → value, column-oriented). keyword, numeric, and date fields have doc_values on by default; analyzed text fields do not (they have fielddata, which is memory-hungry and off by default).

Field types every data engineer must know.

• text. Analyzed into terms; full-text searchable; not aggregatable/sortable by default. Use for prose: message, description, title.
• keyword. Stored verbatim (one term = the whole value); exact-match, sort, and aggregate. Use for enums, IDs, tags, hostnames, status codes: level, service, user_id.
• The text + keyword multi-field. The classic pattern: index a field as text for search and as a keyword sub-field for aggregation. title (text) + title.keyword (keyword). This is the default dynamic mapping for strings and the reason .keyword shows up everywhere.
• Numeric (long, integer, double, scaled_float), date, boolean. Typed, doc_values-backed, range-queryable, aggregatable. Pick the smallest type that fits; scaled_float for money.
• object vs nested. object flattens nested JSON (arrays of objects lose per-object correlation — the "cross-object matching" trap). nested indexes each sub-object as a hidden separate document so per-object queries work, at a storage and query cost. Interviewers love the "array of objects" correlation trap.
• ip, geo_point, flattened, dense_vector. Specialised types; flattened is a pragmatic escape hatch for arbitrary key/value blobs that would otherwise explode the mapping.

Dynamic vs explicit mapping — and the explosion problem.

• Dynamic mapping. By default, an unmapped field is auto-typed on first sight (a string becomes text + .keyword, a number becomes long, etc.). Convenient for prototyping, dangerous in production.
• Mapping explosion. If documents contain unbounded distinct field names — e.g. a labels object keyed by user-supplied strings, or metrics keyed by dynamic dimension names — the mapping grows one field per distinct key. Thousands of fields bloat the cluster state, slow down every operation, and can take the cluster down. This is the classic high-cardinality-keys incident.
• The three defenses. (1) dynamic: "strict" — reject unknown fields. (2) dynamic: "false" — store but don't index unknown fields (they're in _source but not searchable). (3) flattened type or dynamic templates — collapse an open-ended object into a single field. Plus index.mapping.total_fields.limit as a hard cap.
• Dynamic templates. Rules that map fields by name pattern or detected type — e.g. "any field ending in _id → keyword," "any string → keyword only (skip the text sub-field)." The middle ground between fully dynamic and fully explicit.

Index templates — apply mappings before the index exists.

• Why templates. Time-series data uses one index per day/rollover (events-2026.08.03). You can't hand-create a mapping for tomorrow's index; a template applies the mapping automatically to any index matching a pattern.
• Composable templates. Modern Elasticsearch/OpenSearch use composable index templates plus component templates (reusable fragments). Priority resolves conflicts when multiple templates match.
• What goes in a template. Settings (shards, replicas, refresh interval, default pipeline), mappings (the schema), and aliases. This is the one artifact that makes time-series indexing reproducible.

Worked example — the text-vs-keyword aggregation bug

Detailed explanation. The single most common Elasticsearch mapping bug: a field that should be keyword was auto-mapped as text, and now aggregating on it either fails outright (fielddata disabled) or returns per-term buckets instead of per-value buckets. Walk through the bug and the fix on a service field.

• The symptom. terms aggregation on service returns "payment", "gateway" as separate buckets when the value was "payment-gateway" — because the analyzer split on the hyphen.
• The root cause. service was dynamically mapped as text (analyzed), so the aggregation runs over the analyzed terms, not the original value.
• The fix. Map service as keyword (or aggregate on the service.keyword sub-field).

Question. Show the broken mapping, demonstrate the wrong aggregation, and fix it with an explicit keyword mapping.

Input.

Document service value	Analyzed terms (text)	Keyword term
"payment-gateway"	[payment, gateway]	[payment-gateway]
"payment-gateway"	[payment, gateway]	[payment-gateway]
"auth-service"	[auth, service]	[auth-service]

Code.

// BROKEN — service auto-mapped as text, aggregation splits on the hyphen
GET events/_search
{
  "size": 0,
  "aggs": {
    "by_service": {
      "terms": { "field": "service.keyword" }   // note: must use .keyword sub-field
    }
  }
}

// If you aggregate on the analyzed field directly you get an error:
// "Fielddata is disabled on text fields by default. Set fielddata=true on [service]..."

// FIXED — declare service as keyword explicitly in the mapping
PUT events-fixed
{
  "mappings": {
    "properties": {
      "service": { "type": "keyword" },
      "message": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
      }
    }
  }
}

// Now aggregate directly on the keyword field
GET events-fixed/_search
{
  "size": 0,
  "aggs": { "by_service": { "terms": { "field": "service" } } }
}

Step-by-step explanation.

• Auto-mapping made service a text field with a .keyword sub-field. The text version analyzed "payment-gateway" into [payment, gateway]; the .keyword sub-field kept it verbatim. Aggregating on the bare text field errors because analyzed fields have no doc_values.
• The wrong result comes from aggregating on the analyzed terms. If you enabled fielddata=true (do not), the terms agg would bucket by payment, gateway, auth, service — meaningless. The .keyword sub-field buckets correctly by the whole value.
• The fix is to declare the field's intent. service is an enum you filter and aggregate on, so it is keyword — full stop, no text version needed. This halves its storage and removes the footgun.
• message keeps the multi-field pattern because it genuinely needs both: full-text search (text) and occasional exact aggregation (message.keyword with ignore_above: 256 so giant strings don't blow up the keyword index).
• The lesson is intent-driven mapping. Every string field is either "prose I search" (text), "value I filter/aggregate" (keyword), or "both" (multi-field). Auto-mapping picks "both" for everything, which is wasteful and error-prone.

Output.

Approach	Buckets returned	Correct?
agg on text field	error (fielddata disabled)	no
agg on text with fielddata=true	payment, gateway, auth, service	no (split terms)
agg on .keyword sub-field	payment-gateway, auth-service	yes
service mapped as keyword	payment-gateway, auth-service	yes (and cheaper)

Rule of thumb. If you will ever filter, sort, or aggregate a string field exactly, it is a keyword. If you will run free-text search over it, it is text. Only use the text + keyword multi-field when you genuinely need both — and never let auto-mapping decide for you.

Worked example — preventing mapping explosion with strict + flattened

Detailed explanation. A labels object carries arbitrary user-supplied keys (labels.env, labels.team, labels.customer_id_47, ...). Under dynamic mapping, every distinct key becomes a new mapping field; at thousands of keys the cluster state bloats and the cluster degrades. Walk through the two-part defense: flattened for the open-ended object plus strict for the top level.

• The problem. Unbounded distinct keys under labels → unbounded mapping fields.
• The fix. Map labels as flattened (one field, keys stored as sub-values, no per-key mapping) and set the top-level dynamic: strict.
• The trade-off. flattened fields support exact-term queries but not full-text analysis or per-key numeric ranges — acceptable for label-style key/value blobs.

Question. Design a mapping that safely absorbs an open-ended labels object without mapping explosion, and keep the rest of the schema strict.

Input.

Field	Cardinality of keys	Mapping choice
service, level	fixed enum	keyword
message	prose	text
labels.*	unbounded, user-supplied	flattened (one field)
any unexpected top-level field	should not exist	strict → reject

Code.

PUT events-safe
{
  "settings": {
    "index.mapping.total_fields.limit": 100
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": { "type": "date" },
      "service":    { "type": "keyword" },
      "level":      { "type": "keyword" },
      "message":    { "type": "text" },
      "labels":     { "type": "flattened" }
    }
  }
}

// Querying a key inside a flattened field uses dot-notation on the term
GET events-safe/_search
{
  "query": {
    "term": { "labels.env": "prod" }
  }
}

// A document with an unexpected TOP-LEVEL field is rejected:
// POST events-safe/_doc { "service": "x", "surprise": 1 }
// -> mapper_parsing_exception: mapping set to strict, dynamic introduction of [surprise] not allowed

Step-by-step explanation.

• labels as flattened collapses the whole sub-tree into one mapping field. No matter how many distinct keys appear (labels.env, labels.customer_id_9999), the mapping has exactly one entry: labels. The keys become searchable values, not schema fields.
• dynamic: strict at the top level rejects unexpected fields. A producer bug that adds a new top-level field fails loudly instead of silently growing the mapping. Note strict applies to the object it's declared on; flattened handles the intentionally-open sub-tree.
• total_fields.limit: 100 is the backstop. Even if some other object were dynamic, the index refuses to exceed 100 fields — the cluster protects itself.
• The query trade-off is explicit. Inside a flattened field you get exact term matching on keys (labels.env = prod) but not full-text analysis, not per-key numeric ranges, and not independent per-key aggregation types. For label blobs that is exactly right.
• The pattern generalises. Any "bag of arbitrary attributes" — HTTP headers, tags, feature flags, dynamic dimensions — belongs in a flattened field, not as first-class mapped fields.

Output.

Scenario	Dynamic mapping	flattened + strict
10,000 distinct label keys	10,000 mapping fields	1 mapping field
Cluster state size	bloated	bounded
Unexpected top field	silently mapped	rejected
Exact key query (env=prod)	works	works
Full-text on a label value	works	not analyzed (acceptable)

Rule of thumb. The moment a field's keys are data (user-supplied, unbounded, dynamic), reach for flattened, not first-class fields — and pair it with dynamic: strict and a total_fields.limit. Mapping explosion is a self-inflicted outage that these three settings prevent.

Worked example — designing the mapping for a logs index

Detailed explanation. Bring it together: design the complete explicit mapping for a production application-logs index, choosing the right type for each field and the right multi-field pattern where needed. This is the artifact an interviewer asks you to whiteboard.

• Fields. @timestamp (date), level (keyword), service (keyword), host (keyword), message (text + keyword), status_code (short), latency_ms (long), url (keyword + text for search), client_ip (ip), trace_id (keyword).
• Goals. Full-text on message and url; exact filter/agg on level, service, status_code; numeric aggregation on latency_ms; IP-range queries on client_ip.

Question. Write the complete mapping with justified type choices and multi-fields.

Input.

Field	Primary use	Type
@timestamp	time range + bucketing	date
level, service, host, trace_id	exact filter / agg	keyword
status_code	filter / agg (small int)	short
latency_ms	numeric agg	long
message	full-text + exact	text + keyword
url	full-text + exact	text + keyword
client_ip	IP range	ip

Code.

PUT app-logs
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1 },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp":  { "type": "date" },
      "level":       { "type": "keyword" },
      "service":     { "type": "keyword" },
      "host":        { "type": "keyword" },
      "trace_id":    { "type": "keyword" },
      "status_code": { "type": "short" },
      "latency_ms":  { "type": "long" },
      "client_ip":   { "type": "ip" },
      "message": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword", "ignore_above": 1024 } }
      },
      "url": {
        "type": "text",
        "analyzer": "standard",
        "fields": { "keyword": { "type": "keyword", "ignore_above": 2048 } }
      }
    }
  }
}

Step-by-step explanation.

• Enums and IDs are keyword. level, service, host, trace_id are never full-text searched — you filter (level = ERROR) and aggregate (top services) on them exactly. keyword gives doc_values for fast aggregation and no analyzer overhead.
• Small integers use the smallest fitting numeric. status_code fits in a short (values 100–599). Right-sizing numerics saves disk and heap across billions of documents.
• client_ip is the ip type, which supports CIDR-range queries (client_ip: "10.0.0.0/8") and IP-aware aggregations — impossible on a plain keyword.
• message and url are multi-fields. The text version powers full-text search; the .keyword version (with ignore_above to skip pathologically long values) powers exact aggregation and sorting. ignore_above prevents a giant URL from bloating the keyword index.
• dynamic: strict guards the whole thing. Any field not in this list is rejected — so an accidental password field or a stray metric can't silently enter the index.

Output.

Field	Type	Searchable	Aggregatable	Special
message	text (+kw)	full-text	via .keyword
status_code	short	range	yes	tiny footprint
latency_ms	long	range	avg/percentiles
client_ip	ip	CIDR range	yes	IP-aware
level/service	keyword	exact	yes	enum

Rule of thumb. Whiteboard the mapping field-by-field: for each, ask "search it, filter/agg it, or both?" and "what's the smallest correct type?" A mapping designed this way never needs the emergency reindex that an auto-mapped index eventually forces.

SQL-to-Elasticsearch interview question on mappings

A senior interviewer might ask: "You're migrating a products table (columns: id, name, brand, category, price, tags array, description, created_at) from Postgres into Elasticsearch to power a product search. Design the mapping so that name/description are full-text searchable, brand/category/tags are faceted (aggregated) filters, price supports range queries and sorting, and the mapping cannot explode. Explain each type choice."

Solution Using an explicit mapping with multi-fields, keyword facets, and a scaled_float price

PUT products
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.mapping.total_fields.limit": 50,
    "analysis": {
      "analyzer": {
        "product_text": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stop":    { "type": "stop", "stopwords": "_english_" },
        "english_stemmer": { "type": "stemmer", "language": "english" }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "id":       { "type": "keyword" },
      "name": {
        "type": "text",
        "analyzer": "product_text",
        "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
      },
      "description": { "type": "text", "analyzer": "product_text" },
      "brand":    { "type": "keyword" },
      "category": { "type": "keyword" },
      "tags":     { "type": "keyword" },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "created_at": { "type": "date" }
    }
  }
}

// The search this mapping enables: full-text on name/description,
// keyword facets on brand/category/tags, price range + sort
GET products/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "name": "wireless headphones" } } ],
      "filter": [
        { "term":  { "brand": "acme" } },
        { "terms": { "tags": ["bluetooth", "noise-cancelling"] } },
        { "range": { "price": { "gte": 50, "lte": 300 } } }
      ]
    }
  },
  "sort": [ { "price": "asc" } ],
  "aggs": {
    "by_category": { "terms": { "field": "category" } },
    "avg_price":   { "avg":   { "field": "price" } }
  }
}

Step-by-step trace.

Column (Postgres)	Elasticsearch type	Reasoning
name (varchar)	text (+keyword)	full-text search + exact sort
description (text)	text	full-text only
brand, category	keyword	faceted exact filters + aggs
tags (array)	keyword	array of exact values; faceting
price (numeric)	scaled_float (×100)	integer cents under the hood; range + sort
created_at	date	time range + sort
id (pk)	keyword	exact lookup, no analysis

The products search now behaves like a faceted storefront: match on name/description scores relevance, term/terms/range filters narrow exactly, terms aggregations produce the facet counts (categories with counts) shoppers expect, and avg gives the average price of the result set. The scaled_float stores prices as integer cents internally, avoiding floating-point rounding on money while still exposing a decimal API.

Output:

Requirement	Mechanism	Result
Full-text name/description	text + custom analyzer	stemmed, lowercased matching
Facets on brand/category/tags	keyword + terms agg	exact bucket counts
Price range + sort	scaled_float	range filter + numeric sort
No mapping explosion	dynamic: strict + field limit	bounded schema
Money precision	scaling_factor: 100	integer cents, no float drift

Why this works — concept by concept:

• text vs keyword split — name/description are text (analyzed for search); brand/category/tags are keyword (verbatim for exact facets). The same field can be both via a multi-field, but only where genuinely needed.
• scaled_float for money — stores 12.99 as the integer 1299 with scaling_factor: 100, giving exact range/sort behaviour without binary floating-point rounding. The correct money type in Elasticsearch.
• keyword array for tags — an array of keywords indexes each element as a separate term, so terms filters and facet aggregations work per-tag naturally; no nested needed for a flat array of scalars.
• custom product_text analyzer — lowercasing + English stopword removal + stemming so "Wireless Headphones" matches "wireless headphone"; the analyzer is a mapping decision baked into the field, covered next in section 3.
• Cost — one explicit mapping (a few KB) plus a custom analyzer. The eliminated cost is the reindex you'd otherwise run when auto-mapping makes price a float, tags a text field, and the facets return garbage. O(1) design; O(N-docs) is what you avoid.

Parsing
Topic — parsing
Parsing and structured-data modeling problems

Practice →

ETL Topic — etl ETL problems on schema design and type mapping

Practice →

---

3. Analyzers — tokenizers, filters, and the analyze API

An analyzer is a char-filter → tokenizer → token-filter pipeline that decides what terms land in the inverted index — analysis is a data-modeling decision, not a query tweak

The mental model in one line: an analyzer is a three-stage text-processing pipeline — zero or more character filters (rewrite the raw string), exactly one tokenizer (split the string into tokens), and zero or more token filters (lowercase, remove stopwords, stem, add synonyms, generate n-grams) — that runs at index time to produce the terms stored in the inverted index, and because a query can only ever match a term that analysis actually produced, choosing the analyzer is a data-modeling decision that binds every future search. The most-repeated senior insight: if index-time analysis didn't produce a term, no query will ever find it — you cannot "search harder" your way past a modeling mistake.

The three stages of an analyzer.

• Character filters. Operate on the raw string before tokenisation. html_strip removes tags; mapping replaces characters (e.g. & → and); pattern_replace does regex rewrites. Zero or more, applied in order.
• Tokenizer. Exactly one. Splits the character-filtered string into tokens. standard (Unicode word boundaries), whitespace (split on spaces only), keyword (emit the whole input as one token — the "no-op" tokenizer), pattern (regex split), ngram / edge_ngram (substring tokens for partial matching / autocomplete).
• Token filters. Zero or more, applied in order to the token stream. lowercase, stop (drop stopwords like "the"), stemmer (reduce "running"→"run"), synonym, ngram/edge_ngram, asciifolding (é→e). Order matters: lowercase before stemming, synonyms usually after lowercasing.

Built-in analyzers you should recognise.

• standard (the default). standard tokenizer + lowercase filter. Good general-purpose full-text default.
• keyword. No tokenisation — the whole field becomes one term. (This is what the keyword field type uses; do not confuse the analyzer with the type.)
• whitespace. Splits on whitespace only; keeps case and punctuation. Useful for code/identifiers.
• simple, stop, english (and other language analyzers). english adds English stopwords + stemming; language analyzers are the quick path to decent relevance for prose.

The _analyze API — the debugger for search.

• What it does. Runs any analyzer (built-in, custom, or an ad-hoc filter chain) over a sample string and returns the exact tokens produced. This is how you see what will be indexed.
• Why it matters. "Why does my search return nothing?" is answered by running _analyze on both the indexed text and the query text and comparing tokens. If they don't overlap, they can't match.
• Index-time vs search-time. By default the same analyzer runs at index time and query time. You can set a different search_analyzer — critical for n-gram autocomplete (n-gram at index time to build prefixes, standard at search time so the query isn't itself n-grammed).

Why analysis is a data-modeling decision.

• It is irreversible without reindex. The terms are computed at write time and stored. Changing the analyzer only affects future documents unless you reindex everything.
• It defines the match space. Stemming makes run/running/ran interchangeable — great for prose, wrong for exact SKUs. Lowercasing makes search case-insensitive — wrong if case is semantic. Stopword removal makes "to be or not to be" nearly empty — catastrophic for phrase search on quotes.
• It trades recall vs precision. Aggressive analysis (stemming, synonyms, n-grams) increases recall (finds more) at the cost of precision (finds wrong things) and index size. This is a modeling choice, and different fields on the same document often want different analyzers.

Worked example — using the _analyze API to debug a failing search

Detailed explanation. A search for "C++" on a text field returns nothing even though documents clearly contain "C++". Use the _analyze API to see why: the standard tokenizer strips the + characters. Then fix it with a whitespace-based custom analyzer.

• The symptom. match: { skills: "C++" } returns zero hits.
• The diagnosis. _analyze shows "C++" tokenised as [c] — the +s are dropped and the query and documents both reduce to c, which... actually matches. The real failure is "C#" vs "C++" both reducing to [c], so they're indistinguishable. Run _analyze to see it.
• The fix. A whitespace tokenizer + lowercase keeps c++ and c# as distinct tokens.

Question. Use _analyze to reveal the tokenisation, then design an analyzer that preserves symbolic tokens like c++ and c#.

Input.

Input string	standard analyzer tokens	Desired tokens
"C++"	[c]	[c++]
"C#"	[c]	[c#]
"Java Developer"	[java, developer]	[java, developer]

Code.

// 1. Diagnose with the _analyze API
POST _analyze
{
  "analyzer": "standard",
  "text": "C++ and C# Developer"
}
// -> tokens: [ "c", "c", "developer" ]   // "and" is a token too; symbols gone

// 2. Define a custom analyzer that keeps symbols
PUT skills-index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "symbol_safe": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "skills": { "type": "text", "analyzer": "symbol_safe" }
    }
  }
}

// 3. Verify the custom analyzer
POST skills-index/_analyze
{
  "analyzer": "symbol_safe",
  "text": "C++ and C# Developer"
}
// -> tokens: [ "c++", "and", "c#", "developer" ]

Step-by-step explanation.

• _analyze reveals the invisible transform. Running the standard analyzer on "C++ and C# Developer" returns [c, c, developer] — the standard tokenizer discards + and # as non-word characters, so c++ and c# collapse to the same c term. No query can distinguish them.
• The fix changes the tokenizer, not the query. Swapping to the whitespace tokenizer splits only on spaces, preserving C++ and C# as whole tokens; the lowercase filter then normalises them to c++ and c# for case-insensitive matching.
• Verification closes the loop. Re-running _analyze on the custom analyzer confirms the tokens are now [c++, and, c#, developer]. The query "c++" will produce the term c++ and match documents containing that term.
• The stopword nuance. "and" survived because the custom analyzer has no stop filter. For a skills field that's fine; for prose you'd add one. That's a per-field modeling choice.
• The general method. Any "search returns nothing / wrong things" bug is debugged by _analyze-ing the indexed text and the query text and comparing token sets. If the sets don't intersect, no match is possible — the fix is always in analysis, never in the query.

Output.

Query	With standard	With symbol_safe
c++	matches everything (all → c)	matches only C++ docs
c#	matches everything (all → c)	matches only C# docs
java	matches	matches

Rule of thumb. When search behaves unexpectedly, reach for _analyze first — it turns the invisible index-time transform into a visible token list. Ninety percent of "search is broken" tickets are an analyzer mismatch between index time and query time, and _analyze shows it in one call.

Worked example — building an edge_ngram autocomplete analyzer

Detailed explanation. Autocomplete ("search-as-you-type") needs to match a document from a prefix of a word. The standard approach: an edge_ngram token filter at index time generates all prefixes of each token, and a standard analyzer at search time keeps the query as-is (so the query "elas" matches the indexed prefix "elas" of "elasticsearch"). Getting the index-vs-search analyzer split right is the whole trick.

• Index time. edge_ngram (min 2, max 20) turns "elasticsearch" into [el, ela, elas, elast, ...].
• Search time. standard keeps "elas" as [elas] — do not edge-ngram the query, or "elas" would become [el, ela, elas] and match too broadly.
• Result. Typing "elas" matches "elasticsearch" after 4 characters.

Question. Build the index-time edge_ngram analyzer and the distinct search_analyzer, and show the tokens each produces.

Input.

Stage	Analyzer	"elasticsearch" / "elas" tokens
Index time	edge_ngram (2–20)	[el, ela, elas, elast, ...]
Search time	standard	[elas]

Code.

PUT autocomplete-index
{
  "settings": {
    "analysis": {
      "filter": {
        "edge_2_20": { "type": "edge_ngram", "min_gram": 2, "max_gram": 20 }
      },
      "analyzer": {
        "autocomplete_index": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "edge_2_20"]
        },
        "autocomplete_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete_index",
        "search_analyzer": "autocomplete_search"
      }
    }
  }
}

// Verify index-time tokens (prefixes generated)
POST autocomplete-index/_analyze
{ "analyzer": "autocomplete_index", "text": "Elasticsearch" }
// -> [ "el", "ela", "elas", "elast", "elasti", ..., "elasticsearch" ]

// Verify search-time tokens (query NOT expanded)
POST autocomplete-index/_analyze
{ "analyzer": "autocomplete_search", "text": "elas" }
// -> [ "elas" ]

// The query
GET autocomplete-index/_search
{ "query": { "match": { "title": "elas" } } }   // matches "Elasticsearch"

Step-by-step explanation.

• edge_ngram at index time pre-computes every prefix. "elasticsearch" becomes [el, ela, elas, ...], all stored as terms. Now a prefix query is just a normal term lookup — no expensive wildcard scan.
• The search_analyzer is different on purpose. At search time, "elas" must stay [elas]. If you edge-ngrammed the query too, "elas" → [el, ela, elas] would match anything starting with "el" — far too broad. Splitting index and search analyzers is the core technique.
• min_gram: 2 skips single-character prefixes. Indexing [e] would match on the first keystroke and return the entire corpus; starting at 2 keeps autocomplete useful.
• max_gram: 20 bounds index growth. Each token generates up to (max−min+1) prefix terms, so edge_ngram multiplies the term count. Capping the gram length bounds the storage cost.
• The _analyze calls prove correctness. Index-time analysis shows the prefixes exist; search-time analysis shows the query is a single term. Because elas is in the index-time set, the match succeeds — visibly, not by faith.

Output.

Typed query	Index terms it hits	Matches "Elasticsearch"?
e	(none — min_gram 2)	no (too short)
el	el	yes
elas	elas	yes
xyz	(none)	no

Rule of thumb. Autocomplete = edge_ngram at index time + a plain analyzer at search time. If your autocomplete "matches everything after two letters," you edge-ngrammed the query too — set an explicit search_analyzer. Always bound max_gram so the index doesn't balloon.

Worked example — synonyms and stemming for recall

Detailed explanation. A product search should treat "laptop" and "notebook" as the same intent, and "running"/"run" as the same root. Synonyms (a token filter) and stemming (a token filter) both increase recall. Walk through a custom analyzer that lowercases, applies synonyms, removes stopwords, and stems — in the correct order.

• Order matters. lowercase → synonym → stop → stemmer. Synonyms are defined in lowercase, so lowercase first. Stem last so stemming applies to synonym-expanded tokens too.
• Synonym file vs inline. Inline for a handful; a synonyms file (or synonyms set) for large lists.
• The trade-off. More recall, less precision, larger index. A modeling choice per field.

Question. Build a product_relevance analyzer with lowercase, synonyms, English stopwords, and English stemming, and show how "Notebooks" and "laptop" end up as the same term.

Input.

Raw token	After lowercase	After synonym	After stop	After stemmer
"Notebooks"	notebooks	notebooks, laptop	(unchanged)	notebook, laptop
"the"	the	the	(dropped)	—
"Running"	running	running	(unchanged)	run

Code.

PUT catalog
{
  "settings": {
    "analysis": {
      "filter": {
        "retail_synonyms": {
          "type": "synonym",
          "synonyms": [
            "laptop, notebook, portable computer",
            "tv, television"
          ]
        },
        "english_stop":    { "type": "stop",    "stopwords": "_english_" },
        "english_stemmer": { "type": "stemmer", "language": "english" }
      },
      "analyzer": {
        "product_relevance": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "retail_synonyms", "english_stop", "english_stemmer"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "product_relevance" }
    }
  }
}

POST catalog/_analyze
{ "analyzer": "product_relevance", "text": "Running Notebooks and the TV" }
// -> [ "run", "notebook", "laptop", "portabl", "comput", "tv", "televis" ]
//    ("and", "the" dropped by stop; synonyms expanded; everything stemmed)

Step-by-step explanation.

• Lowercase runs first so the synonym rules (written in lowercase) match. "Notebooks" → notebooks before the synonym filter sees it.
• The synonym filter expands one token into several. notebooks emits notebooks, laptop, portable computer at the same position, so a query for any of them matches this document. This is the recall boost.
• The stop filter drops noise words (and, the) so they neither bloat the index nor create spurious matches. It runs after synonyms so a synonym expansion isn't accidentally treated as a stopword-only phrase.
• The stemmer runs last and reduces every surviving token to its root (notebooks→notebook, running→run, television→televis). Because it runs after synonym expansion, both laptop and notebook get stemmed consistently, so query and document roots line up.
• The order is the lesson. Reorder these filters and you get subtle bugs: stem-before-synonym means your synonym list (written in un-stemmed form) stops matching; stop-before-lowercase means capitalised stopwords survive. Filter order is a modeling decision.

Output.

Query	Matches "Running Notebooks"?	Why
laptop	yes	synonym of notebook
run	yes	stemmed root of running
notebook	yes	stemmed root
the	no	stopword, never indexed

Rule of thumb. Chain token filters in the order lowercase → synonym → stop → stemmer, and always verify with _analyze. Synonyms and stemming buy recall; test them against real queries so you don't accidentally make "apple" (fruit) match "apple" (company) or over-stem distinct terms into collisions.

Parsing interview question on analyzers

A senior interviewer might ask: "Users search your product catalog and complain that searching 'e-mail' doesn't find documents containing 'email', 'MacBook' doesn't match 'macbook', and searching 'don't' returns nothing. Walk me through how you'd diagnose each with the _analyze API and design one analyzer that fixes all three — while explaining why analysis is a modeling decision you can't fix at query time."

Solution Using a custom analyzer with char-mapping, lowercase, and controlled tokenisation

PUT catalog-v2
{
  "settings": {
    "analysis": {
      "char_filter": {
        "punctuation_map": {
          "type": "mapping",
          "mappings": [
            "e-mail => email",
            "- => ",
            "' => "
          ]
        }
      },
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" }
      },
      "analyzer": {
        "catalog_search": {
          "type": "custom",
          "char_filter": ["punctuation_map"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "catalog_search" }
    }
  }
}

// Diagnose each complaint BEFORE the fix
POST _analyze { "analyzer": "standard", "text": "e-mail" }   // -> [ "e", "mail" ]
POST _analyze { "analyzer": "standard", "text": "MacBook" }  // -> [ "macbook" ]  (already ok, case handled by lowercase)
POST _analyze { "analyzer": "standard", "text": "don't" }    // -> [ "don't" ]     (apostrophe kept -> mismatch with "dont")

// Verify the fix
POST catalog-v2/_analyze { "analyzer": "catalog_search", "text": "e-mail MacBook don't" }
// -> [ "email", "macbook", "dont" ]

Step-by-step trace.

Complaint	_analyze shows	Root cause	Fix
e-mail ≠ email	[e, mail] vs [email]	standard splits on hyphen	char-filter maps e-mail → email and strips -
MacBook ≠ macbook	[macbook] vs [macbook]	actually case only	lowercase filter
don't finds nothing	[don't] vs [dont]	apostrophe kept, query typed without it	char-filter strips '

The single catalog_search analyzer solves all three: the mapping char filter rewrites e-mail → email and deletes stray hyphens and apostrophes before tokenisation, the standard tokenizer then produces clean word tokens, and lowercase + asciifolding normalise case and accents. Because the same analyzer runs at index and search time, the query text and the indexed text pass through identical transforms and therefore produce identical terms — which is the only way they can match.

Output:

Indexed value	Terms (catalog_search)	Query	Matches?
e-mail alerts	[email, alert]*	email	yes
MacBook Pro	[macbook, pro]	macbook	yes
don't panic	[dont, panic]	dont	yes

*(with stopword/stemming added the terms differ slightly; shown without for clarity)

Why this works — concept by concept:

• Character filters run before tokenisation — the mapping char filter rewrites the raw string (e-mail → email, strip - and ') so the tokenizer never sees the problematic punctuation. Fixing it pre-tokenisation is cleaner than post-hoc token surgery.
• Same analyzer at index and query time — because Elasticsearch applies the field's analyzer to the query string too, identical transforms guarantee the query term set and the document term set are computed the same way. Mismatched analyzers are the number-one "no results" cause.
• lowercase + asciifolding — case-insensitivity and accent-folding (café → cafe) are token filters that make near-identical human spellings collapse to one term, boosting recall predictably.
• _analyze as the diagnostic — every complaint was reproduced as a token-set mismatch and confirmed fixed by re-running _analyze. You never guess; you look at the terms.
• Cost — one analyzer definition and (if the analyzer changes for an existing field) a reindex, because analysis is baked into stored terms. That reindex cost is exactly why analysis is a modeling decision made up front — O(1) to design, O(N-docs) to change after the fact.

Parsing
Topic — parsing
Parsing, tokenisation, and text-normalisation problems

Practice →

Streaming Topic — streaming Streaming problems on real-time text processing

Practice →

---

4. Ingest pipelines and the bulk API

ingest pipelines transform documents inside the cluster before they hit a shard, and the bulk api is how you load millions of them efficiently — this is the T and the L of ETL living in Elasticsearch

The mental model in one line: an ingest pipeline is an ordered chain of processors (grok, dissect, set, rename, convert, date, geoip, script, enrich, drop) that runs on an ingest node to reshape each document before it is indexed — replacing a lot of Logstash/Spark preprocessing — while the bulk api batches many index/update/delete operations into a single newline-delimited-JSON (NDJSON) request so you load at tens of thousands of docs/sec instead of one HTTP round-trip per document, and the two together are the transform-and-load half of ETL executed inside the search cluster. A data engineer who knows only single-document POST /_doc writes will never hit acceptable throughput; bulk + ingest is the production path.

Ingest processors — the transform toolbox.

• Parsing. grok (named regex patterns for unstructured logs), dissect (faster fixed-delimiter parsing), kv (key=value splitting), csv, json. Turn a raw log line into structured fields.
• Reshaping. set (add/overwrite a field), rename, remove, convert (string→long/boolean/etc.), gsub (regex replace), lowercase/uppercase, trim, split, join.
• Enrichment. date (parse a timestamp string into a real date, set @timestamp), geoip (IP → lat/lon/country), user_agent (parse UA strings), enrich (join against a lookup index), set with a scripted value.
• Control flow. drop (discard a document conditionally), pipeline (call another pipeline), on_failure (per-processor or pipeline-level error handling), and every processor's if condition (a Painless script gate).

The enrich processor — a join inside the cluster.

• What it does. Looks up each document's key in a pre-built enrich index and merges matching fields in — e.g. join product_id against a product-catalog enrich index to add product_name and category. A dimension-table join at ingest time.
• The setup. Create a source index → define an enrich policy (match field + fields to add) → execute the policy (materialises the enrich index) → reference it in an enrich processor. Re-execute the policy when the source data changes.
• The trade-off. Enrich data is a point-in-time snapshot (from the last policy execution), so it suits slowly-changing dimensions, not rapidly-mutating lookups.

The reindex API — backfills and re-processing.

• What it does. Copies documents from a source index to a destination index, optionally running an ingest pipeline and/or a query filter. The tool for "I changed the mapping/analyzer and need to re-process existing data."
• Why data engineers live in it. Every irreversible mapping/analyzer decision is un-done by a reindex. Reindex-with-pipeline lets you retrofit a transform onto historical data.
• Doing it safely. slices for parallelism, wait_for_completion=false to run as a task, throttling with requests_per_second, and the alias-swap pattern (reindex into a new index, then atomically move the read alias) for zero-downtime schema migrations.

The bulk API — high-throughput loading.

• The format. NDJSON: each operation is two lines — an action/metadata line ({"index": {"_index": "...", "_id": "..."}}) and, for index/create/update, a source line. Newline-delimited, not a JSON array.
• Partial success. A bulk request returns per-item results; some items can fail (mapping conflict, version conflict) while others succeed. You must inspect the errors flag and per-item statuses — the HTTP response is 200 even with failed items.
• Sizing. Batch by size (aim ~5–15 MB per bulk request) and/or count (1,000–5,000 docs), tune with concurrency. Too large → memory pressure and timeouts; too small → HTTP overhead dominates.
• The Python client helpers. helpers.bulk (simple, batches for you), helpers.streaming_bulk (yields per-item results for backpressure), helpers.parallel_bulk (multiple threads). These wrap the NDJSON mechanics and error collection.

Worked example — a grok + convert + date ingest pipeline for NGINX logs

Detailed explanation. Raw NGINX access-log lines are unstructured text. Build an ingest pipeline that grok-parses each line into fields, converts the numeric fields, parses the timestamp into @timestamp, adds a geoip enrichment, and drops health-check noise — so the app team can index raw strings and get structured, typed, enriched documents.

• Input. 192.168.1.10 - - [03/Aug/2026:10:15:32 +0000] "GET /api/orders HTTP/1.1" 200 1534
• Goal fields. client_ip (ip), method, path, status (int), bytes (long), @timestamp (date), geo (from geoip).
• Drop rule. Discard requests to /healthz.

Question. Write the ingest pipeline that parses, types, timestamps, enriches, and filters the NGINX log line.

Input.

Raw token	Grok field	Final type
192.168.1.10	client_ip	ip
GET / /api/orders	method / path	keyword
200	status	integer
1534	bytes	long
03/Aug/2026:10:15:32 +0000	timestamp	date → @timestamp

Code.

PUT _ingest/pipeline/nginx-ingest
{
  "description": "Parse, type, timestamp, enrich, and filter NGINX access logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IPORHOST:client_ip} - - \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}\" %{NUMBER:status:int} %{NUMBER:bytes:long}"
        ]
      }
    },
    { "drop": { "if": "ctx.path != null && ctx.path.startsWith('/healthz')" } },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
        "target_field": "@timestamp"
      }
    },
    { "convert": { "field": "status", "type": "integer", "ignore_missing": true } },
    { "geoip":   { "field": "client_ip", "target_field": "geo", "ignore_missing": true } },
    { "remove":  { "field": ["timestamp", "message"], "ignore_missing": true } }
  ],
  "on_failure": [
    { "set": { "field": "ingest_error", "value": "{{ _ingest.on_failure_message }}" } }
  ]
}

// Test the pipeline with the _simulate API before wiring it up
POST _ingest/pipeline/nginx-ingest/_simulate
{
  "docs": [
    { "_source": { "message": "192.168.1.10 - - [03/Aug/2026:10:15:32 +0000] \"GET /api/orders HTTP/1.1\" 200 1534" } },
    { "_source": { "message": "10.0.0.5 - - [03/Aug/2026:10:15:33 +0000] \"GET /healthz HTTP/1.1\" 200 2" } }
  ]
}

Step-by-step explanation.

• grok structures the raw line. Named patterns (%{IPORHOST}, %{HTTPDATE}, %{WORD}) match against the message field and extract typed captures. %{NUMBER:status:int} even casts inline. This replaces a Logstash grok filter, running inside the cluster.
• drop filters noise conditionally. The if Painless condition discards /healthz requests so health-check spam never reaches a shard. The null check guards documents where grok failed to set path.
• date builds the real @timestamp. It parses the 03/Aug/2026:10:15:32 +0000 string with an explicit format into a proper date field so time-range queries and time-bucket aggregations work. Without this, the timestamp is just text.
• convert and geoip type and enrich. convert ensures status is an integer even if grok's inline cast is bypassed; geoip turns client_ip into geo.country_name, geo.location, etc. — a lookup that would otherwise be a Spark join.
• on_failure and remove handle hygiene. A pipeline-level on_failure captures parse errors into an ingest_error field (for a dead-letter query) instead of dropping the doc silently; remove deletes the now-redundant raw message and the intermediate timestamp string. _simulate lets you test all of this without indexing anything.

Output.

Input line	Result
... "GET /api/orders ..." 200 1534	{client_ip, method:GET, path:/api/orders, status:200, bytes:1534, @timestamp, geo:{...}}
... "GET /healthz ..." 200 2	dropped (never indexed)
malformed line	indexed with ingest_error set

Rule of thumb. Build ingest pipelines against the _simulate API before you attach them, always add a pipeline-level on_failure so bad documents are quarantined not lost, and push parsing/typing/enrichment into the pipeline rather than making every producer do it. The pipeline is your in-cluster transform stage — treat it like versioned ETL code.

Worked example — the enrich processor as a dimension join

Detailed explanation. Documents arrive with a bare product_id; you want product_name and category denormalised in so search and aggregation don't need a join. The enrich processor performs this dimension-table join at ingest time. Walk through the four steps: source index → enrich policy → execute → processor.

• Source (dimension) index. products-catalog with product_id, product_name, category.
• Enrich policy. Match on product_id, enrich with product_name + category.
• Execute. Materialise the enrich index (a snapshot).
• Processor. enrich in the pipeline joins incoming product_id.

Question. Set up an enrich policy and processor that denormalises product attributes into incoming order events.

Input.

Order event (in)	Enrich source match	Enriched order (out)
{product_id: "P-42", qty: 2}	P-42 → {name:"Widget", category:"tools"}	{product_id, qty, product:{name, category}}

Code.

// 1. The dimension/source index (slowly-changing)
PUT products-catalog/_doc/P-42
{ "product_id": "P-42", "product_name": "Widget", "category": "tools" }

// 2. Define the enrich policy (match_field + enrich_fields)
PUT _enrich/policy/product-policy
{
  "match": {
    "indices": "products-catalog",
    "match_field": "product_id",
    "enrich_fields": ["product_name", "category"]
  }
}

// 3. Execute it to materialise the enrich index (re-run when catalog changes)
POST _enrich/policy/product-policy/_execute

// 4. Use it in an ingest pipeline
PUT _ingest/pipeline/orders-enrich
{
  "processors": [
    {
      "enrich": {
        "policy_name": "product-policy",
        "field": "product_id",
        "target_field": "product",
        "max_matches": 1
      }
    }
  ]
}

// Verify with _simulate
POST _ingest/pipeline/orders-enrich/_simulate
{ "docs": [ { "_source": { "product_id": "P-42", "qty": 2 } } ] }
// -> _source: { product_id:"P-42", qty:2, product:{ product_name:"Widget", category:"tools" } }

Step-by-step explanation.

• The source index holds the dimension data. products-catalog is your product dimension table; each doc keys on product_id. This is the slowly-changing side of the join.
• The enrich policy declares the join. match_field: product_id is the join key; enrich_fields lists the columns to pull in. This is the equivalent of SELECT product_name, category FROM products WHERE product_id = ?.
• Executing the policy materialises a read-optimised enrich index. Enrichment reads from this snapshot, not the live source, so it is fast and consistent — but you must re-execute the policy when the catalog changes to refresh it. That's the "slowly-changing dimension" constraint.
• The enrich processor performs the per-document join. For each incoming order, it looks up product_id in the enrich index and nests the matched product_name/category under product. max_matches: 1 treats it as a one-to-one join.
• _simulate proves the join before production. The order event {product_id, qty} comes out with product: {product_name, category} denormalised in — no query-time join, no application-side lookup. Search and aggregation now work directly on product.category.

Output.

Stage	Data
Order in	{product_id:"P-42", qty:2}
Enrich lookup	P-42 → {product_name:"Widget", category:"tools"}
Order indexed	{product_id:"P-42", qty:2, product:{product_name:"Widget", category:"tools"}}

Rule of thumb. Use the enrich processor for slowly-changing dimension joins at ingest time, and remember to re-execute the enrich policy whenever the source dimension changes — the enrich index is a snapshot, not a live view. For fast-mutating lookups, do the join downstream instead.

Worked example — a resilient bulk loader with the Python client

Detailed explanation. Load 500 million documents through the nginx-ingest pipeline without melting the cluster or losing errors. Use helpers.parallel_bulk with tuned chunk size, a default pipeline, per-item error collection into a dead-letter file, and backpressure via bounded concurrency. This is the production backfill loader.

• Throughput knobs. chunk_size (docs per request), thread_count, queue_size, and cluster-side refresh_interval (set to -1 during the load).
• Correctness. raise_on_error=False + inspect each item; write failures to a dead-letter file.
• Safety. Set number_of_replicas: 0 and refresh_interval: -1 during load, restore after — the standard bulk-backfill tuning.

Question. Write the Python bulk loader that streams documents through the ingest pipeline, tunes throughput, and dead-letters per-item failures.

Input.

Knob	Load-time value	Steady-state value
number_of_replicas	0	1
refresh_interval	-1	5s
chunk_size	2000	—
thread_count	4	—

Code.

import json
from elasticsearch import Elasticsearch, helpers

es = Elasticsearch("https://search-cluster:9200", api_key="...", request_timeout=120)

INDEX = "nginx-2026.08"

# 1. Tune the index for a heavy backfill (no replicas, no refresh)
es.indices.put_settings(index=INDEX, body={
    "index": {"number_of_replicas": 0, "refresh_interval": "-1"}
})

# 2. Stream actions; the server-side ingest pipeline runs per document
def actions(lines):
    for raw in lines:                      # raw NGINX log lines
        yield {
            "_index": INDEX,
            "pipeline": "nginx-ingest",    # server-side transform
            "_source": {"message": raw.rstrip("\n")},
        }

# 3. parallel_bulk with per-item error handling -> dead-letter file
failed = 0
with open("dead_letter.ndjson", "w") as dlq:
    for ok, item in helpers.parallel_bulk(
        es,
        actions(open("/data/access.log")),
        chunk_size=2000,
        thread_count=4,
        queue_size=8,
        raise_on_exception=False,
        raise_on_error=False,
    ):
        if not ok:
            failed += 1
            dlq.write(json.dumps(item) + "\n")   # capture the failing item

print(f"backfill complete; {failed} docs dead-lettered")

# 4. Restore steady-state settings and force a refresh
es.indices.put_settings(index=INDEX, body={
    "index": {"number_of_replicas": 1, "refresh_interval": "5s"}
})
es.indices.refresh(index=INDEX)

Step-by-step explanation.

• Step 1 tunes the index for write throughput. Setting number_of_replicas: 0 means each document is written once (not replicated) during the load; refresh_interval: -1 disables the periodic segment refresh that otherwise burns CPU making documents searchable every second. Both are restored afterward.
• Step 2 attaches the pipeline per action. Each yielded action names pipeline: nginx-ingest, so the grok/date/geoip transform runs server-side — the loader ships raw log lines and the cluster does the T. helpers batches these into NDJSON bulk requests automatically.
• Step 3 parallelises with bounded concurrency. parallel_bulk runs thread_count workers, each sending chunk_size-document bulk requests; queue_size bounds in-flight batches so the producer can't outrun the cluster (backpressure). raise_on_error=False yields (ok, item) per document instead of aborting the whole load.
• Per-item failures are dead-lettered, not lost. A mapping conflict or pipeline failure on one document writes that item to dead_letter.ndjson and increments a counter; the other 1,999 docs in the batch still index. This is the partial-success contract of the bulk API made operational.
• Step 4 restores durability and visibility. After the load, replicas go back to 1 (HA restored) and refresh_interval back to 5s; an explicit refresh makes the freshly-loaded documents immediately searchable. Skipping this step leaves the index unreplicated and effectively invisible to search.

Output.

Metric	With naive single-doc POST	With tuned parallel_bulk
Throughput	~500 docs/s	~40,000+ docs/s
Replica writes during load	yes (2× work)	none (replicas=0)
Refresh overhead	every 1s	disabled during load
Failed docs	abort whole load	dead-lettered, load continues
Time for 500M docs	days	hours

Rule of thumb. For any large backfill: drop replicas to 0, set refresh_interval: -1, load with parallel_bulk (chunk ~2,000, a few threads, bounded queue), dead-letter per-item failures, then restore replicas + refresh and reindex the dead-letter file after fixing the mapping. Never abort a 500M-doc load because 12 documents were malformed.

Streaming interview question on ingest pipelines and bulk loading

A senior interviewer might ask: "You need to backfill two years of raw JSON events (about 800 million documents) into a new index, applying a grok parse, a timestamp parse, and an enrich join against a customer dimension — with zero downtime for the existing search alias. Walk me through the ingest pipeline, the reindex/bulk strategy, the throughput tuning, error handling, and the alias-swap cutover."

Solution Using reindex-with-pipeline, tuned bulk settings, dead-lettering, and an alias swap

// 1. Build the transform pipeline (parse + timestamp + enrich)
PUT _ingest/pipeline/events-v2
{
  "processors": [
    { "json":  { "field": "message", "add_to_root": true, "if": "ctx.message != null" } },
    { "date":  { "field": "ts", "formats": ["ISO8601"], "target_field": "@timestamp" } },
    { "enrich":{ "policy_name": "customer-policy", "field": "customer_id",
                 "target_field": "customer", "max_matches": 1, "ignore_missing": true } },
    { "remove":{ "field": ["message", "ts"], "ignore_missing": true } }
  ],
  "on_failure": [ { "set": { "field": "_ingest_error", "value": "{{ _ingest.on_failure_message }}" } } ]
}

// 2. Create the destination with load-optimised settings + strict mapping
PUT events-v2-000001
{
  "settings": { "number_of_shards": 6, "number_of_replicas": 0, "refresh_interval": "-1" },
  "mappings": { "dynamic": "strict", "properties": {
    "@timestamp": {"type":"date"}, "customer_id":{"type":"keyword"},
    "level":{"type":"keyword"}, "message_text":{"type":"text"},
    "customer": {"properties": {"name":{"type":"keyword"},"tier":{"type":"keyword"}}}
  }}
}

// 3. Reindex the old data THROUGH the pipeline, sliced + throttled + async
POST _reindex?wait_for_completion=false&requests_per_second=20000
{
  "source": { "index": "events-v1", "size": 2000, "slice": { "max": 6 } },
  "dest":   { "index": "events-v2-000001", "pipeline": "events-v2" }
}
// returns a task id; poll GET _tasks/<task_id>

// 4. After reindex completes: restore durability, then swap the alias atomically
PUT events-v2-000001/_settings
{ "index": { "number_of_replicas": 1, "refresh_interval": "5s" } }

POST _aliases
{
  "actions": [
    { "remove": { "index": "events-v1",        "alias": "events" } },
    { "add":    { "index": "events-v2-000001", "alias": "events" } }
  ]
}

Step-by-step trace.

Phase	Action	Why
Pipeline	json + date + enrich + on_failure	parse, timestamp, dimension join, quarantine errors
Dest index	6 shards, 0 replicas, refresh −1	~50–300 GB total sized to ~10–50 GB/shard; write-tuned
Reindex	sliced (6), throttled 20k rps, async task	parallel + backpressure; doesn't saturate the cluster
Restore	replicas→1, refresh→5s	durability + searchability back
Cutover	atomic _aliases remove+add	zero-downtime; readers never see a gap

The existing events alias points at events-v1 throughout the entire backfill, so live search is uninterrupted. The reindex streams every v1 document through events-v2 (parsing the raw JSON, building @timestamp, joining the customer dimension), sliced six ways for parallelism and throttled to 20k docs/sec so it never starves live queries. Only after the new index is fully built, replicated, and refreshed does the single atomic _aliases call move readers over — old and new never overlap, and rollback is one more alias swap.

Output:

Concern	Mechanism	Result
Transform on backfill	reindex dest.pipeline	historical data reshaped
Throughput	slice.max=6 + requests_per_second	parallel, throttled
Bad documents	pipeline on_failure → _ingest_error	quarantined, queryable
Zero downtime	atomic alias remove+add	no read gap
Rollback	swap alias back to v1	instant revert

Why this works — concept by concept:

• Reindex-with-pipeline — dest.pipeline runs the ingest transform on every copied document, so a mapping/analyzer change is retrofitted onto historical data in one operation instead of a bespoke Spark job.
• Sliced scroll + throttle — slice.max parallelises the reindex across shards while requests_per_second caps the rate, giving parallel throughput without starving live search — the two knobs that make a big reindex safe.
• Write-optimised destination — 0 replicas and refresh_interval: -1 during the load remove replication and refresh overhead; restoring them afterward returns the index to a durable, searchable steady state.
• Atomic alias swap — _aliases applies remove+add in a single cluster-state update, so readers move from v1 to v2 with no in-between state. This is the zero-downtime index-migration pattern, and the reason production indices are always accessed through an alias, never by name.
• Cost — one reindex pass (O(N-docs), throttled to protect the cluster) plus a strict mapping and a pipeline. The eliminated cost is downtime during migration and a hand-rolled reprocessing job. The alias indirection makes the cutover and any rollback O(1).

ETL
Topic — etl
ETL problems on transform pipelines and backfills

Practice →

Streaming Topic — streaming Streaming problems on high-throughput bulk ingestion

Practice →

---

5. Shards, scaling and interview signals

A shards and replicas mistake is the most common cause of an unhealthy cluster — right-size shards, replicate for HA, aggregate correctly, and know when Elasticsearch is the wrong tool

The mental model in one line: a shard is a self-contained Lucene index that is the unit of horizontal scaling and distribution — an index is split into primary shards (the data, fixed at creation) and each primary can have replica shards (copies on other nodes for HA and read throughput) — and the two failure modes that dominate real clusters are over-sharding (too many tiny shards bloating cluster-state and heap) and aggregating on the wrong field type; a data engineer must size shards to ~10–50 GB, set replicas for durability, use aggregations correctly, and recognise the workloads where Elasticsearch is simply the wrong database. Getting shard math and aggregation type-safety right is what separates a healthy green cluster from a 3 AM red-cluster page.

Shards and replicas — the scaling primitives.

• Primary shard. A slice of the index's data; a full Lucene index in its own right. The number of primaries is fixed at index creation (you can't add more without reindex or _split/_shrink). Documents are routed to a primary by a hash of the document ID.
• Replica shard. A copy of a primary on a different node. Replicas provide high availability (survive a node loss) and extra read/search throughput (queries can hit replicas). number_of_replicas is changeable any time. replicas = 1 (one copy) is the sane default; 0 only for reproducible/transient data.
• Node roles. Data nodes hold shards; master nodes manage cluster state; ingest nodes run pipelines; coordinating nodes fan out queries. In big clusters these are separated; in small ones a node does everything.
• Cluster health. Green (all primaries + replicas assigned), yellow (all primaries assigned, some replicas not — often just "replicas=1 on a single node"), red (some primary unassigned — data unavailable). Red is an incident; yellow is often benign.

Shard sizing — the number that matters most.

• The target. ~10–50 GB per shard is the widely-used sweet spot. Below ~10 GB you tend to over-shard; above ~50 GB recovery/rebalancing gets slow. For time-series, size the rollover so each daily/rolled index lands in that band.
• The over-sharding tax. Every shard costs heap (segment metadata, field mappings) and cluster-state overhead regardless of size. Thousands of tiny shards can exhaust master heap and slow every operation. The classic anti-pattern: number_of_shards: 5 on a 100 MB/day index → 5 tiny shards/day → thousands of near-empty shards.
• The formula, roughly. primaries ≈ ceil(expected_index_size / target_shard_size). A 300 GB index at 40 GB/shard → ~8 primaries. A 2 GB index → 1 primary. Don't default to 5.
• Rollover + ILM. Time-series indices use an alias + rollover (roll to a new index at a size/age/doc threshold) plus Index Lifecycle Management (ILM) to move indices hot → warm → cold → delete. Hot nodes (fast SSD, active writes) hold recent data; cold nodes (cheap disk) hold searchable-but-old data; then delete on retention.

Aggregations — and the analyzed-field trap.

• Two families. Bucket aggregations group documents (terms, date_histogram, range, histogram); metric aggregations compute over a bucket (avg, sum, min, max, percentiles, cardinality). They nest: "average latency (metric) per service (bucket) per hour (bucket)."
• The analyzed-field trap. Aggregating on a text field either errors (fielddata disabled) or buckets by analyzed terms (wrong) — always aggregate on a keyword field or .keyword sub-field. This is the single most common aggregation bug (covered in section 2) and a favourite interview probe.
• cardinality is approximate. Distinct-count uses HyperLogLog++ — fast and low-memory but approximate at high cardinality. Know this before you quote an exact unique-user count from it.
• Cost awareness. High-cardinality terms aggregations and deep nesting are memory-heavy; size controls how many buckets return but the agg still computes over all matching docs. search.max_buckets guards against runaway aggregations.

When NOT to use Elasticsearch — the senior signal.

• As the system of record. Elasticsearch is a search/analytics index, not a durable source of truth. It has no real transactions, near-real-time (not immediate) consistency, and can lose data on split-brain misconfig. Keep the authoritative copy in Postgres/S3/Kafka and index into Elasticsearch.
• For relational joins. No general joins. nested/join types are narrow and costly; denormalise (via the enrich processor or at ETL time) instead. If your workload is join-heavy relational analytics, it's a warehouse job.
• For high-write OLTP / frequent single-doc updates. Updates are delete-then-reindex of the whole document into a new segment; heavy update workloads thrash merges. Elasticsearch loves append-mostly, read-heavy workloads.
• For small data. If it fits comfortably in Postgres with a GIN index or a single-node solution, a whole Elasticsearch cluster is operational overkill. Reach for it when full-text relevance, faceted search at scale, or log/observability analytics justify the cluster.

Worked example — sizing shards for a time-series index

Detailed explanation. An index ingests ~50 GB/day of logs with 30-day retention. Choose the shard count per rolled index, the replica count, and the rollover/ILM policy to keep every shard in the 10–50 GB band and avoid both over- and under-sharding. Walk through the math.

• Daily volume. ~50 GB/day primary data.
• Target shard size. ~40 GB.
• Retention. 30 days hot-ish, then delete.

Question. Compute the shard count per daily index and write the ILM + rollover policy.

Input.

Parameter	Value
Daily primary volume	50 GB
Target shard size	40 GB
Replicas	1
Retention	30 days

Code.

// primaries per daily index = ceil(50 GB / 40 GB) = 2
PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,          // 2 x ~25 GB primaries -> in the band
      "number_of_replicas": 1,        // + 2 replica shards on other nodes
      "index.lifecycle.name": "logs-ilm",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

// ILM: roll at 40 GB or 1 day; warm at 2d; delete at 30d
PUT _ilm/policy/logs-ilm
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "40gb", "max_age": "1d" } } },
      "warm":   { "min_age": "2d",  "actions": { "shrink": { "number_of_shards": 1 },
                                                 "forcemerge": { "max_num_segments": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

Step-by-step explanation.

• The shard count comes from the math, not a default. ceil(50 GB / 40 GB) = 2 primaries per daily index → two ~25 GB shards, comfortably inside 10–50 GB. Defaulting to 5 would give five 10 GB shards/day → 150 primaries over 30 days plus replicas → needless over-sharding.
• max_primary_shard_size drives rollover. ILM rolls to a new index when a primary shard hits 40 GB or the index is 1 day old — whichever first. This keeps shards bounded even if volume spikes, so a traffic surge doesn't create one giant 200 GB shard.
• The warm phase shrinks and force-merges. After 2 days (no more writes), shrink collapses the 2 primaries to 1 (older data doesn't need write parallelism) and forcemerge to a single segment reclaims deleted-doc space and speeds up reads. This roughly halves the shard count for aged data.
• The delete phase enforces retention. At 30 days, ILM deletes the index automatically — no cron job, no manual cleanup. Retention is policy, not tribal knowledge.
• Replicas give HA without touching primary math. number_of_replicas: 1 doubles storage but means any single node loss keeps the cluster green. Replica count is independent of the primary-sizing decision and can be changed live.

Output.

Metric	Naive (5 shards, no ILM)	Sized (2 shards + ILM)
Shards/day (with replicas)	10	4 (2 after warm-shrink)
Shards over 30 days	~300	~90
Shard size	~10 GB	~25 GB
Retention cleanup	manual	automatic
Over-sharding risk	high	controlled

Rule of thumb. Size primaries by ceil(index_size / 40 GB), never by the default 5; drive rollover with max_primary_shard_size; shrink + force-merge in the warm phase; and let ILM delete on retention. A time-series cluster's health is mostly a function of getting this one policy right.

Worked example — a correct multi-level aggregation

Detailed explanation. Build "average and p95 latency per service, over time, for errors only" — a nested bucket-and-metric aggregation that must aggregate on keyword fields and a numeric field, never on analyzed text. Walk through the structure and the type-safety.

• Filter. level = ERROR (keyword term).
• Bucket 1. date_histogram on @timestamp (hourly).
• Bucket 2. terms on service (keyword).
• Metrics. avg and percentiles(95) on latency_ms (long).

Question. Write the aggregation and explain why every field choice is type-safe.

Input.

Clause	Field	Type	Why valid
filter	level	keyword	exact term
bucket	@timestamp	date	time buckets
bucket	service	keyword	exact grouping
metric	latency_ms	long	numeric stats

Code.

GET app-logs/_search
{
  "size": 0,
  "query": { "term": { "level": "ERROR" } },
  "aggs": {
    "per_hour": {
      "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" },
      "aggs": {
        "per_service": {
          "terms": { "field": "service", "size": 20 },
          "aggs": {
            "avg_latency": { "avg": { "field": "latency_ms" } },
            "p95_latency": { "percentiles": { "field": "latency_ms", "percents": [95] } }
          }
        }
      }
    }
  }
}

Step-by-step explanation.

• size: 0 returns no hits, only aggregations. You don't want the matching documents, just the computed buckets — this avoids shipping millions of docs back.
• The term filter uses a keyword field. level is a keyword, so level = ERROR is an exact, un-analyzed match — fast (filter context, cacheable, no scoring). Filtering first shrinks the doc set the aggregations run over.
• date_histogram buckets time correctly because @timestamp is a real date type (built by the ingest date processor). fixed_interval: 1h gives even hourly buckets; on a text timestamp this would be impossible.
• terms on service (keyword) buckets by the whole value. Because service is a keyword, each distinct service is one bucket — not split by analyzer. size: 20 caps returned buckets; the agg still scans all matching docs.
• avg and percentiles need a numeric field. latency_ms is a long with doc_values, so both metrics compute directly from the columnar store. Percentiles use a t-digest approximation — fast and accurate enough for SLOs. Every field in the whole aggregation is a keyword, date, or numeric — never analyzed text — which is exactly what makes it correct.

Output.

Hour	Service	avg_latency	p95_latency
10:00	payment-gateway	142 ms	480 ms
10:00	auth-service	88 ms	210 ms
11:00	payment-gateway	156 ms	505 ms

Rule of thumb. Aggregate only on keyword, numeric, date, boolean, or ip fields — never on analyzed text. Filter in filter context (term/range) to shrink the doc set before bucketing, cap terms with size, and remember cardinality and percentiles are approximate. Type-correct fields are what make aggregations both fast and right.

Worked example — diagnosing a red cluster

Detailed explanation. The cluster went red: some primary shards are unassigned, so part of the data is unavailable. Walk through the systematic diagnosis using _cluster/health, _cat/shards, and _cluster/allocation/explain, and the common root causes and fixes.

• Red = unassigned primary. Data is missing/unavailable, not just un-replicated (that's yellow).
• Common causes. Node left the cluster (disk full watermark, OOM), too many shards per node, or a corrupted shard.
• The tools. _cluster/health, _cat/shards?v, _cluster/allocation/explain.

Question. Give the diagnostic sequence and the fix for the most common red-cluster cause (disk watermark).

Input.

Symptom	Likely cause
red, unassigned primaries	node left / disk full / too many shards
allocation/explain says "disk watermark"	data node above flood-stage disk
allocation/explain says "max shards per node"	over-sharding hit the cap

Code.

// 1. Confirm red and count unassigned
GET _cluster/health
// -> { "status": "red", "unassigned_shards": 3, ... }

// 2. Find the unassigned shards
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state
// -> logs-2026.08.03 0 p UNASSIGNED NODE_LEFT

// 3. Ask WHY a specific shard can't be assigned
GET _cluster/allocation/explain
{ "index": "logs-2026.08.03", "shard": 0, "primary": true }
// -> "cannot allocate ... disk usage exceeded flood-stage watermark (95%)"

// 4a. Fix: free disk / raise watermark temporarily, then release the write block
PUT _cluster/settings
{ "transient": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.high": "92%"
}}
PUT */_settings
{ "index.blocks.read_only_allow_delete": null }   // clear the auto read-only block

// 4b. Real fix: add disk/nodes or delete old indices (ILM), then re-check health

Step-by-step explanation.

• Step 1 confirms severity. _cluster/health distinguishes red (unassigned primary — data unavailable) from yellow (unassigned replica — HA reduced but data intact). Red is a page; yellow often isn't.
• Step 2 localises the problem. _cat/shards with unassigned.reason shows exactly which shards are unassigned and why (NODE_LEFT, ALLOCATION_FAILED, DISK_WATERMARK). Sorting by state surfaces the unassigned ones.
• Step 3 gets the authoritative reason. allocation/explain is the single best tool: for a chosen shard it explains, node by node, why Elasticsearch won't place it. "Disk flood-stage watermark exceeded" is the most common answer — a data node crossed 95% disk, so Elasticsearch stopped allocating shards there and set indices read-only.
• Step 4a is the immediate mitigation. Temporarily nudging the watermarks and clearing the read_only_allow_delete block lets writes resume and shards re-allocate — buys time. This is a stopgap, not a fix.
• Step 4b is the real fix. Add disk or nodes, or delete old indices (ideally via an ILM retention policy so it never recurs). The durable cure for a disk-watermark red cluster is capacity + retention, and for a "max shards per node" red cluster it's fewer, larger shards — i.e., the sizing discipline from the earlier example.

Output.

Diagnostic	Command	Tells you
Severity	_cluster/health	red vs yellow, unassigned count
Which shards	_cat/shards	index/shard + unassigned reason
Why	allocation/explain	exact allocation decision
Fix (stopgap)	watermark + clear block	writes resume
Fix (durable)	add disk / ILM delete / fewer shards	prevents recurrence

Rule of thumb. Red cluster? _cluster/health → _cat/shards → allocation/explain, in that order — never guess. The most common cause is a disk watermark from an unretained, over-sharded index, and the durable fix is capacity plus an ILM retention policy plus correct shard sizing. Prevention (sizing + ILM) beats every 3 AM mitigation.

Design interview question on scaling and tool choice

A senior interviewer might ask: "A team wants to make Elasticsearch the primary database for a new orders service — every order write goes to Elasticsearch, and they'll read orders back by ID and run relational joins against a customers index. They picked 30 shards for a 5 GB/day index 'to be safe.' As the data engineer, push back: explain the shard sizing, the replica strategy, why this is a tool-choice mistake, and the architecture you'd recommend instead."

Solution Using right-sized shards, Elasticsearch as a secondary index, and Postgres as the source of truth

// 1. Right-size the index: 5 GB/day, 14-day retention -> 1 shard/day, not 30
PUT _index_template/orders-search-template
{
  "index_patterns": ["orders-search-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,          // 5 GB << 40 GB target -> ONE primary
      "number_of_replicas": 1,        // HA: survive a node loss
      "index.lifecycle.name": "orders-ilm"
    },
    "mappings": { "dynamic": "strict", "properties": {
      "order_id":    { "type": "keyword" },
      "customer_id": { "type": "keyword" },
      "status":      { "type": "keyword" },
      "total_cents": { "type": "long" },
      "items_text":  { "type": "text" },
      "created_at":  { "type": "date" },
      "customer":    { "properties": { "name": {"type":"keyword"}, "tier": {"type":"keyword"} } }
    }}
  }
}

// 2. Denormalise the customer dimension at ingest (no query-time joins)
PUT _ingest/pipeline/orders-search
{
  "processors": [
    { "enrich": { "policy_name": "customer-policy", "field": "customer_id",
                  "target_field": "customer", "max_matches": 1, "ignore_missing": true } }
  ]
}

Recommended architecture (source of truth stays in Postgres):

  Postgres (orders)  --CDC (Debezium/log-based)-->  Kafka  -->  bulk + ingest  -->  Elasticsearch
     ^  system of record                                                              ^ secondary search index
     |  transactions, joins, PK reads                                                 |  full-text + facets + aggs

Step-by-step trace.

Their plan	Problem	Recommendation
30 shards for 5 GB/day	30 × ~170 MB shards = massive over-sharding	1 primary shard (5 GB ≪ 40 GB)
ES as source of truth	no transactions, NRT consistency, can lose data	Postgres = source of truth; ES = secondary index
Query-time joins to customers	ES has no real joins	denormalise via enrich at ingest
Read orders by ID from ES	works but wasteful; PK reads belong in Postgres	serve PK reads from Postgres; ES for search
Replicas unspecified	no HA	replicas: 1

The push-back is threefold. First, shard math: a 5 GB/day index needs one primary shard, not 30 — 30 shards means ~170 MB each, and over two weeks that's hundreds of near-empty shards taxing heap and cluster state for zero benefit. Second, tool choice: Elasticsearch is a near-real-time search/analytics index, not a transactional system of record — it has no ACID transactions, only eventual (refresh-delayed) visibility, and no general joins, so making it the orders database risks lost writes and forces denormalisation anyway. Third, the right shape: keep Postgres as the authoritative store (transactions, PK reads, relational joins), stream changes via CDC into Elasticsearch as a secondary index, and denormalise the customer dimension with the enrich processor so search/facets/aggregations run without joins.

Output:

Requirement	Wrong (ES as DB)	Right (ES as index)
Durable orders	at risk in ES	Postgres (ACID)
PK read by order_id	ES (wasteful)	Postgres (indexed)
Full-text on items	ES	ES (its strength)
Facets/aggregations	ES	ES (its strength)
customer join	impossible in ES	denormalised at ingest
Shards for 5 GB/day	30 (over-sharded)	1

Why this works — concept by concept:

• Shard sizing by volume — 5 GB against a 40 GB target is one shard. Over-sharding taxes heap and cluster state per shard regardless of size, so 30 shards is pure overhead; the fix is arithmetic, not intuition.
• Elasticsearch as a secondary index — the authoritative, transactional copy lives in Postgres; Elasticsearch is fed by CDC and serves the read patterns it's good at (full-text, facets, aggregations). This respects Elasticsearch's near-real-time, non-transactional nature instead of fighting it.
• Denormalise instead of join — because Elasticsearch has no general joins, the enrich processor folds the customer dimension into each order document at ingest, turning a would-be query-time join into a precomputed field. Denormalisation is the Elasticsearch way.
• Replicas for HA — number_of_replicas: 1 keeps the index available through a single node loss; it's independent of the primary-sizing decision and adjustable live.
• Cost — one shard plus a replica plus a CDC stream, versus a fragile 30-shard "database." The eliminated costs are lost-write risk, impossible joins, and the over-sharding tax. Right tool, right shape: O(1) shard math, O(N) only in the CDC stream that Elasticsearch is designed to consume.

Analytics
Topic — real-time-analytics
Real-time analytics and aggregation problems

Practice →

ETL
Topic — etl
ETL problems on sink sizing and tool selection

Practice →

---

Cheat sheet — Elasticsearch / OpenSearch recipes

• The four surfaces you own. Mapping (typed schema), analyzer (per-text-field tokenisation), ingest pipeline (in-cluster transform), shards/replicas/ILM (scaling + lifecycle). Everything applies to both Elasticsearch and OpenSearch — the fork (2021 relicense → AWS OpenSearch → 2024 AGPL) diverges on advanced query languages and ML, not on this core model.
• text vs keyword decision. text = analyzed, full-text searchable, not aggregatable/sortable (no doc_values). keyword = verbatim single term, exact filter + sort + aggregate. Use the text + .keyword multi-field only when you need both. If you ever terms-agg or sort on it, it's a keyword.
• Inverted index mental model. Term → posting list (sorted doc IDs + positions + freqs), so full-text lookup is a dictionary hit not a scan. doc_values is the columnar sibling for sort/agg. Analyzed text has no doc_values by default — hence the aggregation error.
• Mapping-explosion guard. dynamic: "strict" (reject unknown fields) or "false" (store-not-index), flattened type for open-ended key/value objects, index.mapping.total_fields.limit as a hard cap, dynamic templates to route fields by name/type. Never let user-supplied keys become mapping fields.
• Index + component templates. PUT _component_template/base for reusable settings; PUT _index_template/x { index_patterns, composed_of, priority, template:{settings, mappings} } binds schema + index.default_pipeline + shard count to a pattern (events-*). The one artifact that makes time-series indexing reproducible.
• Analyzer anatomy. char filters (html_strip, mapping, pattern_replace) → one tokenizer (standard, whitespace, keyword, edge_ngram) → token filters (lowercase, stop, stemmer, synonym, asciifolding) in order. Debug with POST _analyze {analyzer, text}. Filter order: lowercase → synonym → stop → stemmer.
• Autocomplete recipe. edge_ngram filter (min 2, max 20) as the index analyzer + a plain search_analyzer so the query isn't itself n-grammed. Always bound max_gram. If autocomplete "matches everything after two letters," you edge-ngrammed the query.
• Ingest processor toolbox. Parse (grok, dissect, kv, json, csv), reshape (set, rename, remove, convert, gsub, split), enrich (date, geoip, user_agent, enrich), control (drop, if, pipeline, on_failure). Test with POST _ingest/pipeline/x/_simulate. Always add a pipeline-level on_failure to quarantine, not lose, bad docs.
• Enrich = dimension join. source index → PUT _enrich/policy (match_field + enrich_fields) → POST .../_execute (materialise) → enrich processor. Re-execute the policy when the source changes — it's a point-in-time snapshot, so it fits slowly-changing dimensions.
• Bulk API essentials. NDJSON: alternating action-metadata line + source line. HTTP 200 even with per-item failures — always check the errors flag and per-item status. Size ~5–15 MB or 1k–5k docs/request. Python: helpers.bulk / streaming_bulk / parallel_bulk with raise_on_error=False and a dead-letter sink.
• Backfill tuning. Before a big load: number_of_replicas: 0 + refresh_interval: -1. Reindex through a pipeline with slice.max (parallel) + requests_per_second (throttle) + wait_for_completion=false (async task). After: restore replicas/refresh, _refresh, reindex the dead-letter file. Cut over with an atomic _aliases remove+add.
• Shard math. primaries ≈ ceil(index_size / 40 GB); target 10–50 GB/shard; never default to 5. Time-series: drive rollover by max_primary_shard_size, shrink + forcemerge in warm, delete on retention via ILM. Over-sharding taxes heap + cluster state per shard regardless of size.
• Aggregation rules. Bucket (terms, date_histogram, range) + metric (avg, percentiles, cardinality) nest. Aggregate only on keyword/numeric/date/boolean/ip — never analyzed text. cardinality and percentiles are approximate. Filter in filter context first to shrink the doc set; cap terms with size.
• When NOT to use ES. Not a system of record (no ACID, NRT consistency) — keep truth in Postgres/S3/Kafka. No general joins — denormalise via enrich. Not for high-write OLTP / frequent single-doc updates (delete-then-reindex thrashes merges). Not for small data that fits a Postgres GIN index. Reach for it for full-text relevance, faceted search at scale, and log/observability analytics.
• Red vs yellow cluster. Green = all shards assigned; yellow = replica unassigned (data safe, HA reduced); red = primary unassigned (data unavailable — incident). Diagnose red with _cluster/health → _cat/shards?v (unassigned.reason) → _cluster/allocation/explain. Most common cause: disk flood-stage watermark on an over-sharded, unretained index.

Frequently asked questions

Elasticsearch vs OpenSearch — which should a data engineer pick?

Both descend from the same Apache-2.0 Elasticsearch codebase, so the core model this guide covers — mappings, the inverted index, analyzers, ingest pipelines, the bulk API, shards and replicas, aggregations — is essentially identical on either. The split happened in 2021 when Elastic relicensed Elasticsearch and Kibana away from Apache 2.0 (to SSPL / Elastic License), prompting AWS to fork the last open version into OpenSearch (now under the Linux Foundation's OpenSearch Software Foundation); in 2024 Elastic added AGPLv3 to make Elasticsearch OSI-open-source again, but by then both products existed and diverge on newer features. For a data engineer, pick based on ecosystem and licensing constraints, not the fundamentals: choose OpenSearch if you want a fully Apache-2.0/permissive stack or run on AWS's managed OpenSearch Service; choose Elasticsearch if you want Elastic's newest features (ES|QL, certain vector/ML capabilities) and are comfortable with its license. The skills transfer directly either way — someone fluent in the mapping/analyzer/ingest/shard model is productive on both.

text vs keyword — when do I use each?

Use keyword for any string you filter, sort, or aggregate on exactly — enums (status, level), identifiers (user_id, order_id), hostnames, tags, categories. A keyword field is stored verbatim as a single term with doc_values, so term filters and terms aggregations are exact and fast. Use text for prose you run free-text search over — message, description, title — where the string is run through an analyzer into multiple terms; a text field is full-text searchable but cannot be aggregated or sorted efficiently (analyzed fields have no doc_values, and aggregating on one either errors or buckets by analyzed terms). When a field genuinely needs both — a product name you search and sort — use the text + keyword multi-field (name for search, name.keyword for exact ops). The failure mode to avoid is letting dynamic mapping make everything text + .keyword: it wastes storage and, worse, tempts you to aggregate on the analyzed version and get split-term garbage.

What is an ingest pipeline, and when do I use one over Logstash or Spark?

An ingest pipeline is an ordered chain of processors (grok, set, convert, date, geoip, enrich, drop, script) that runs on an Elasticsearch/OpenSearch ingest node and transforms each document before it is indexed — it's the "T" of ETL executed inside the cluster. Use it when the transform is per-document and modest: parsing a log line, casting types, parsing a timestamp into @timestamp, geo-enriching an IP, joining a slowly-changing dimension via the enrich processor, or dropping noise. It's ideal because it needs no external service, runs close to the data, and attaches to the index via index.default_pipeline so every writer gets it. Reach for Logstash/OpenSearch Ingestion when you need buffering, many input/output plugins, or persistent queues between sources and the cluster; reach for Spark/Flink when the transform needs cross-document state, large joins, windowed aggregation, or reprocessing at warehouse scale. A common pattern is layered: heavy transforms in Spark, light per-document shaping in an ingest pipeline as the last mile before indexing.

How many shards should an index have?

Size primaries by data volume, not by a default: primaries ≈ ceil(expected_index_size / target_shard_size), with a target of roughly 10–50 GB per shard. A 5 GB index wants one primary; a 300 GB index wants about 8; a 50 GB/day time-series index wants ~1–2 primaries per daily/rolled index. The default of 5 is almost always wrong for small indices and causes over-sharding — thousands of tiny shards that each consume heap and cluster-state overhead regardless of size, eventually degrading or crashing the cluster. For time-series data, don't hand-pick per-index counts; use an alias plus rollover (roll at max_primary_shard_size) and an ILM policy to keep every rolled index in the band, then shrink/forcemerge aged indices and delete on retention. Add replicas (number_of_replicas: 1 is the sane default) separately for high availability and read throughput — replica count is independent of primary sizing and changeable at any time.

Why is my aggregation slow or returning wrong buckets?

Almost always because you're aggregating on an analyzed text field. Aggregating on a bare text field either errors ("Fielddata is disabled on text fields by default") or, if fielddata is enabled, buckets by the analyzed terms — so "payment-gateway" becomes separate payment and gateway buckets, which is wrong. The fix is to aggregate on a keyword field (or the .keyword sub-field of a multi-field), which has doc_values and stores the value verbatim. Beyond the type issue: filter in filter context (term/range inside a bool.filter) before aggregating to shrink the document set; cap terms aggregations with size (they still scan all matching docs, but return fewer buckets); remember cardinality (distinct count) and percentiles are approximate (HyperLogLog++ and t-digest), so don't quote them as exact; and watch high-cardinality terms and deep nesting, which are memory-heavy and guarded by search.max_buckets. Type-correct fields plus filter-first is what makes aggregations both fast and correct.

When should a data engineer NOT use Elasticsearch?

Don't use it as your system of record — it's a near-real-time search/analytics index with no ACID transactions, refresh-delayed (not immediate) read consistency, and data-loss risk under misconfiguration; keep the authoritative copy in Postgres/S3/Kafka and index into Elasticsearch. Don't use it for relational joins — it has no general join; the nested/join types are narrow and costly, so you denormalise (via the enrich processor or upstream ETL) instead, and if your workload is join-heavy relational analytics it belongs in a warehouse. Don't use it for high-write OLTP or frequent single-document updates — an update is a delete-then-reindex into a new segment, so update-heavy workloads thrash Lucene merges; Elasticsearch prefers append-mostly, read-heavy data. And don't stand up a whole cluster for small data that a Postgres GIN/full-text index or a single-node solution handles fine. Reach for Elasticsearch/OpenSearch when you genuinely need full-text relevance ranking, faceted search at scale, or log/observability analytics over large, append-mostly datasets — and pair it with a real source of truth feeding it via CDC or a streaming pipeline.

Practice on PipeCode

• Drill the ETL practice library → for the schema-design, type-mapping, ingest-transform, and backfill problems that map directly onto Elasticsearch mappings, ingest pipelines, and reindex-with-pipeline.
• Rehearse on the streaming practice library → for the high-throughput bulk-loading, CDC-into-index, and real-time write-path patterns that keep a search cluster fed without melting it.
• Sharpen the aggregation axis with the real-time analytics practice library → for the bucket-and-metric, time-bucketing, and cardinality problems behind Elasticsearch aggregations and dashboards.
• Work the parsing practice library → for the tokenisation, text-normalisation, and grok-style structured-extraction problems that underpin analyzers and ingest processors.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the mapping / analyzer / ingest / shard decision model against real graded inputs.

Lock in search-cluster muscle memory

Docs explain the settings. PipeCode drills explain the decision — when a field must be `keyword` not `text`, when analysis has to happen at index time, when an ingest pipeline beats a Spark job, when 30 shards is 29 too many, and when Elasticsearch is the wrong database entirely. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers who own the search cluster actually face.

Practice ETL problems →
Practice real-time analytics problems →