Your OpenSearch aggregation is lying to you: it returns 200 OK

Anagha Biju — Fri, 04 Sep 2026 03:54:28 +0000

A dashboard panel went blank in production. No error in the logs, no failed request, no alert. The panel just showed "no data", and the query had been fine for months.

The cause: someone renamed an index field from customer_country to country, updated the mapping, and missed one saved aggregation that still referenced customer_country. OpenSearch didn't complain. It returned an empty bucket list and a 200.

Once I went looking, I found this wasn't one weird edge case. It's the default behaviour of the whole aggregation framework, and there are several distinct ways to hit it. Some of them don't even return an empty panel. They return a plausible one, which is much worse.

Why aggregations fail quietly

Here's the thing that makes all of this click, and it took me embarrassingly long to internalise:

Aggregations don't validate. They count.

There is no step where the engine compares your aggs block against the mapping and objects to what it finds. A terms aggregation on country walks the doc values for country and buckets what's there. If nothing is there, the honest answer is zero buckets.

And now the key part: from inside that loop, a field that doesn't exist and a field that no document happens to have are indistinguishable. Both are "no values found". The second one is a completely legitimate result: an empty index, a filtered-out time range, a genuinely sparse optional field. If the engine threw an error on the first, it would have to throw on the second too, and half of all legitimate queries would start failing.

So it returns [] and a 200, and it is arguably right to. The problem is that "correct" and "useful to you at 2am" aren't the same thing.

Everything below follows from that one design decision.

Lie #1: the field that isn't there

{ "aggs": { "by_country": { "terms": { "field": "custmer_country" } } } }

Typo, renamed field, field that only exists in the other index behind your alias. All of them produce the same silent empty result, and you find out when a human notices a blank chart, which might be next week.

There's a nastier variant that catches people who did everything right. When OpenSearch dynamically maps a string, it creates a text field with a keyword sub-field, and that sub-field gets ignore_above: 256 by default. Values longer than 256 characters are not indexed into .keyword at all.

So terms on url.keyword will happily bucket your short URLs and silently drop every long one. No error. The field exists, the mapping is correct, the aggregation runs, and the answer is quietly incomplete in a way that correlates with exactly the values you were probably interested in.

Lie #2: the field that's the wrong kind

{ "aggs": { "by_city": { "terms": { "field": "city" } } } }

If city is a text field, this does not aggregate on city values.

text fields go through an analyzer on the way in: lowercased, tokenized, sometimes stemmed. "New York" isn't stored as New York, it's stored as two terms, new and york. An aggregation walks the terms in the index, so the buckets you get back are tokens rather than cities: new (1 doc) and york (1 doc).

keyword fields skip the analyzer entirely. One document, one term, exactly the bytes you sent. That's why the fix is almost always a sub-field you forgot:

{ "aggs": { "by_city": { "terms": { "field": "city.keyword" } } } }

There's a second layer here worth knowing, because it explains an error message that otherwise looks arbitrary. Aggregations need to go from document to value, but an inverted index goes the other way, from term to documents. keyword fields solve this with doc values: a columnar, on-disk structure written at index time, memory-mapped and cheap to scan.

text fields don't get doc values, because building them for every token of every analyzed field would be enormous. The only way to aggregate a text field is fielddata, which builds that same structure on the fly and holds it on the JVM heap. It has OOM'd real clusters, which is why it ships disabled and you get this instead:

Fielddata is disabled on text fields by default. Set fielddata=true on [city]
in order to load fielddata in memory by uninverting the inverted index.

That message is the good outcome. If someone before you turned fielddata on to make an error go away, you don't get the message. You get a chart of plausible-looking word fragments and no indication that anything is wrong.

Lie #3: the interval that isn't the interval you meant

{ "aggs": { "over_time": { "date_histogram": { "field": "created_at" } } } }

This one 400s. date_histogram requires calendar_interval or fixed_interval; the old interval parameter was removed in Elasticsearch 7 / OpenSearch 1. If that query lives in a .json template only exercised when one particular dashboard loads, production is where you'll meet it.

But the 400 is the lucky outcome, because the reason interval was removed is the real lesson. It accepted both kinds of value and guessed which you meant, and the two are genuinely different things:

calendar_interval follows the calendar. 1d is a real day, which is 23 or 25 hours across a DST boundary. 1M is a real month, so 28, 30, or 31 days. It only accepts a single unit, which makes 1M valid and 2M invalid.
fixed_interval is exact multiples of a fixed duration. 30d is always exactly 720 hours, regardless of what the calendar or DST thinks.

So "calendar_interval": "1M" and "fixed_interval": "30d" are not two ways of saying the same thing. Month-over-month revenue bucketed on 30d drifts against the calendar and slowly stops matching the finance team's numbers. And that query returns 200 every single time.

Rule of thumb: humans read calendar intervals, machines read fixed ones. If a person is going to look at the bucket label, you want calendar_interval.

Lie #4: the top 10 that isn't the top 10

This is the one I'd most want you to take away, because it produces confidently wrong numbers from a completely valid query.

{ "aggs": { "top_products": { "terms": { "field": "product_id", "size": 10 } } } }

Nothing wrong with it. On a multi-shard index it can still be wrong.

A terms aggregation is distributed. The coordinating node asks each shard for its own top terms, then merges what comes back. It never sees the full term list. That's the whole point: materialising every term from every shard is exactly the cost you're trying to avoid.

Now picture a product that ranks #11 on all five of your shards. It's in nobody's top 10, so no shard mentions it, so it doesn't appear in your results at all, even though summed across all five it might outrank everything that did come back. And a product that made the cut on three shards but not the other two returns a doc_count that's simply missing two shards' worth of documents. The number looks fine. It's just too low.

OpenSearch mitigates this by fetching more than you asked for from each shard. That's shard_size, which defaults to size * 1.5 + 10. And it's honest about the remaining risk, in two response fields most people never look at:

{
  "top_products": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 84213,
    "buckets": [ "..." ]
  }
}

sum_other_doc_count: documents that fell outside your top size. If it's large relative to your buckets, your "top 10" is a thin slice of reality. It's also the honest source for an "everything else" segment in a pie chart.
doc_count_error_upper_bound: the worst case amount by which a returned doc_count might be understated. Zero is your green light. Anything above zero means at least one number on that dashboard could be too low. Set show_term_doc_count_error: true for the per-bucket breakdown.

Go and look at these two fields on your own dashboard queries. If you've never checked, the odds are good that at least one is nonzero.

Raising shard_size is much cheaper than raising size, since it costs transport and coordinating-node memory rather than deeper per-shard work, so it's the first knob to reach for. And if you genuinely need every term, terms is the wrong aggregation; composite is the one that paginates.

The honest one

Credit where it's due, one of these fails loudly and immediately:

{
  "aggs": {
    "revenue": {
      "sum": { "field": "amount" },
      "aggs": { "by_channel": { "terms": { "field": "channel" } } }
    }
  }
}

Instant 400. And once you have the mental model it's obvious forever: a bucket aggregation hands its children a set of documents. A metric aggregation hands you a number. You can't nest anything under a number.

Which means revenue-per-channel requires inverting the nesting: bucket becomes the parent, metric becomes the child.

{
  "aggs": {
    "by_channel": {
      "terms": { "field": "channel" },
      "aggs": { "revenue": { "sum": { "field": "amount" } } }
    }
  }
}

If you remember one sentence from this post, make it the one about documents and numbers. It answers most "why won't this nest" questions before you have to look anything up.

I got tired of remembering all of this

Every check above is decidable from the query text and the mapping. You don't need a running cluster to know that sum can't have children, that date_histogram needs an interval, or that city is a text field. So I wrote opensearch-agg-inspector to do it statically.

npx opensearch-agg-inspector dashboard.json --mapping mapping.json

✖ dashboard.json:3:5  error  invalid-date-histogram
  date_histogram is missing "calendar_interval" or "fixed_interval". This will throw a runtime error in OpenSearch 1.x+ and Elasticsearch 7.x+.
  → Add calendar_interval (e.g. "1d", "1M") or fixed_interval (e.g. "1h", "30m").

✖ dashboard.json:4:5  error  metric-sub-aggregation
  Aggregation at "aggs.revenue" is a metric aggregation ("sum") but has sub-aggregations. The cluster will reject this. Only bucket aggregations can have children.

⚠ dashboard.json:6:17  warning  unknown-field
  Field "chanel" does not exist in the provided mapping. The aggregation will silently return no data.
  → Check the field name spelling and confirm it exists in the index mapping.

2 errors, 1 warning across 3 aggregations

Eight rules today. Four are the lies above; the rest are performance footguns like a terms size in the thousands with no matching shard_size. Output is file:line:col, clickable in your editor, and it exits non-zero, so the useful home for it is a lint:queries script in CI sitting next to your saved dashboard JSON. Export the mapping once and commit it alongside:

curl -s localhost:9200/orders/_mapping | jq '.orders.mappings.properties' > mapping.json

There's also an inspect() function if you assemble aggregations at runtime from user input and would rather reject a broken one than return silent bad data.

Being upfront about what it can't do: it reads the aggs block only, not query. Your mapping is its source of truth for "does this field exist", so a partial mapping means false positives on unknown-field. Downgrade or disable that rule if that's your situation. It targets OpenSearch 1.x+ / Elasticsearch 7.x+ semantics and needs Node 20. And it can't do much about lie #4 beyond nagging you about shard_size, because no static tool can tell you your term distribution is skewed. That one you have to go and read in the response.

Zero runtime dependencies, ESM + CJS + TypeScript types, MIT. https://github.com/ab1109/opensearch-agg-inspector

It's v0.1.0, so the rule set is deliberately small. If there's a quiet aggregation bug that's bitten you and isn't in there, I'd like to hear about it in an issue.

DEV Community: Anagha Biju