Cutting Elasticsearch log storage roughly in half without changing queries

#elasticsearch #logging #lucene #java

10x ships logs to Elasticsearch in a compact form: each format string kept once as a template, each event as the template's hash plus the values that change per line. On a public 200 MB sample that form measured 13.8 MB on disk against 28.6 MB raw, lossless: decoding returns every original line. Existing text queries keep working unchanged through L1ES, an Apache-2.0 plugin that rewrites them at the Lucene layer; one source tree builds it for every Elasticsearch 8.x and OpenSearch 2.x.

Most of a log index is the same format strings stored millions of times. 10x encodes logs in the forwarding pipeline, before they reach Elasticsearch: each format string ships once, as a template, and each event ships as the template's hash plus the values that changed. On a public 200 MB sample, that cut the on-disk Elasticsearch index from 28.6 MB to 13.8 MB, lossless.

L1ES is the plugin that makes existing text queries work unchanged over that form and puts the original line back before Kibana shows it. The plugin is Apache 2.0 at github.com/log-10x/elasticsearch-plugin; the 10x engine that writes the form is the paid product and is not in that repo. Reading the data back needs nothing paid: the templates sit in a plain Elasticsearch index, and standalone decoders reproduce every line from them.

A stored event is a template hash plus its values

One event from the OpenTelemetry demo's accounting service arrives in Elasticsearch like this:

{"stream":"stdout","log":"Accounting service started","docker":{"container_id":"80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7"},"kubernetes":{"container_name":"accounting","namespace_name":"default","pod_name":"accounting-6f7cb8cd4d-hnqb4","container_image":"ghcr.io/open-telemetry/demo:2.1.3-accounting",...,"pod_ip":"192.168.33.161","host":"ip-192-168-42-205.ec2.internal",...}}

The raw event is 923 bytes; the middle of the envelope is trimmed for width.

Little in that line is unique to the event. The JSON keys and the fixed message words form its template: the full JSON skeleton with $ where values go, stored once in a separate template index. The data index stores the event as the template's hash plus its values:

~3;&4e;iuzQ,80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7,accounting,6f7cb8cd4d,hnqb4,ghcr,2,1,3,3c3e...3418,308010eb,c129,-4320,a5ca,1ad8becbedda,192,168,33,161,-192,-168,-42,-205,accounting,accounting,...

The same event in compact form is 349 bytes (trimmed): the leading ~ marks an encoded event, 3;&4e;iuzQ is the template hash, and everything after the first comma is the values, in template order.

The 10x engine derives the templates ahead of time from the source environment, so the hashes stay stable. Templates clustered from runtime data by tools like Drain drift over time.

Storing an event in two indices breaks Elasticsearch text search

To Elasticsearch the two indices are unrelated; a match query on the data index sees only hashes and values. Every word that came from a template's fixed text is missing there: a search for started returns nothing, and no mapping change fixes that.

Two obvious ways out fail. Decoding every event at ingest restores search but erases the disk saving. A rewriting proxy in front of the cluster can edit query JSON, but it works at the REST API and cannot reach into a Lucene segment. So L1ES is a native plugin inside the Elasticsearch JVM, at the Lucene layer, where it can both rewrite the query and check a document inside its segment.

The plugin classifies templates first and decodes documents only when it must

When a search arrives, L1esQueryRewriter walks the query tree and swaps each text query node (match, match_phrase, multi_match) for an L1ES-aware version; compound nodes (bool, constant_score, boosting, dis_max) recurse, and everything else passes through. Each rewritten node then sorts the candidate templates into three states. A query token is satisfied by a word in the template's fixed text or by a value in a $ slot. A template is MUST_MATCH when every query token is in its fixed text, CANNOT_MATCH when it has no $ slots and a token is missing, and MIGHT_MATCH when a missing token could be a value.

Only MIGHT_MATCH documents pay a per-document check: L1esScorerWrapper wraps the inner Lucene scorer in a TwoPhaseIterator whose exact check decodes the candidate and tests the query against the original line. After scoring, L1esFetchSubPhase rebuilds the original line into _source, so Kibana Discover shows the log as printed.

The template-hash field is optional, so existing indices need no reindex

L1esMatchQuery first checks whether the data index carries a keyword field, l1x_tid, holding each document's template hash. With it, classification runs against an in-memory snapshot of the templates, with no Elasticsearch round-trip. Without it, the plugin queries the template index directly, a slower but correct path.

I could have made the field mandatory and deleted the slower path. I did not, because a plugin that demands a mapping change before it does anything is blocked until a reindex is scheduled. Any index that already holds encoded data can install the plugin and search today; adding l1x_tid later enables the fast path on the next search, with no restart. The price is a second code path that has to stay correct, and I still think the trade was right.

What the plugin does not handle

Single-token queries are answered at the template level, so searching error returns every event of any template whose fixed text contains error, even when the event is about something else. The extra hits are over-matches; no content is altered and no matching event is dropped. Multi-token AND and phrase queries stay precise, because the per-document check verifies the decoded line.

Query types the rewriter does not cover (term, wildcard, regexp, and query_string, Kibana's Lucene syntax mode) see the encoded text directly: they match values but miss words that come from a template's fixed text. Anything else that reads the field, an aggregation or a runtime field, sees the encoded form too. KQL compiles to the three rewritten query types, a field-less search becoming multi_match, so most Kibana usage takes the rewritten path. If the per-document check throws, the plugin keeps the document.

Relevance scores change: a document matched through its template scores as a filter hit rather than BM25 over the original line, and log views sort by time. Highlighting is not rebuilt: it runs against the stored encoded text, so matched words are not marked in results.

Decode work moves to query time: queries over the compact form cost more CPU than the same queries over raw text. That is query-time work spent to save disk; search does not get faster.

The measured result is roughly half the on-disk index

I indexed the 200 MB sample into Elasticsearch 8.17.0 twice, raw (one message text field per event) and compact, both with best_compression, one segment, and no replicas. Raw here means the whole line stored as one text field; a mapping that parses fields changes both sides and is not measured here. The comparison holds one thing constant: each side keeps enough to reproduce every original line. The raw index measured 28.6 MB across 197,430 documents. The compact pair measured 13.2 MB of data plus 0.6 MB of templates, 13.8 MB total. It holds 159,454 documents because multi-line events merge into one during encoding; nothing is dropped, and decoding returns every original line. The template index is part of the data: cluster snapshots must include it, because events decode only against their templates.

Logsdb mode with synthetic _source on the same raw sample reclaimed under 4%, 28.7 MB to 27.7 MB. A block codec removes repetition only inside its window; storing the template once removes it from the data entirely.

The raw sample, the encoded form, and the templates are published in one release; indexing the files reproduces the numbers. Every size in this post is on-disk index size.

One source tree builds the plugin for every Elasticsearch 8.x and OpenSearch 2.x

Elasticsearch loads a plugin only when it was built for the exact running version; the descriptor pins it, and the rule is Elasticsearch's. Version coverage is therefore build-system work, and the repo treats it that way: one source tree compiles against any Elasticsearch 8.x or OpenSearch 2.x by setting one Gradle property, the zip is named for the version it targets, and the OpenSearch variant is a subproject of the same tree that behaves identically, registered as a search pipeline. 8.17.0 and 2.19.0 are the defaults the repo pins and tests.

Elasticsearch 9 and OpenSearch 3 moved to Lucene 10; that port is a small, known change (three files touch changed Lucene APIs) and has not shipped yet. The plugin requires Java 17 or later. To decode the compact form outside Elasticsearch, there are standalone Java and JavaScript decoders.