DEV Community: Log10x

Log compaction that stays searchable, measured on 15 public datasets

Tal Weiss — Thu, 10 Sep 2026 16:47:47 +0000

A log platform (Splunk, for example) meters ingest by the byte, and most bytes in a log line are repetitive text. Part of it is what the code prints: the words of a format string, which are identical every time that statement runs. The rest is added around the message: the level, the service and pod names, the fields a logging framework stamps on every line (in Java, the MDC).

10x runs as a sidecar to the log forwarder, stores each line's repeated text once as a template, and ships each line as that template's hash plus the line's values.

Because each compact line is still a row the log platform can query, the reduction stays searchable in place instead of sitting in a gzip blob. The read path for Splunk and Elasticsearch is built in the sibling posts, and on ClickHouse a Merge table reads offloaded lines from the bucket beside the table.

Here is one line from a Fluentd container, as the forwarder read it:

2025-10-02 00:17:22 +0000 [info]: #0 starting fluentd worker pid=18 ppid=6 worker=0

The engine stores the repeated part of that line once, as a template. Each $ marks a slot where a value goes:

{
  "templateHash": "9aZXwGwVG/",
  "template": "$(yyyy-MM-dd HH:mm:ss) +$ [info]: #0 starting fluentd worker pid=$ ppid=$ worker=$"
}

The line itself ships as a single row. After the leading ~ comes the template's hash, the same 9aZXwGwVG/ as in the dictionary entry above, and then that line's values in order: the timestamp as epoch milliseconds, the timezone digits, and the three numbers.

~9aZXwGwVG/,1759364242000,0000,18,6,0

The reverse step, expand, splices the values back into the template's slots and returns the original line, as the round-trip check further down measures.

This post measures the reduction on fifteen public datasets, run through the engine in its stock configuration: a 215 MB stream of Kubernetes container output, captured through Fluentd from a cluster running the OpenTelemetry demo, plus 14 of the 16 public LogHub datasets at full size.

The Docker image these runs used, log10x/pipeline-10x, is public, as are all fifteen datasets, so every number below reruns with the one command at the end of the post.

Inputs and method

Each run is one pass of the engine over one dataset: the original file goes in, and two files come out, the encoded events and the template dictionary. The LogHub datasets are the full files from the project's Zenodo record rather than its 2,000-line benchmark excerpts. A full file reduces a few points more than the excerpt, because the one-time dictionary is spread over more events.

The two LogHub sets left out, Thunderbird (29.6 GB) and Windows (26 GB), are the collection's largest and get a separate run, so the weighted figure below covers the 14 included sets rather than LogHub as a whole.

The engine tokenizes each line and matches the tokens against a symbol library, which is the vocabulary of recurring tokens it recognizes. A library can be compiled from a specific codebase, and none of these runs did that. Every dataset went through the default symbol library that ships in the image, which is what a first install runs.

Reduction means the bytes of the compact form (the encoded events plus the template dictionary needed to expand them) against the bytes of the original file, with no gzip or ZSTD on either side.

10x compacts logs before the platform meters them: a sidecar to the log forwarder stores the repeated text once as a template and ships each line as the template's hash plus values, and expand returns the original line. Compact lines stay queryable in place rather than sitting in a gzip file. Measured on fifteen public datasets, a 215 MB Kubernetes stream reduced 64%, and 14 full LogHub datasets 44.7% overall, from 4.7% to 58.4%.

What the runs measured

All fifteen datasets are below, sorted by reduction. Every row except OTel-demo is a LogHub set.

Dataset	Input	Reduction	Templates	Events
OTel-demo	215 MB	64%	3,028	157k
Hadoop	48.6 MB	58.4%	888	187k
Spark	2.94 GB	57.5%	3,377	33M
Apache	5.1 MB	55.3%	371	52k
OpenSSH	73.4 MB	49.8%	3,032	655k
Proxifier	2.5 MB	46.7%	1,854	21k
Zookeeper	10.4 MB	45.6%	157	74k
Linux	2.3 MB	44.5%	1,266	26k
HPC	33.6 MB	38.9%	2,045	330k
BGL	743 MB	35.1%	136,461	4.7M
Mac	16.9 MB	32.5%	2,466	107k
OpenStack	61.4 MB	30.6%	8,133	208k
HDFS	1.58 GB	29.1%	279	11.2M
Android	192 MB	20%	29,400	1.6M
HealthApp	23.5 MB	4.7%	1,341	205k

The Events column counts events after multi-line grouping, which the stock configuration uses to join runs of consecutive related raw lines into one event. That is how the 215 MB stream's 197,430 raw lines count as 157,228 events. No line is dropped, and expand returns the raw lines.

The 215 MB stream is the widest public data offers, 36 Kubernetes containers with infrastructure included, and it compacts 64%. The 14 LogHub datasets hold 52,276,243 events and 5.73 GB of input. Weighted by input bytes, they reduce 44.7%.

The spread tracks how much of each line is repeated text

The reduction tracks how much of each line is repeated text and how many lines share each template rather than which application wrote it.

HealthApp (4.7%) and Android (20%) are the floor. Both are mobile logs whose component names and tags ship as values, beside the timestamps and IDs, so little of each line is repeated text. HealthApp's most common template, $(yyyyMMdd-H:m:s:SSS)|Step_$|$|$ $=$, keeps only |Step_ and the separators.

The middle of the table, 29% to 58%, sorts by the same variable. HDFS and OpenStack sit at the low end because their lines are mostly values: block IDs, PIDs, and byte counts, with little fixed text between them, and the values ship on every event regardless. Hadoop, Spark, and Apache sit at the high end because they write more repeated message text around their values.

The same shape holds wherever a line is a row of values with thin separators. Access logs and VPC flow logs are in that class, so expect them near the low end.

Compaction and compression

A general-purpose compressor can beat these numbers on log text. Compact is built for a different property: the reduced stream stays searchable, line by line, inside the platform.

A compact event sits in the platform as a row, so a query filters on what is stored in the clear (the container, the template hash, the raw encoded text), and expand runs last, on the rows that survive the filter. Gzip applied to the same stream leaves a blob that answers no search until the whole file is unzipped. That read path is built per platform, an Elasticsearch plugin at the Lucene layer or a Splunk app at search time.

The other difference is position: compact runs in the sidecar, so the reduction takes effect before the platform meters ingest and before the indexer sizes its work. An at-rest codec runs after all of that, once the platform has already counted and indexed the original bytes. Running gzip on the transport changes nothing either, because platforms meter the log data itself and the transfer encoding is invisible to the bill.

That reduction reaches the bill where a platform prices by log volume. Splunk licenses by the volume ingested per day, so the reduction comes off the metered quantity directly.

A self-managed store pays for the bytes it holds, but it also compresses what it holds, so the wire reduction does not carry to disk as-is. The disk number is whatever an in-place measurement says, and inside Elasticsearch that measurement came out at roughly half the on-disk size for typical log bodies. A platform that prices by event count works on a different axis, and compact is not the tool there.

Round-trip check

A reduction is only worth trusting if expand returns exactly what came in, so here is that check on two of the datasets: encode a file, concatenate the template dictionary and the encoded stream into one compact file, decode that file, and compare the result to the original. On both, expand returns the log text unchanged, with two exceptions: line endings come back as LF, and a file that does not end in a newline gets one.

The 215 MB OpenTelemetry stream returns all 197,430 lines byte-identical. The full Apache LogHub set, run the same way, returns all 56,482 lines identical.

The decode pass reproduces the original from the two files the encode pass wrote, without a key or any extra state. The decode command below reads tenx-decode.config.yaml, published in the benchmark harness at github.com/log-10x/benchmarks:

cat <out>/templates.json <out>/encoded.log > <out>/compact.log
docker run --rm \
  -e INPUT_FILE=/out/compact.log \
  -e OUTPUT_DIR=/out \
  -v <out>:/out \
  -v <asset>/tenx-decode.config.yaml:/cfg/tenx-decode.config.yaml:ro \
  log10x/pipeline-10x:1.1.133 @/cfg/tenx-decode.config.yaml
# decoded.log matches the original, line for line

Reproducing a row

The datasets are public: the 215 MB stream is a capture of the OpenTelemetry demo's Kubernetes output, published in the log10x public-assets bucket, and the LogHub full datasets are on Zenodo. Each run is one container invocation. The encode command reads tenx-encode.config.yaml, published beside the decode config in the same benchmark harness:

docker run --rm \
  -e INPUT_FILE=/in/events.log \
  -e OUTPUT_DIR=/out \
  -v <out>:/out \
  -v <input>:/in/events.log:ro \
  -v <asset>/tenx-encode.config.yaml:/cfg/tenx-encode.config.yaml:ro \
  log10x/pipeline-10x:1.1.133 @/cfg/tenx-encode.config.yaml

Here <input> is the dataset file, <out> is a writable output directory, and <asset> is the directory holding the config. The pass writes the encoded events to encoded.log and the template dictionary to templates.json. Sum the two file sizes against the input and the table above reproduces.

Pull the image, run the command on a day of output from one cluster, and read the reduction from the two files it writes.

I work at log10x, which builds the engine measured here. The datasets and the image are public and linked above so every row can be checked independently. This post was drafted with AI assistance and edited by me.

Cutting Elasticsearch log storage roughly in half without changing queries

Dor Levi — Mon, 20 Jul 2026 13:31:34 +0000

Most of a log index is the same format strings stored millions of times. 10x encodes logs in the forwarding pipeline, before they reach Elasticsearch, shipping each format string once as a template and each event as the template's hash plus the values that changed. On a public 200 MB sample, that cut the on-disk Elasticsearch index from 28.6 MB to 13.8 MB, lossless.

L1ES is the plugin that makes existing text queries work unchanged over that form and puts the original line back before Kibana shows it. The plugin is MIT licensed at github.com/log-10x/elasticsearch-plugin. The 10x engine that writes the form is not in that repo; it runs on the built-in Free tier on up to 10 nodes, with no licence token. The templates sit in a plain Elasticsearch index, though, and standalone decoders reproduce every line from them.

The decoders that read the compact form back are MIT licensed: com.log10x:log10x-decoder-core on Maven Central, with a CLI beside it, the JavaScript decoder on GitHub, and the format specification in log10x-format. Reading the data back needs no engine and no licence.

How an event is stored

One event from the OpenTelemetry demo's accounting service arrives in Elasticsearch like this:

{"stream":"stdout","log":"Accounting service started","docker":{"container_id":"80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7"},"kubernetes":{"container_name":"accounting","namespace_name":"default","pod_name":"accounting-6f7cb8cd4d-hnqb4","container_image":"ghcr.io/open-telemetry/demo:2.1.3-accounting",...,"pod_ip":"192.168.33.161","host":"ip-192-168-42-205.ec2.internal",...}}

The raw event is 923 bytes, the envelope's middle trimmed for width.

Little in that line is unique to the event. The JSON keys and the fixed message words form its template, the full JSON skeleton with $ where values go, stored once in a separate template index. The data index stores the event as the template's hash plus its values:

~3;&4e;iuzQ,80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7,accounting,6f7cb8cd4d,hnqb4,ghcr,2,1,3,3c3e...3418,308010eb,c129,-4320,a5ca,1ad8becbedda,192,168,33,161,-192,-168,-42,-205,accounting,accounting,...

The same event in compact form is 349 bytes, trimmed the same way. The leading ~ marks an encoded event, 3;&4e;iuzQ is the template hash, and everything after the first comma is the values, in template order.

The 10x engine derives the templates ahead of time from the source environment, so the hashes hold on every node sharing the pinned library and engine version, for the lines that library covers. A template learned from runtime data by tools like Drain is a running summary of what one instance saw, so the same line can carry different templates on different nodes. A hash of the template inherits that.

Why text search breaks

To Elasticsearch the two indices are unrelated, so a match query on the data index sees only hashes and values. Every word that came from a template's fixed text is missing there. A search for started returns nothing, and no mapping change fixes that.

Decoding every event at ingest restores search but erases the disk saving. A rewriting proxy in front of the cluster can edit query JSON, though it cannot reach into a Lucene segment from the REST API. So L1ES is a native plugin inside the Elasticsearch JVM, at the Lucene layer, where it can both rewrite the query and check a document inside its segment.

How the plugin answers a query

When a search arrives, L1esQueryRewriter\ walks the query tree and swaps each text query node (match, match_phrase, multi_match) for an L1ES-aware version, while compound nodes (bool, constant_score, boosting, dis_max) recurse and everything else passes through. Each rewritten node then sorts the candidate templates into three states. A query token is satisfied by a word in the template's fixed text or by a value in a $ slot. MUST_MATCH means the fixed text already carries every word the query asked for. CANNOT_MATCH means one of those words is missing from a template that has no $ slots to hold it. MIGHT_MATCH is the rest, where the missing word could turn up in a value.

Only MIGHT_MATCH documents pay a per-document check. L1esScorerWrapper wraps the inner Lucene scorer in a TwoPhaseIterator whose exact check decodes the candidate and tests the query against the original line. After scoring, L1esFetchSubPhase rebuilds the original line into _source, so Kibana Discover shows the log as printed.

The optional template-hash field

L1esMatchQuery\ first checks whether the data index carries a keyword field, l1x_tid, holding each document's template hash. With it, classification runs against an in-memory snapshot of the templates, with no Elasticsearch round-trip. Without it, the plugin queries the template index directly, which is slower but correct.

I could have made the field mandatory and deleted the slower path. I did not, because a plugin that demands a mapping change before it does anything is blocked until a reindex is scheduled. Any index that already holds encoded data can install the plugin and search today. Adding l1x_tid later enables the fast path on the next search, with no restart. The price is a second code path that has to stay correct.

What the plugin does not handle

Single-token queries are answered at the template level, so searching error returns every event of any template whose fixed text carries that word, even when the event is about something else. The extra hits are over-matches, adding events rather than altering or dropping them. Multi-token AND and phrase queries stay precise, because the per-document check verifies the decoded line.

Query types the rewriter does not cover (term, wildcard, regexp, and query_string, Kibana's Lucene syntax mode) see the encoded text directly, matching values but missing words that come from a template's fixed text, and so does anything else that reads the field, such as an aggregation or a runtime field. KQL compiles to the three rewritten query types, a field-less search becoming multi_match, so most Kibana usage takes the rewritten path. If the per-document check throws, the plugin keeps the document.

Relevance scores change, because a document matched through its template scores as a filter hit rather than BM25 over the original line. Log views sort by time, though. Highlighting runs against the stored encoded text without being rebuilt, so matched words are not marked in results. Decode work moves to query time as well, which costs more CPU than the same queries over raw text.

What the index measured

I indexed the 200 MB sample into Elasticsearch 8.17.0 twice, raw (one message text field per event) and compact, both with best_compression, one segment, and no replicas. A mapping that parses fields changes both sides and is not measured here. Each side keeps enough to reproduce every original line.

The raw index measured 28.6 MB across 197,430 documents. The compact pair measured 13.2 MB of data plus 0.6 MB of templates, 13.8 MB total, across 159,454 documents, because multi-line events merge into one during encoding. Decoding returned all 197,430 original lines byte-identical when the sample was round-tripped. The boundary sits one step earlier, at the file read, where bytes that are not valid UTF-8 are substituted before the engine tokenizes, the same substitution any UTF-8 log reader makes. The template index is part of the data. Cluster snapshots must include it, because events decode only against their templates.

On 8.17.0, logsdb mode with synthetic _source on the same raw sample reclaimed under 4%, 28.7 MB to 27.7 MB. A block codec strips repetition only inside its window, while storing the template once takes it out of the data entirely. On Elasticsearch 9.5.5, with the record parsed into fields, logsdb with pattern_text sorted on its template id held the same sample in 34.7% of the raw index's store, against 53.3% for the compact form on logsdb.

The raw sample, the encoded form, and the templates are published in one release. Indexing the files reproduces the numbers. Every size in this post is on-disk index size.

Building for one version at a time

Elasticsearch loads a plugin only when it was built for the exact running version, and the descriptor pins it. Version coverage is therefore build-system work. One source tree targets an Elasticsearch or OpenSearch version through one Gradle property, the zip is named for the version it targets, and the OpenSearch variant is a subproject of the same tree that behaves identically, registered as a search pipeline. 8.17.0 and 2.19.0 are the versions the repo pins, and the latest release carries a zip for each.

The plugin requires Java 17 or later, and standalone Java and JavaScript decoders read the compact form outside Elasticsearch. On Elasticsearch 9.3 and later, Elastic's pattern_text field type stores message templates natively.