DEV Community

Zayd Mulani
Zayd Mulani

Posted on

I built a log compressor you can grep without decompressing

#rust #cli #opensource #devops

Logs pile up. You compress them to keep them, and then the one time you need an answer, you sit there while zgrep inflates gigabytes. The other option is a search cluster that indexes everything and ends up bigger than the logs.

I wanted something in between: archive-sized on disk, searchable in place, nothing to run. So I built logjar, a Rust library and single binary.

$ logjar pack -o hdfs.ljar HDFS.log        # 1.5 GiB -> 52 MiB
$ logjar grep -c "blk_-1608999687919862906 " hdfs.ljar
268                                        # 0.9s
Enter fullscreen mode Exit fullscreen mode

The same search with gzip -dc | grep takes 7.9 seconds.

The idea: most of a log line was written by printf

Look at a line from Hadoop's HDFS log:

081109 203615 148 INFO dfs.DataNode$PacketResponder: Received block blk_-1608999687919862906 of size 91178 from /10.250.10.6
Enter fullscreen mode Exit fullscreen mode

Most of that is fixed text from one print statement. Only a few values change. logjar splits each line into three parts:

timestamp:  081109 203615
template:   <int> INFO dfs.DataNode$PacketResponder: Received block <var> of size <int> from /<var>
variables:  148, blk_-1608999687919862906, 91178, 10.250.10.6
Enter fullscreen mode Exit fullscreen mode

HDFS has 11.2 million lines but only 65 distinct templates. Each template is stored once. The variables go into columns grouped by template and slot, so all the block sizes sit together, then all the IP addresses. Values in one column look alike, and that's what zstd compresses best. Timestamps become deltas.

This idea isn't mine. It comes from two research systems, CLP (OSDI '21) and LogGrep (EuroSys '23). logjar is an independent implementation with its own file format and search method.

Compression

Measured on the LogHub datasets. Every row is checked to unpack byte for byte.

dataset xz -9 zstd -19 logjar
Zookeeper 29.6x 39.0x 165.3x
SSH 20.3x 24.6x 95.5x
Hadoop 38.1x 42.8x 93.5x
Apache 28.4x 28.8x 57.8x
HDFS (1.5 GiB) 16.5x 16.1x 28.8x
BGL 22.9x 21.7x 25.6x
OpenStack 16.1x 16.7x 20.4x

BGL is the weak one. Every line has a second timestamp with microseconds in the middle of the line, and logjar only parses timestamps at the start, so each of those ends up as a one-off dictionary entry.

Search without decompressing

This is the part I like most. Before decoding a single line, logjar proves which templates can't match.

The query compiles to a small automaton. logjar then runs that automaton over each template symbolically:

  • literal text is just its bytes
  • an <int> slot means "one or more characters from [-0-9]"
  • a <var> slot means "any of this segment's actual dictionary values"

If no accepting state is reachable, then no line using that template can match, so those lines are skipped. If every template in a segment is skipped, the segment's big data blocks are never decompressed at all.

For the block-id search above, 45 of 48 segments are skipped this way.

The check only ever over-approximates, so it can't hide a match. I test that directly on random templates, and end to end by comparing every search against a naive scan, using random globs, random regexes, time windows and appended data.

It works for regexes too, and timestamps are indexed:

logjar grep -E '(WARN|ERROR) .*timeout' app.ljar
logjar grep --after 2026-11-01 --before 2026-11-02T06:00 "disk full" app.ljar
Enter fullscreen mode Exit fullscreen mode

With a one-hour time window, the HDFS search takes 0.04 seconds, because the footer index rules out 46 of 48 segments before anything is read.

Where it loses

Plain grep on the uncompressed file, already cached in memory, is faster for simple searches: 0.2s against logjar's 0.4-0.9s. It's a memchr over bytes already in RAM, and that will always win. That file also takes 28x the disk space. Against anything compressed, logjar is 10-20x faster, and the more selective the query, the bigger the gap.

The bugs the tests found

Two bugs surfaced before release, and both were found by comparing logjar against something I trust:

  1. Case-insensitive negated classes. With -i, [^a] still matched A, because I folded case after negating the class. A test that checks my regex engine against the regex crate on 120,000 random pattern/input pairs caught it on the first run.
  2. $ on Windows logs. On CRLF files, [0-9]$ matched nothing. Archives keep the \r (they're lossless), so the line never ended in a digit. Comparing match counts with GNU grep on a real Zookeeper log caught it. $ now matches before the \r, and the pruning logic mirrors that so it stays sound.

Neither would have shown up in hand-written tests.

Try it 

cargo install logjar
Enter fullscreen mode Exit fullscreen mode

Prebuilt binaries for Windows, Linux (x64 and arm64) and macOS are on the releases page. The file format is documented in docs/FORMAT.md, and the pruning design is in docs/DESIGN.md.

It also works as a library:

let archive = Archive::open("app.ljar")?;
let q = Query::regex(r"after \d+ retries")?;
archive.search(&q, &[], |m| {
    println!("{}", String::from_utf8_lossy(&m.line));
    true
})?;
Enter fullscreen mode Exit fullscreen mode

Known limits are listed in the README: no shared dictionary across segments, no schema-aware handling of JSON logs, and no index on variable values yet. If you have a log format it handles badly, I'd like to see it.

Top comments (0)