DEV Community

Eduardo Rojas
Eduardo Rojas

Posted on

The Most Complete LZ4 Implementation in Dart

#dart #compression #flutter #webassembly

When you need LZ4 compression in Dart, you have options. But not all LZ4 implementations are equal — the LZ4 specification is more than just block decompression. It includes frame format, legacy format, dictionaries, skippable frames, high-compression mode, and streaming. Many implementations cover only a subset.

This article walks through the full LZ4 feature surface and shows what a complete implementation looks like, with code examples from dart_lz4.

The LZ4 Feature Surface

The LZ4 ecosystem has several layers:

  1. Block format — the core compression algorithm. Compresses a single block of bytes.
  2. Frame format — the standard container format (magic 0x184D2204). Wraps blocks with headers, flags, checksums, and optional content size.
  3. Legacy frame format — the older format (magic 0x184C2102), produced by lz4 -l.
  4. Skippable frames — metadata containers that decoders skip but encoders can embed.
  5. LZ4HC — high-compression mode. Same decompression, better ratio at the cost of speed.
  6. Dictionaries — preset dictionaries for compressing small inputs with known patterns.
  7. Streaming — chunk-by-chunk encode/decode via StreamTransformer.
  8. xxHash32 — the checksum used by LZ4 frames.

A "complete" LZ4 implementation should handle all of these. Let's look at each.

Block Format

The basics — compress and decompress a buffer:

import 'package:dart_lz4/dart_lz4.dart';

final src = Uint8List.fromList('hello world'.codeUnits);
final compressed = lz4Compress(src);
final decoded = lz4Decompress(compressed, decompressedSize: src.length);
Enter fullscreen mode Exit fullscreen mode

Block decompression requires the original size — this is inherent to the LZ4 format, not a limitation of a particular implementation.

Zero-Copy Decompression

For performance-critical paths, you can decompress directly into a pre-allocated buffer:

final dst = Uint8List(src.length);
final bytesWritten = lz4DecompressInto(compressed, dst);

// Or at an offset within a shared buffer:
final offsetBytes = lz4DecompressInto(compressed, dst, dstOffset: 64);
Enter fullscreen mode Exit fullscreen mode

This avoids intermediate allocations — important in high-throughput pipelines.

LZ4HC: High-Compression Mode

LZ4HC uses a more thorough search strategy to find better matches. The decompressed output is identical — any LZ4 decoder can read LZ4HC output. The tradeoff is compression speed: LZ4HC is slower but produces smaller output.

final compressed = lz4Compress(
  src,
  level: Lz4CompressionLevel.hc,
  hcOptions: Lz4HcOptions(maxSearchDepth: 64),
);
Enter fullscreen mode Exit fullscreen mode

dart_lz4 supports HC levels 1–12, matching the C reference implementation's range. This matters because higher levels can meaningfully improve ratio on compressible data while remaining fast enough for interactive use.

Frame Format

The frame format is the standard LZ4 container. It wraps blocks with a descriptor that specifies flags, block size, and optional metadata:

final frame = lz4FrameEncode(src);
final decoded = lz4FrameDecode(frame);
Enter fullscreen mode Exit fullscreen mode

Frame Options

The frame descriptor supports several options:

final frame = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(
    blockSize: Lz4FrameBlockSize.k64KB,
    blockChecksum: true,
    contentChecksum: true,
    contentSize: src.length,
    compression: Lz4FrameCompression.fast,
    acceleration: 1,
  ),
);
Enter fullscreen mode Exit fullscreen mode

Each option has a purpose:

  • blockSize — maximum block size (64KB, 256KB, 1MB, or 4MB)
  • blockChecksum — per-block xxHash32 for corruption detection
  • contentChecksum — end-of-frame xxHash32 over all decompressed data
  • contentSize — 64-bit total decompressed size in the header
  • blockIndependence — independent blocks (default) or linked blocks with 64KB history

Dependent Blocks

Linked blocks can reference up to 64 KiB of history from prior blocks, improving ratio on data with patterns spanning block boundaries:

final frame = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(blockIndependence: false),
);
Enter fullscreen mode Exit fullscreen mode

Legacy Frame Format

The legacy format (0x184C2102) is what lz4 -l produces. It's simpler — no descriptor, no flags, no optional fields. Some systems still produce or consume this format:

final frame = lz4LegacyEncode(src);
final decoded = lz4FrameDecode(frame);
Enter fullscreen mode Exit fullscreen mode

A complete implementation should both encode and decode legacy frames, not just decode them.

Skippable Frames

Skippable frames let you embed custom metadata in an LZ4 stream. Decoders skip them; encoders can use them for application-specific data:

import 'dart:convert';

final metadata = Uint8List.fromList(utf8.encode('{"version": 1}'));
final skippable = lz4SkippableEncode(metadata, index: 0);

// Concatenate with a regular frame
final combined = Uint8List.fromList([...skippable, ...lz4FrameEncode(data)]);

// Decoders skip the metadata and decode only the payload
final decoded = lz4FrameDecode(combined);
Enter fullscreen mode Exit fullscreen mode

The index parameter (0–15) selects the magic number in the skippable range (0x184D2A50–0x184D2A5F).

Dictionary Support

Dictionaries let you compress small inputs more effectively by providing preset context. This is useful when compressing many small messages with shared patterns (e.g., log entries, API payloads):

// Encoding with a dictionary
final compressed = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(dictionary: myDictionary),
);

// Decoding with a dictionary resolver
final decoded = lz4FrameDecode(
  frameBytes,
  dictionaryResolver: (dictId) {
    if (dictId == 0x123456) return myDictionaryBytes;
    return null;
  },
);
Enter fullscreen mode Exit fullscreen mode

The dictionary resolver is a callback — the decoder calls it with the dictId from the frame header, and you return the matching dictionary bytes. This design supports multiple dictionaries without embedding them in the library.

Streaming

Streaming encode/decode processes data chunk by chunk via StreamTransformer:

// Streaming decode
final decodedChunks = byteChunksStream.transform(
  lz4FrameDecoder(maxOutputBytes: 128 * 1024 * 1024),
);

// Streaming encode
final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoder(),
);
Enter fullscreen mode Exit fullscreen mode

Streaming supports the same Lz4FrameOptions as the sync API:

final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoderWithOptions(
    options: Lz4FrameOptions(
      blockSize: Lz4FrameBlockSize.k64KB,
      blockIndependence: false,
    ),
  ),
);
Enter fullscreen mode Exit fullscreen mode

Buffer Pooling

For zero-allocation streaming, buffer pools reuse memory across operations:

final pool = SimpleLz4BufferPool(
  maxTotalBuffers: 64,
  maxBuffersPerBucket: 8,
);

final decodedStream = byteChunksStream.transform(
  lz4FrameDecoder(bufferPool: pool),
);
Enter fullscreen mode Exit fullscreen mode

The pool uses power-of-two slab bucketing (64B to 8MB) matching LZ4 block sizes. A SecureLz4BufferPool variant zeroes buffers on return for CWE-226 mitigation in security-sensitive contexts.

dart:convert Integration

The Lz4Codec wraps LZ4 frame encode/decode as a standard dart:convert Codec:

import 'dart:convert';

final codec = Lz4Codec();
final compressed = codec.encode(data);
final decoded = codec.decode(compressed);

// Fuse with other codecs:
final jsonLz4 = json.fuse(codec);
final payload = utf8.encode('{"hello":"world"}');
final compressedJson = jsonLz4.encode(payload);
final restored = jsonLz4.decode(compressedJson);
Enter fullscreen mode Exit fullscreen mode

This makes LZ4 composable with the entire Dart conversion ecosystem — utf8, json, base64, and any custom codec.

When to Use dart_lz4

dart_lz4 is a pure-Dart implementation with zero runtime dependencies. It works on all Dart platforms including Web and WASM. It does not use FFI — native LZ4 bindings via dart:ffi will be faster for large payloads on native platforms.

Choose dart_lz4 when:

  • You need LZ4 on Web or WASM (FFI-based libraries can't run there without native binaries)
  • You want zero native dependencies (simpler deployment, no platform-specific binaries)
  • You need features beyond basic block compression (dictionaries, legacy frames, skippable frames, streaming with buffer pools)
  • You want bounds-safe decoding with output limits on untrusted input
  • You value supply-chain hardening (OpenSSF Gold, fuzzing in CI, 100% test coverage)

Choose FFI-based libraries when:

  • You need maximum throughput on native platforms
  • You don't need Web/WASM support
  • You're compressing large payloads where the native speed advantage matters

These are complementary tools for different use cases, not competitors in the same lane.

Summary

Feature Support
Block encode/decode Yes
Zero-copy decompression Yes
LZ4HC levels 1–12 Yes
Frame format (all flags) Yes
Legacy frame format Encode + decode
Skippable frames Encode + decode
Dictionary encode/decode Yes
Streaming (StreamTransformer) Yes
Buffer pooling (simple + secure) Yes
dart:convert Codec Yes
xxHash32 Yes (VM + Web parity)
Web/WASM Yes
FFI acceleration No (pure Dart)

dart_lz4 is on pub.dev with a 160/160 score, OpenSSF Best Practices Gold badge, 361 tests, and frame fuzzing in CI. Source code and documentation at GitHub.

Top comments (0)