DEV Community: Eduardo Rojas

The Most Complete LZ4 Implementation in Dart

Eduardo Rojas — Sun, 13 Sep 2026 18:33:49 +0000

When you need LZ4 compression in Dart, you have options. But not all LZ4 implementations are equal — the LZ4 specification is more than just block decompression. It includes frame format, legacy format, dictionaries, skippable frames, high-compression mode, and streaming. Many implementations cover only a subset.

This article walks through the full LZ4 feature surface and shows what a complete implementation looks like, with code examples from dart_lz4.

The LZ4 Feature Surface

The LZ4 ecosystem has several layers:

Block format — the core compression algorithm. Compresses a single block of bytes.
Frame format — the standard container format (magic 0x184D2204). Wraps blocks with headers, flags, checksums, and optional content size.
Legacy frame format — the older format (magic 0x184C2102), produced by lz4 -l.
Skippable frames — metadata containers that decoders skip but encoders can embed.
LZ4HC — high-compression mode. Same decompression, better ratio at the cost of speed.
Dictionaries — preset dictionaries for compressing small inputs with known patterns.
Streaming — chunk-by-chunk encode/decode via StreamTransformer.
xxHash32 — the checksum used by LZ4 frames.

A "complete" LZ4 implementation should handle all of these. Let's look at each.

Block Format

The basics — compress and decompress a buffer:

import 'package:dart_lz4/dart_lz4.dart';

final src = Uint8List.fromList('hello world'.codeUnits);
final compressed = lz4Compress(src);
final decoded = lz4Decompress(compressed, decompressedSize: src.length);

Block decompression requires the original size — this is inherent to the LZ4 format, not a limitation of a particular implementation.

Zero-Copy Decompression

For performance-critical paths, you can decompress directly into a pre-allocated buffer:

final dst = Uint8List(src.length);
final bytesWritten = lz4DecompressInto(compressed, dst);

// Or at an offset within a shared buffer:
final offsetBytes = lz4DecompressInto(compressed, dst, dstOffset: 64);

This avoids intermediate allocations — important in high-throughput pipelines.

LZ4HC: High-Compression Mode

LZ4HC uses a more thorough search strategy to find better matches. The decompressed output is identical — any LZ4 decoder can read LZ4HC output. The tradeoff is compression speed: LZ4HC is slower but produces smaller output.

final compressed = lz4Compress(
  src,
  level: Lz4CompressionLevel.hc,
  hcOptions: Lz4HcOptions(maxSearchDepth: 64),
);

dart_lz4 supports HC levels 1–12, matching the C reference implementation's range. This matters because higher levels can meaningfully improve ratio on compressible data while remaining fast enough for interactive use.

Frame Format

The frame format is the standard LZ4 container. It wraps blocks with a descriptor that specifies flags, block size, and optional metadata:

final frame = lz4FrameEncode(src);
final decoded = lz4FrameDecode(frame);

Frame Options

The frame descriptor supports several options:

final frame = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(
    blockSize: Lz4FrameBlockSize.k64KB,
    blockChecksum: true,
    contentChecksum: true,
    contentSize: src.length,
    compression: Lz4FrameCompression.fast,
    acceleration: 1,
  ),
);

Each option has a purpose:

blockSize — maximum block size (64KB, 256KB, 1MB, or 4MB)
blockChecksum — per-block xxHash32 for corruption detection
contentChecksum — end-of-frame xxHash32 over all decompressed data
contentSize — 64-bit total decompressed size in the header
blockIndependence — independent blocks (default) or linked blocks with 64KB history

Dependent Blocks

Linked blocks can reference up to 64 KiB of history from prior blocks, improving ratio on data with patterns spanning block boundaries:

final frame = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(blockIndependence: false),
);

Legacy Frame Format

The legacy format (0x184C2102) is what lz4 -l produces. It's simpler — no descriptor, no flags, no optional fields. Some systems still produce or consume this format:

final frame = lz4LegacyEncode(src);
final decoded = lz4FrameDecode(frame);

A complete implementation should both encode and decode legacy frames, not just decode them.

Skippable Frames

Skippable frames let you embed custom metadata in an LZ4 stream. Decoders skip them; encoders can use them for application-specific data:

import 'dart:convert';

final metadata = Uint8List.fromList(utf8.encode('{"version": 1}'));
final skippable = lz4SkippableEncode(metadata, index: 0);

// Concatenate with a regular frame
final combined = Uint8List.fromList([...skippable, ...lz4FrameEncode(data)]);

// Decoders skip the metadata and decode only the payload
final decoded = lz4FrameDecode(combined);

The index parameter (0–15) selects the magic number in the skippable range (0x184D2A50–0x184D2A5F).

Dictionary Support

Dictionaries let you compress small inputs more effectively by providing preset context. This is useful when compressing many small messages with shared patterns (e.g., log entries, API payloads):

// Encoding with a dictionary
final compressed = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(dictionary: myDictionary),
);

// Decoding with a dictionary resolver
final decoded = lz4FrameDecode(
  frameBytes,
  dictionaryResolver: (dictId) {
    if (dictId == 0x123456) return myDictionaryBytes;
    return null;
  },
);

The dictionary resolver is a callback — the decoder calls it with the dictId from the frame header, and you return the matching dictionary bytes. This design supports multiple dictionaries without embedding them in the library.

Streaming

Streaming encode/decode processes data chunk by chunk via StreamTransformer:

// Streaming decode
final decodedChunks = byteChunksStream.transform(
  lz4FrameDecoder(maxOutputBytes: 128 * 1024 * 1024),
);

// Streaming encode
final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoder(),
);

Streaming supports the same Lz4FrameOptions as the sync API:

final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoderWithOptions(
    options: Lz4FrameOptions(
      blockSize: Lz4FrameBlockSize.k64KB,
      blockIndependence: false,
    ),
  ),
);

Buffer Pooling

For zero-allocation streaming, buffer pools reuse memory across operations:

final pool = SimpleLz4BufferPool(
  maxTotalBuffers: 64,
  maxBuffersPerBucket: 8,
);

final decodedStream = byteChunksStream.transform(
  lz4FrameDecoder(bufferPool: pool),
);

The pool uses power-of-two slab bucketing (64B to 8MB) matching LZ4 block sizes. A SecureLz4BufferPool variant zeroes buffers on return for CWE-226 mitigation in security-sensitive contexts.

dart:convert Integration

The Lz4Codec wraps LZ4 frame encode/decode as a standard dart:convert Codec:

import 'dart:convert';

final codec = Lz4Codec();
final compressed = codec.encode(data);
final decoded = codec.decode(compressed);

// Fuse with other codecs:
final jsonLz4 = json.fuse(codec);
final payload = utf8.encode('{"hello":"world"}');
final compressedJson = jsonLz4.encode(payload);
final restored = jsonLz4.decode(compressedJson);

This makes LZ4 composable with the entire Dart conversion ecosystem — utf8, json, base64, and any custom codec.

When to Use dart_lz4

dart_lz4 is a pure-Dart implementation with zero runtime dependencies. It works on all Dart platforms including Web and WASM. It does not use FFI — native LZ4 bindings via dart:ffi will be faster for large payloads on native platforms.

Choose dart_lz4 when:

You need LZ4 on Web or WASM (FFI-based libraries can't run there without native binaries)
You want zero native dependencies (simpler deployment, no platform-specific binaries)
You need features beyond basic block compression (dictionaries, legacy frames, skippable frames, streaming with buffer pools)
You want bounds-safe decoding with output limits on untrusted input
You value supply-chain hardening (OpenSSF Gold, fuzzing in CI, 100% test coverage)

Choose FFI-based libraries when:

You need maximum throughput on native platforms
You don't need Web/WASM support
You're compressing large payloads where the native speed advantage matters

These are complementary tools for different use cases, not competitors in the same lane.

Summary

Feature	Support
Block encode/decode	Yes
Zero-copy decompression	Yes
LZ4HC levels 1–12	Yes
Frame format (all flags)	Yes
Legacy frame format	Encode + decode
Skippable frames	Encode + decode
Dictionary encode/decode	Yes
Streaming (StreamTransformer)	Yes
Buffer pooling (simple + secure)	Yes
dart:convert Codec	Yes
xxHash32	Yes (VM + Web parity)
Web/WASM	Yes
FFI acceleration	No (pure Dart)

dart_lz4 is on pub.dev with a 160/160 score, OpenSSF Best Practices Gold badge, 361 tests, and frame fuzzing in CI. Source code and documentation at GitHub.

Defending Against Decompression Bombs in Dart

Eduardo Rojas — Sun, 13 Sep 2026 18:33:13 +0000

Defending Against Decompression Bombs in Dart

A decompression bomb is a small compressed input that expands to an enormous size when decompressed. A 10 MB file might decompress to 10 GB — or more. If your application accepts compressed input from untrusted sources (file uploads, API payloads, user-generated content), this is a real attack vector.

This article walks through what decompression bombs are, why Dart applications are vulnerable, and how to defend against them — with code examples from dart_lz4, a pure-Dart LZ4 implementation.

What is a Decompression Bomb?

CWE-400 defines the vulnerability: "The software does not properly control the allocation and maintenance of a limited resource, thereby enabling an actor to influence the amount of resources consumed."

In the context of compression, the resource is memory. LZ4 — like all LZ-family algorithms — uses back-references to encode repeated data. A single byte in the compressed stream can reference a previously decompressed region and copy it forward. This means a compact compressed input can produce output many orders of magnitude larger than itself.

Consider this: LZ4's maximum block size is 4 MB. A well-crafted frame can chain hundreds of these blocks. A 1 MB compressed file could legitimately decompress to 4 GB if the blocks are independent, or even more with linked blocks referencing prior history.

Why Dart Applications Are Vulnerable

Dart's memory model makes this particularly dangerous:

Heap allocation is automatic. Uint8List allocations grow the Dart heap. There's no mmap with MAP_ANONYMOUS to get OS-level overcommit protection — the Dart VM allocates real memory.
Web targets are worse. On Web (JS/WASM), a 4 GB allocation will crash the browser tab. There's no graceful recovery.
Streaming amplifies the risk. If you're decoding a compressed stream chunk by chunk, you might not know the total decompressed size until you've already allocated it. Without a bound, an attacker can keep feeding compressed chunks that each expand to the maximum block size.
The dart:convert pattern hides the danger. If you use a Codec<List<int>, List<int>> wrapper, the decode call looks innocent:

final codec = Lz4Codec();
final decoded = codec.decode(compressedBytes); // How big is decoded?

Without a limit, decoded could be gigabytes.

The Defense: Bounded Decompression

The fundamental defense is simple: always set a maximum output size when decompressing untrusted input.

Setting `maxOutputBytes` on Frame Decode

The sync frame decoder accepts an explicit limit:

import 'package:dart_lz4/dart_lz4.dart';

// Set a reasonable upper bound for your use case
final decoded = lz4FrameDecode(
  untrustedFrame,
  maxOutputBytes: 64 * 1024 * 1024, // 64 MiB max
);

If the decompressed output would exceed this limit, the decoder throws an Lz4Exception before the allocation happens. No partial output, no memory exhaustion — just a clean error you can handle.

Streaming Decoder: The Higher-Risk Path

Streaming decoders are more dangerous because the input arrives in chunks. Without a bound, an attacker can keep feeding compressed data indefinitely. The streaming decoder also accepts maxOutputBytes:

final decodedChunks = byteChunksStream.transform(
  lz4FrameDecoder(maxOutputBytes: 128 * 1024 * 1024), // 128 MiB max
);

This was a real vulnerability in dart_lz4. Before v1.4.0, the streaming decoder had no default output limit — only the sync decoder did. If you used the streaming decoder without explicitly setting maxOutputBytes, you were unprotected. The v1.4.0 release closed this gap by applying a 256 MiB default to both decoders.

The `dart:convert` Codec

The Lz4Codec wrapper enforces the same 256 MiB default:

final codec = Lz4Codec(); // 256 MiB default
final decoded = codec.decode(untrustedInput);

// Or set your own limit:
final codec = Lz4Codec(maxOutputBytes: 32 * 1024 * 1024); // 32 MiB

This is important because the Codec pattern doesn't naturally expose safety parameters — you'd expect codec.decode(input) to "just work." The default limit ensures it does, safely.

What Should the Limit Be?

There's no universal answer. The limit should be based on what your application actually expects:

Use case	Suggested limit
API payload (JSON)	8–16 MiB
File upload (images)	64–128 MiB
Log stream	256 MiB
Trusted internal pipeline	Unlimited (pass `maxOutputBytes: -1` or a very large value)

The key principle: the limit should be the largest size your application would legitimately accept, not the largest size the format can produce.

Beyond Output Limits: Buffer Zeroization

There's a second, subtler vulnerability: CWE-226, "Sensitive Information in Resource Not Cleared Before Reuse."

When you reuse buffers across decompression operations (via a buffer pool for performance), the buffers may contain residual data from previous operations. If a subsequent decompression reads past the logical end of its output — due to a bug, a truncated block, or a malicious input — it can observe bytes from a prior, possibly sensitive, decompression.

dart_lz4 provides two buffer pool implementations:

// Standard pool — fast, but does NOT zero buffers on return
final pool = SimpleLz4BufferPool(
  maxTotalBuffers: 64,
  maxBuffersPerBucket: 8,
);

// Secure pool — zeroes buffers on return (CWE-226 mitigation)
final securePool = SecureLz4BufferPool();

SecureLz4BufferPool calls buffer.fillRange(0, buffer.length, 0) on every buffer return, ensuring no residual data persists. The performance cost is small — a single fillRange call per buffer — but the security guarantee is significant for multi-tenant or security-sensitive workloads.

Use SecureLz4BufferPool when:

Processing data from different users/tenants through the same pipeline
Handling sensitive payloads (session tokens, PII, decrypted data)
Operating in environments where memory isolation matters

Use SimpleLz4BufferPool when:

Single-tenant, high-throughput pipelines
Non-sensitive data
Buffers are guaranteed to be fully overwritten

Checksums Are Not Authentication

LZ4 frames support block checksums and content checksums (xxHash32). These detect accidental corruption — bit flips, truncated files, transmission errors. They do not provide cryptographic authentication.

If an attacker can modify the compressed input, they can produce a valid frame with different decompressed output that still passes the checksum. For authentication, use a MAC (e.g., HMAC-SHA256) or a digital signature over the compressed frame.

Summary

Threat	Mitigation	CWE
Decompression bomb (memory exhaustion)	`maxOutputBytes` on all decoders	CWE-400
Residual memory leakage (buffer reuse)	`SecureLz4BufferPool` with zeroization	CWE-226
Corrupted input	Block/content checksums	—
Tampered input	MAC or signature (not checksum)	—

The core principle: treat compressed input like any other untrusted input. Set bounds, validate output, and don't assume the decompressed size matches the compressed size. A 10 MB upload should never produce 10 GB of output.

dart_lz4 is a pure-Dart LZ4/LZ4HC implementation with bounds-safe decoding, buffer pool architecture, frame fuzzing in CI, and an OpenSSF Best Practices Gold badge. It works on all Dart platforms including Web and WASM.

DEV Community: Eduardo Rojas

The Most Complete LZ4 Implementation in Dart

The LZ4 Feature Surface

Block Format

Zero-Copy Decompression

LZ4HC: High-Compression Mode

Frame Format

Frame Options

Dependent Blocks

Legacy Frame Format

Skippable Frames

Dictionary Support

Streaming

Buffer Pooling

dart:convert Integration

When to Use dart_lz4

Summary

Defending Against Decompression Bombs in Dart

Defending Against Decompression Bombs in Dart

What is a Decompression Bomb?

Why Dart Applications Are Vulnerable

The Defense: Bounded Decompression

Setting maxOutputBytes on Frame Decode

Streaming Decoder: The Higher-Risk Path

The dart:convert Codec

What Should the Limit Be?

Beyond Output Limits: Buffer Zeroization

Checksums Are Not Authentication

Summary

Setting `maxOutputBytes` on Frame Decode

The `dart:convert` Codec