If you have ever used ZFS, zram on Linux, Kafka, or ClickHouse, you have probably used LZ4 without knowing it. It is one of those libraries that quietly sits underneath a lot of infrastructure because it does one thing extremely well: it compresses and decompresses data fast.
This article covers what LZ4 is, when you should (and should not) use it, the things that trip developers up, and how it compares to other free and paid compression tools.
What is LZ4?
LZ4 is a lossless compression algorithm and C library written by Yann Collet (who also created Zstandard). It was first released in 2011, is licensed under the BSD 2-Clause license for the library, and has more than 12k stars on GitHub.
The design goal is simple: trade some compression ratio for raw speed.
- Compression: more than 500 MB/s per core
- Decompression: multiple GB/s per core, often hitting RAM bandwidth limits
- Ratio: roughly 2:1 on typical mixed data at the default level
In other words, LZ4 does not make your files as small as gzip or Zstandard can, but it is so fast that compression often costs less than the I/O or network time it saves.
The numbers
These figures come from the benchmark published in the LZ4 README (lzbench, Silesia corpus, single thread, Core i7-9700K @ 4.9GHz):
| Compressor | Ratio | Compression | Decompression |
|---|---|---|---|
| memcpy (no compression) | 1.000 | 13700 MB/s | 13700 MB/s |
| LZ4 default | 2.101 | 780 MB/s | 4970 MB/s |
| LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
| Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
| Zstandard 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
| zlib deflate -1 | 2.730 | 100 MB/s | 415 MB/s |
| LZ4 HC -9 | 2.721 | 41 MB/s | 4900 MB/s |
| zlib deflate -6 | 3.099 | 36 MB/s | 445 MB/s |
Two things stand out:
- LZ4 decompresses several times faster than everything else in the table.
- LZ4 HC reaches a ratio close to zlib -1 while keeping the same blazing decompression speed. That matters a lot for "compress once, read many times" workloads.
Note: these numbers are from older versions of each library. Always benchmark with your own data (more on that below).
Getting started
Install
Most Linux distributions and package managers already ship it:
# Debian / Ubuntu
sudo apt install lz4 liblz4-dev
# Fedora / RHEL
sudo dnf install lz4 lz4-devel
# macOS
brew install lz4
# Windows (vcpkg)
vcpkg install lz4
Or build from source:
git clone https://github.com/lz4/lz4.git
cd lz4
make
sudo make install
Command line
The lz4 CLI feels a lot like gzip:
# Compress (creates file.txt.lz4)
lz4 file.txt
# Decompress
lz4 -d file.txt.lz4
# Higher compression (levels 3-12 use the HC compressor)
lz4 -9 big.log
# Use multiple threads (v1.10+), 0 = auto-detect cores
lz4 -T0 huge-file.bin
# Compress a directory with tar
tar -I lz4 -cf backup.tar.lz4 ./my-folder
# Benchmark levels 1 through 9 on your own file
lz4 -b1e9 sample.dat
That last command is the most useful one in this article. Run it on real data before choosing a level.
C API
The core API is small. Here is a complete round trip:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <lz4.h>
int main(void) {
const char *src = "Hello LZ4! Hello LZ4! Hello LZ4! Hello LZ4!";
const int src_size = (int)strlen(src) + 1;
/* Worst-case output size for incompressible input */
const int max_dst = LZ4_compressBound(src_size);
char *compressed = malloc(max_dst);
const int comp_size = LZ4_compress_default(src, compressed, src_size, max_dst);
if (comp_size <= 0) { fprintf(stderr, "compression failed\n"); return 1; }
/* You must know (or store) the original size yourself */
char *restored = malloc(src_size);
const int dec_size = LZ4_decompress_safe(compressed, restored, comp_size, src_size);
if (dec_size < 0) { fprintf(stderr, "decompression failed\n"); return 1; }
printf("%d -> %d bytes, restored: %s\n", src_size, comp_size, restored);
free(compressed);
free(restored);
return 0;
}
Compile with:
gcc demo.c -llz4 -o demo
Python
pip install lz4
import lz4.frame
data = b"some repetitive payload " * 10_000
compressed = lz4.frame.compress(data)
restored = lz4.frame.decompress(compressed)
assert restored == data
print(f"{len(data)} -> {len(compressed)} bytes")
Other languages
LZ4 has bindings or ports in almost every ecosystem:
| Language | Common package |
|---|---|
| Python |
lz4 (PyPI) |
| Go | github.com/pierrec/lz4/v4 |
| Rust |
lz4_flex (pure Rust) or lz4 (bindings) |
| Java |
at.yawk.lz4:lz4-java (see the security note below) |
| C#/.NET | K4os.Compression.LZ4 |
What developers really need to know
This is the section to read even if you skip everything else.
1. Block format vs frame format
LZ4 has two formats, and mixing them up is the most common source of bugs.
-
Block format (
lz4.h): raw compressed bytes. No header, no checksum, no record of the original size. You must store the uncompressed size yourself and pass it at decompression time. -
Frame format (
lz4frame.h): a self-describing container with a magic number, optional content size, optional xxHash checksums, and support for streaming arbitrarily large data.
Rule of thumb: use the frame format unless you are building your own container and know exactly why you need raw blocks. The frame format is what the CLI produces, and it is what other tools and languages expect for interoperability.
2. Always use the safe decompressor
Use LZ4_decompress_safe() (or the frame API). It checks bounds and will not write past your buffer even if the input is corrupted or malicious.
Avoid LZ4_decompress_fast(). It is deprecated, and it trusts the input, which makes it unsafe for anything that comes from a network, a user, or a disk you do not fully control.
3. Compression level does not slow down decompression
LZ4 offers:
-
Acceleration (
LZ4_compress_fastwith a factor > 1): even faster compression, lower ratio. - Default (level 1): the classic fast mode.
- Level 2: new in v1.10, a middle ground between fast and HC.
- HC levels 3 to 12: much slower compression, better ratio.
All of these decompress at essentially the same speed. So for assets, packages, or anything written once and read many times, use a high HC level at build time and enjoy fast reads forever.
4. Small data needs a dictionary
Compressing a 300-byte JSON message on its own gives poor results because there is no history to find repeats in. LZ4 supports dictionary compression, which pre-loads shared context. Only the last 64 KB of a dictionary is used.
You can train a dictionary with Zstandard's builder and use it with LZ4:
zstd --train samples/*.json -o messages.dict
lz4 -D messages.dict message.json
Dictionary support was fully stabilized in v1.10.
5. Multithreading arrived in v1.10
Version 1.10.0 (July 2024, "Multicores edition") added multithreaded compression to the CLI, which gives large speedups on big files with many cores. Decompression is still effectively single-threaded, but it is already fast enough to saturate most NVMe drives.
If your distro ships an older 1.9.x build, you will not get -T.
6. Do not compress data that is already compressed
JPEGs, MP4s, ZIPs, and encrypted data will not shrink. LZ4 is fast enough that this is rarely a disaster, but you still burn CPU for nothing. Many systems check the ratio after compressing and store the raw data if the gain is too small.
7. Check your bindings for security updates
The C library is heavily fuzzed (it is part of OSS-Fuzz), but bindings are separate projects with their own bugs. A notable example: the original org.lz4:lz4-java is no longer maintained, and the community fork moved to the at.yawk.lz4:lz4-java Maven coordinates, which includes fixes for CVE-2025-12183 and CVE-2025-66566. If you are on the JVM, check which artifact your dependency tree actually pulls in (Kafka clients and many other libraries depend on it transitively).
Where LZ4 is used in the real world
- Linux kernel: zram, zswap, SquashFS, and kernel image compression
- Filesystems: ZFS and OpenZFS use LZ4 as the go-to on-the-fly compression
- Databases and analytics: ClickHouse (default codec), plus many column stores
- Messaging: Kafka and Pulsar support LZ4 for message batches
- Game engines and asset pipelines: fast asset loading
The common thread: places where latency and throughput matter more than squeezing out the last few percent of disk space.
Comparison with other tools
Free and open source
| Tool | License | Speed | Ratio | Best for |
|---|---|---|---|---|
| LZ4 | BSD-2 | Fastest decompression, very fast compression | Low to medium (HC) | Real-time, in-memory, storage engines, read-heavy data |
| Zstandard (zstd) | BSD / GPLv2 | Fast, tunable from very fast to very slow | Medium to very high | The best general-purpose default today |
| Snappy (Google) | BSD-3 | Fast | Low (similar to LZ4) | Existing Google/Hadoop/Parquet ecosystems |
| gzip / zlib (deflate) | zlib | Slow compression, slow decompression | Medium | Maximum compatibility (HTTP, archives, everywhere) |
| Brotli (Google) | MIT | Slow at high levels, decent decompression | High, especially for text | Static web assets served over HTTP |
| xz / LZMA | Public domain / LGPL | Very slow compression | Very high | Distribution packages, long-term archives |
| LZO | GPL | Fast | Low | Legacy systems |
How to read this:
- LZ4 vs Zstandard: Zstandard (same author) is the better all-rounder. At its fastest levels it is competitive with LZ4 on compression speed and gets a noticeably better ratio, but LZ4 still decompresses roughly 2-4x faster. Pick LZ4 when decompression speed or CPU budget is the bottleneck; pick zstd when storage or bandwidth costs dominate.
- LZ4 vs Snappy: they target the same niche with similar ratios. In most benchmarks LZ4 is faster on both compression and decompression. Snappy is mainly worth choosing when your ecosystem already standardizes on it.
- LZ4 vs gzip: gzip compresses better but is many times slower in both directions. Use gzip when you need universal compatibility, not speed.
- LZ4 vs Brotli / xz: different goals. Those tools aim for small output and accept slow compression; LZ4 aims for speed.
Paid and commercial options
There are few commercial libraries in this space because the open source options are so strong, but a few are worth knowing:
| Tool | Model | What it offers | When it makes sense |
|---|---|---|---|
| Oodle (Epic Games Tools, formerly RAD Game Tools) | Proprietary, commercially licensed; bundled with Unreal Engine | Codecs such as Kraken, Mermaid, Selkie and Leviathan, tuned for very fast decompression with better ratios than LZ4 at similar speeds; also network packet compression | Game studios shipping large assets, especially on consoles |
| Intel QuickAssist Technology (QAT) | Hardware accelerator, paid via supported CPUs or add-in cards | Offloads compression (including deflate and LZ4) from the CPU | High-throughput servers where CPU cycles are expensive |
| Commercial archivers (WinZip, WinRAR, PKZIP) | Paid end-user software | GUI tools, encryption, archive management | Desktop users, not libraries you embed in an application |
Honest take: for the vast majority of developers there is no compelling reason to pay for a compression library. The paid options shine in narrow cases, mainly game asset delivery (Oodle) and hardware offload at data center scale (QAT). For everything else, LZ4 and Zstandard cover the field.
Wrapping up
LZ4 is not trying to win the compression ratio contest. It is trying to make compression so cheap that you can turn it on almost everywhere, and it succeeds.
Top comments (0)