Episode 1: What Happens Inside a ZST File
Episode 1: What Happens Inside a ZST FileZST files are becoming increasingly common in large downloads, software packages, backups, and server data. They are often considerably smaller than the original files, yet they can still be decompressed very quickly.
But how does that work?
Why Are Files Compressed?
A file consists of data. The more data it contains, the more storage space it requires and the longer it takes to transfer.
Compression attempts to represent the same information using less data. When the file is decompressed, the exact original file must be restored.
After decompression, the file is identical to the original, bit for bit. Nothing is omitted or simplified.
Repetitions Take Up Unnecessary Space
Consider this sentence:
Small cats cuddle on small cushions,
young cats cuddle on colorful cushions,
and old cats cuddle on soft cushions.The following parts occur several times:
A = cats cuddle on
B = cushionsIf we replace the recurring text with the variables A and B, we can represent the sentence more compactly:
Small A small B,
young A colorful B,
and old A soft B.This representation is not yet complete. We must also store what A and B mean:
A = cats cuddle on
B = cushionsThe original sentence can be reconstructed from this information. Each A is replaced with cats cuddle on, and each B is replaced with cushions.
This is already the fundamental idea behind lossless compression:
Recurring data is not stored in full every time. Instead, it is stored once and subsequently replaced with shorter references.### Zstandard Does Not Use Variables
Our variables A and B are only intended to make the principle easier to understand.
Zstandard does not understand words or sentences. It does not know what cats or cushions are. To the program, a file is simply a sequence of bytes.
When Zstandard finds a sequence of bytes that occurred shortly before, it can store a reference to the existing data. This reference essentially contains two values: the distance to the existing data and the length of the sequence to be copied.
In other words, Zstandard does not store something like this:
A = cats cuddle onInstead, it stores an instruction meaning:
Copy a sequence of bytes
that begins in the previously processed data
and append it at the current position.A repetition does not need to be a complete word. Parts of words, punctuation, file structures, and any other recurring sequences of bytes can also be detected.
That is why this method works not only with text, but also with source code, log files, database exports, and many other types of data.
How Is the File Decompressed?
During decompression, Zstandard reads the stored data in sequence.
New data is written directly to the output file. When the program encounters a reference, that reference contains a distance and a length.
The distance specifies how far before the current write position the sequence to be copied begins. The length specifies how many bytes are copied from there and appended at the current write position.
Up to this point, the following data has been decompressed:
Small cats cuddle on small cushions,
youngNow we count the bytes it contains:
Bytes 0–4 “Small”
Bytes 5–19 “ cats cuddle on”
Bytes 20–35 “ small cushions,” + line break
Bytes 36–40 “young”The space before cats is part of the sequence we are looking for. It begins at byte 5 and is 15 bytes long:
Byte 5 through byte 19
↓
cats cuddle onAfter young, the next write position is byte 41. The distance from there to the beginning of the known sequence at byte 5 is:
41 − 5 = 36 bytesThe reference therefore contains:
Reference
Distance 36 bytes
Length 15 bytesZstandard copies the 15 bytes that begin 36 bytes before the current write position and appends them at that position.
Already decompressed:
Small cats cuddle on small cushions,
young
Copied byte sequence:
cats cuddle onThe result is:
Small cats cuddle on small cushions,
young cats cuddle onZstandard then reads the next new data and the next reference. In this way, the original file is reconstructed step by step.
During compression, Zstandard first had to search for suitable repetitions. During decompression, the distance and length are already stored in the ZST file. The references only need to be read and carried out.
This is one of the main reasons why Zstandard can decompress data so quickly.
How Is the Remaining Data Compressed?
The references already explain an important part of Zstandard compression.
However, not all data is repeated. The words Small, small, young, colorful, old, and soft must also be stored.
Zstandard therefore also examines which values occur particularly often in the remaining data.
Frequently occurring values can be stored using shorter representations than rare values. This reduces the file size even further.
Zstandard therefore combines two fundamental techniques:
- Recurring byte sequences are described by their distance and length.
- The remaining data and references are encoded in an additional compact form. The actual mathematical methods are considerably more complex. For our understanding, the important point is that Zstandard does not save space only through complete repetitions. It also represents the remaining information as compactly as possible.
Why Is Zstandard So Fast?
A compression level is selected before compression begins.
The user may choose this level, for example in an application’s settings or with a command in the terminal. If no level is specified, the program normally uses a predefined default level.
The level determines how much time and memory Zstandard may use when searching for repetitions.
A low level performs a less extensive search. This makes compression particularly fast, but the resulting file may be slightly larger.
A higher level searches more thoroughly for good and longer matches. This can produce a smaller compressed file, but compression requires more time and memory.
The selected level does not change the content. Regardless of the level, decompression always produces the exact original file.
It only affects the balance between compression time, memory usage, and file size.
The previously used compression level does not need to be selected or even known during decompression. All identified references and all other required information are already stored in the ZST file.
Decompression therefore usually remains very fast: Zstandard does not need to search for repetitions again. It only needs to read and carry out the stored instructions.
What Else Is Special About a ZST File?
Zstandard processes larger files in sections and stores the compressed data in blocks. This means that the entire file does not necessarily need to be held in memory at once.
A ZST file can also contain a checksum. This can be used to detect whether the compressed data has been damaged. However, the checksum is optional and is therefore not present in every ZST file.
What Is a Zstandard Dictionary?
Our example using A and B resembles a dictionary. For ordinary references, however, Zstandard does not use such a list. It describes existing data using a distance and a length.
Normally, Zstandard can refer to byte sequences that occurred in the previously processed portion of the same file. A very small file, however, contains only a limited amount of previous data that can be referenced. Even when many small files have a similar structure, each file is initially processed on its own.
For such cases, a Zstandard dictionary can be created in advance. It contains typical byte sequences learned from many similar files. This allows Zstandard to use a short reference the first time such a sequence occurs.
The same dictionary is required for decompression. However, an external dictionary is not usually required for ordinary .zst downloads.
Which Files Compress Well?
The resulting file size depends on its content.
Text files, source code, log files, database exports, and other structured data often contain many repetitions. They can therefore usually be compressed effectively.
Formats such as JPEG, MP3, MP4, and ZIP are already compressed. Many useful repetitions have already been removed. Compressing them again with Zstandard may save very little additional space.
Zstandard therefore cannot reduce every file to a particular percentage of its original size. The decisive factor is how much redundancy the data contains.
What Is the Difference Between ZST and ZIP?
A ZIP file can contain multiple files and folders. It is an archive format that also uses compression.
Zstandard, by contrast, is primarily a compression method for a single stream of data.
For example, a single file becomes:
Report.txt → Report.txt.zstAfter decompression, Report.txt is restored.
An ordinary ZST file does not itself manage multiple filenames, folder structures, and file permissions. When multiple files need to be packaged together, Zstandard is therefore often combined with TAR.
What Is a TAR File?
TAR combines multiple files and folders into a single archive file. It does not normally compress the data.
First, several files are combined into one TAR archive:
Image.jpg
Text.txt
Documents/
↓
Archive.tarThe archive can then be compressed with Zstandard:
Archive.tar
↓
Archive.tar.zstThe .tar.zst extension therefore describes two consecutive steps:
- .tar means that multiple files and folders have been combined.
- .zst means that the TAR archive has been compressed with Zstandard. During extraction, these steps are performed in reverse order. First, the Zstandard compression is removed. The TAR archive is then opened and its folder structure is restored.
A .tar.zst file is therefore not an entirely separate archive format. It is a TAR archive compressed with Zstandard.
Why Is Zstandard Becoming More Common?
Zstandard offers a very good balance between file size and speed.
It can compress data quickly, decompress it very quickly, and can be adapted to different requirements through its various compression levels.
This makes it particularly suitable for software packages, backups, server logs, database exports, containers, system images, and large downloads.
Fast decompression is especially valuable for downloads. A file may be created only once, but subsequently downloaded and decompressed on thousands of devices.
The provider can therefore invest a little more time in compression, while every recipient benefits from a smaller file and faster decompression.
Summary
Zstandard reduces file sizes by avoiding the repeated storage of identical data. Instead, it uses short references and represents frequently occurring values in a particularly compact form.
During decompression, these references are carried out and the original data is restored exactly.
Zstandard does not remove any information. It simply finds a more space-efficient representation of the same data.
This combination of effective compression and very fast decompression is making ZST an increasingly important format for software, servers, and large downloads.
Zstandard is not the only compression method. ZIP, GZIP, Brotli, LZ4, and XZ have different priorities. In Episode 2, we compare their file sizes and their compression and decompression speeds.### Open ZST Files on Mac
macOS cannot extract .zst and .tar.zst files directly in Finder.
With unpackZST, both formats can be opened using drag and drop — without the terminal or any additional configuration.
Top comments (0)