DEV Community

Cover image for HatchZip promises to extract an 839 MB archive in 839 MB of disk. I measured the blocks.
frank chu
frank chu

Posted on

HatchZip promises to extract an 839 MB archive in 839 MB of disk. I measured the blocks.

A 180 GB game download that unpacks to 200 GB needs 380 GB of disk with a normal extractor, because at the worst moment the full archive and the full output exist at the same time. HatchZip — 1,692 stars in its first week on GitHub, v0.1.3 — claims to fix that by extracting from the end of the archive and truncating each finished part off the archive file as it goes, so "you only need free space for the output."

That's a falsifiable claim with a number attached, so I measured it. The catch, which took me three runs to find, is that the obvious way to measure it lies to you — and the tool's own space-saving only switches on when your disk is actually nearly full.

The test archive

I built the worst case for compression and the normal case for game downloads: 40 files of 20 MiB of pure random data (already-compressed assets behave the same way), zipped at level 1.

files:            40 × 20 MiB random + 1 directory = 41 entries
uncompressed:     838.9 MB
game.zip:         839,007,384 bytes (random data doesn't compress;
                  the archive is slightly larger than its contents)
hatchzip:         0.1.3, official Linux CLI tarball,
                  sha256 matched the release's published checksum
Enter fullscreen mode Exit fullscreen mode

Baseline, unzip, sampling allocated blocks (st_blocks, not file sizes) every 20 ms:

unzip:      peak 1,677.9 MB on disk, final 1,677.9 MB, 9.4 s
Enter fullscreen mode Exit fullscreen mode

Exactly archive + output, as advertised. Now HatchZip, on a disk with 97.5 GB free:

hatchzip:   peak 1,677.9 MB on disk, final 838.9 MB, 8.7 s
Enter fullscreen mode Exit fullscreen mode

Identical peak. My first sampler used stat file sizes and showed the same thing from another angle: the archive's size sat at 839.0 MB for the entire run and then the file vanished. No progressive truncation at all. If I'd stopped there, this would be a debunking post. It isn't — but the reason is in the tool's design doc, and it's a real caveat.

The batch is sized by your free space

From HatchZip's own docs/how-it-works.md: units (one ZIP entry each) are processed in batches, and a batch takes "units while their extracted size fits in the free space that exists right now, minus --keep-free. A batch holds at most 512 units and 2 GB." Truncation happens per batch, at commit time.

My whole archive is 839 MB. On a disk with 97.5 GB free, the entire archive is one batch: extract everything, sync, then truncate once. The space saving is real but conditional — it materializes only when free space is scarce enough to force small batches. On a roomy disk, HatchZip is just an extractor that deletes the archive at the end.

Its own pre-flight check tells you the floor, though:

$ hatchzip check game.zip
game.zip  839.0 MB zip, 838.9 MB uncompressed in 41 entries
  available                    97.5 GB
  standard extraction needs    838.9 MB
  required                     21.0 MB (+2.0 GB reserved)
Enter fullscreen mode Exit fullscreen mode

21.0 MB is the largest single entry. That's the theoretical minimum headroom: output size plus one unit, plus a 2 GB safety reserve it refuses to touch by default.

Simulating a full disk

--keep-free lets you lie to the tool about how full the disk is. I set it so HatchZip believed only ~300 MB was usable, and watched:

run 1:  extracted 14 files, archive truncated 839.0 MB → 545.3 MB, then:
        error: not enough disk space to continue: 'src/asset_11.bin'
        requires 21.0 MB and 6.4 MB is available (91.9 GB reserved).
        14.6 MB more is required. Progress is saved; free up space
        and resume the extraction.
run 2:  "resuming: 545.3 MB of 838.9 MB remaining", 27 entries left;
        extracted 14 more, stopped again at 251.7 MB remaining
run 3:  (default 2 GB reserve) resumed and completed;
        archive deleted; all 40 files present
Enter fullscreen mode Exit fullscreen mode

Three things worth noting. First, the truncation is real when batches are small: the archive physically shrank from 839.0 to 545.3 MB mid-run, in step with the output growing — total on-disk bytes hovered near 839 MB instead of 1,678 MB. Second, the failure mode is boring in the best way: it stops before writing a file it can't finish, leaves two sidecar files (game.zip.hatchzip, the journal, and game.zip.hatchzip-tail, the saved ZIP central directory), and the identical command resumes exactly where it stopped. Third, my "6.4 MB available" stop was partly self-inflicted — the reserve arithmetic counts every other file on the real filesystem, including my earlier test runs. The tool would rather stop three times than fill your disk once. That's the right bias for something that eats its own input.

Integrity check after the stop/resume/stop/resume dance:

md5 of the 40 extracted files' md5sums:  539de588defe22cf09d2bf217a78e2e1
md5 of the original 40 files' md5sums:   539de588defe22cf09d2bf217a78e2e1
Enter fullscreen mode Exit fullscreen mode

Byte-identical.

The floor is the largest unit — except when it isn't

One 600 MiB file in a zip (629.3 MB archive) changes the math:

$ hatchzip check big.zip
  standard extraction needs    629.1 MB
  required                     538.5 MB (+2.0 GB reserved)
Enter fullscreen mode Exit fullscreen mode

Not 629 MB, but not 21 MB either. For a single file ≥512 MiB, HatchZip streams it and punches holes in the archive every 256 MiB of output (the design doc's resume-point machinery, borrowed from zlib's zran.c for deflate streams), so the floor drops below the file size but stays in the same ballpark. And the doc is honest about the worst case: a solid 7z block is one unit, and 7-Zip's defaults can put the whole archive in one block, "which leaves nothing to gain." The space check reports that before touching anything.

Two smaller findings from the trenches: outside a terminal, extract refuses to run without -y ("HatchZip removes the archive during extraction; pass --yes to confirm") — destructive-by-default tools should all do this. And .tar.gz can't be truncated from the end at all (one deflate stream, decoded front to back), so there the length stays put and space comes back only via hole punching. Apparent size is the wrong instrument for almost everything this tool does; measure blocks or measure df.

Verdict

The headline claim holds, with an asterisk the size of your free space: peak usage is output + largest batch, batches shrink as the disk fills, and on a disk with room to spare it degenerates gracefully into a normal extraction. The resume journal survived two forced stops with zero corruption. For the actual use case in the README — a nearly full drive and a 180 GB download — that's exactly the behavior you want, and it's rarer than it should be: a tool whose space plan you can read (hatchzip check) before it modifies a single byte.

Sources: github.com/xevrion/hatchzip (README and docs/how-it-works.md), release v0.1.3. All archives, measurements, and forced-space runs are my own, on a 2 vCPU AMD EPYC VM; block sampling via st_blocks at 20 ms intervals.

Top comments (0)