Update, 2026-11-07: v0.15.0 is out, and it is the version to use:
go get github.com/scigolib/hdf5@v0.15.0.
- Upgrade from v0.14.3. A group that switched to dense link storage while being written (at its ninth link, for example in a group that also holds an external link) could leave a file the C library rejects as truncated:
end-of-file address N is beyond the file size M. v0.15.0 fixes it. An affected file is repaired by zero-extending it to N (on Linux and macOS:truncate -s N file.h5).- New: user blocks (
WithUserBlock,FileWriter.WriteUserBlock, the equivalent ofH5Pset_userblock). That is where a MATLAB v7.3 file keeps its MAT header.- Behaviour change:
FileWriter.Deleteon a non-empty group now deletes the group and everything only it links to, asH5Ldeletedoes. Code that relied on the old refusal should check that the group is empty first.The follow-up article, about the
.matreader and writer built on this library: Read and Write MATLAB .mat Files in Go Without CGO.
In June 2025 I was porting SURD, a causal-discovery method from a Nature Communications paper, from its Python reference to Go (causalgo). The reference data came as MATLAB .mat files. In Python that's one scipy.io.loadmat call. In Go I found nothing I could rely on.
Those files happened to be the older v5 format. But a v7.3 .mat file, the format MATLAB needs for any variable of 2 GB or more, is an HDF5 file, so a Go reader that only handled v5 would fail on the first large dataset. And HDF5 itself is how a large part of science stores its data: climate models, telescopes, particle physics. In Go there were cgo bindings to the C library and a few pure-Go attempts, abandoned years ago and far from complete.
If I was going to write these libraries at all, it made sense to make them universal from the start. So four days later I started two repositories instead of one narrow reader: scigolib/matlab and scigolib/hdf5, a full HDF5 library, read and write, in pure Go with no cgo. It has been public for a year now, with 34 releases, and this is the first time I write about it.
Only later did I understand why the earlier attempts stopped. HDF5 is a scientific binary format with decades of history and a nuance in every structure: several versions of each header and message, variable-width fields set per file, half a dozen chunk index types, a filter pipeline, shared messages, and edge cases that only the C library's own test files show you.
What's in it today:
- superblock v0, v2 and v3; object headers v1 and v2; compact and dense attributes and links;
- contiguous, chunked and compact layouts; GZIP, shuffle and Fletcher32 (LZF and BZIP2 on read);
- compound, array, enum, variable-length, reference and opaque types;
- chunk indexes of every format version, including the ones HDF5 1.10+ writes (fixed and extensible array, B-tree v2, single chunk, implicit);
- reading tested against the HDF5 C library's own test files: 379 of the 383 valid ones read;
- files written by the library are checked with
h5dumpin CI.
One thing goes beyond the C library: when the version 2 B-trees behind dense attributes rebalance after deletions. C merges and redistributes their nodes on every delete (its older version 1 B-trees are never rebalanced). Here you choose: immediately, in batches, in a background goroutine, a strategy picked from the workload, or not at all for append-only files.
What isn't there yet. Each of these fails with an explicit error, not wrong data:
- virtual datasets;
- the N-bit, scale-offset and SZIP filters;
- the shared-message heap.
Not everything is done, but it's close.
scigolib/matlab reads and writes .mat files from v5 to v7.3, the v7.3 ones on top of it.
Then v0.14.3 happened. It started with one report:
One changed byte in a 5,656-byte file made the library try to allocate 40.9 GB.
A 5 KB file, a 40 GB allocation
Issue #75 from @TuSKan. Take a valid file, change the datatype size in the dataset's metadata, and ask for 12 values:
vals, err := ds.ReadSlice([]uint64{0}, []uint64{12})
// v0.14.2: tries to allocate 40,873,492,480 bytes, the process dies
v0.14.2 had just shipped a fix for this class of bug, with a security advisory (GHSA-8p88-h28q-w6m9). Whole-dataset reads were bounded. The hyperslab readers weren't: they parsed the dataset metadata again on their own and skipped the check.
That is the real bug. The fix for one path didn't reach the other paths, because each path had its own copy of the parsing.
The C library doesn't work like that. When it opens a dataset, it validates the storage once (H5D__contig_check, H5D__compact_init, H5D__chunk_set_sizes), and everything after works from that. v0.14.3 does the same:
- one validated dataset descriptor, built once per dataset;
- every reader takes its sizes from it;
- every result buffer is sized through one helper;
- a guard test fails if any allocation sized from file metadata appears outside that helper.
With that in place, the review found four more paths of the same class: the B-tree walk, filter output, variable-length amplification and the compound result size. All four are closed.
One case can't be bounded by the file size. A chunked dataset may legally be larger than the file, because unwritten chunks read as the fill value. Those reads get a per-file limit: File.SetReadLimit, 4 GiB by default.
Round-trip tests that lied
Issue #65 from @kf-domeng. A compound dataset written by the library opened fine in the library. h5py and H5Web refused it: invalid number of members: 0.
My round-trip tests were green the whole time. The writer and the reader had the same misunderstanding of the compound datatype message, so they agreed with each other. A round trip through your own code proves only that your code is consistent with itself.
The fix was to stop having two opinions about the format:
- one datatype codec for every class, mirroring the C library's encoder and decoder;
- the test: decode and re-encode all 1554 datatype messages in the official test files and compare with C's bytes.
They match. The one deliberate difference is 8 old v1 compounds with array members, which get upgraded exactly the way C upgrades them.
Compounds were not the worst case. Dense attribute storage, which HDF5 uses once an object has many attributes, had never been C-compatible at all:
- the attribute info message was written and read as message type 15, which in C is the shared-message table (the attribute info message is 21);
- the attribute name index was built with link records instead of attribute records, and C checks the record type;
- fractal heap objects were addressed from the end of the block header instead of the block start, so C answered "invalid fractal heap offset" for every object we wrote.
All of it passed our tests, because our reader made the same mistakes. Run against the C suite, the reader found 0 of the 9 attributes in tattr4_be.h5, and returned no error. Old files written that way still read, and their storage is moved to the C layout the first time their attributes change.
Compound types also got a public API, validated the way H5Tinsert validates members:
ct, err := hdf5.NewCompoundType(16, []hdf5.CompoundMember{
{Name: "id", Offset: 0, Type: hdf5.Int32},
{Name: "energy", Offset: 8, Type: hdf5.Float64},
})
if err != nil {
return err
}
dw, err := fw.CreateCompound("/events", ct, []uint64{2})
if err != nil {
return err
}
err = dw.Write([]map[string]any{
{"id": int32(1), "energy": 13.6},
{"id": int32(2), "energy": 27.2},
})
116 datasets that read "fine"
This one I found myself, and it bothered me more than the crash.
HDF5 has optional filters the library doesn't implement: N-bit, scale-offset, SZIP, blosc and others. The old code skipped an unknown filter and returned the stored bytes as if they were the values. In the C test suite, 116 datasets read without an error and returned garbage.
A crash you notice. Wrong numbers in a science pipeline you may never notice.
Now an unsupported filter returns ErrNotSupported, as the C library's filter pipeline does. Three smaller ones of the same kind were fixed along the way:
- unwritten chunks read as the fill value, not as zero;
- unsigned compound members above 2^31 no longer come back negative;
- Fletcher32 checksums are verified on read; they used to be stripped unchecked.
Errors you can act on
All the failures above now come back as errors you can tell apart with errors.Is:
f, err := hdf5.Open("data.h5")
if err != nil {
return err
}
defer f.Close()
f.SetReadLimit(1 << 30) // reads of data the file itself doesn't bound: at most 1 GiB
vals, err := ds.ReadSlice(start, count)
switch {
case errors.Is(err, hdf5.ErrCorrupt): // the file contradicts itself
case errors.Is(err, hdf5.ErrLimit): // a read above the limit
case errors.Is(err, hdf5.ErrNotSupported): // e.g. an SZIP-compressed dataset
case errors.Is(err, hdf5.ErrNoData): // nothing stored, no fill value
}
@skyqvn asked in #67 for structured metadata instead of parsing Info() strings. Datasets now have Shape, MaxShape, Datatype and Layout.
The writer moved closer to C as well. In more places it now writes exactly what the C library writes: the fill value message, filter flags, pipeline v2, dataspace v2 and message order. Tests compare its output with C-written files.
It also got faster. Against v0.14.2, opening and walking a file takes half the time with a tenth of the memory, reading many small slices of a chunked dataset takes 58–72% less time, and across the benchmarks the geometric mean is 44% less time and 64% less memory. (benchstat, six runs of each benchmark in release_bench_test.go, run against both versions on the same laptop.) Most of it came from one fix: object headers are now read with one read per chunk instead of two per message.
Our own old files
The hardest question in this release came from our own sibling project. scigolib/matlab ran its tests against the new, stricter superblock check, and 8 of its .mat test files were rejected. They had been written by this library in November 2025.
The library was right to reject them. Versions up to v0.13.4 wrote a CRC32 checksum where HDF5 requires Jenkins lookup3, and h5dump can't open those files either. Later versions had their own gaps: v2 object headers with no checksum at all, an end-of-file address not updated on close. Files from those versions are not valid HDF5, even though our own reader always opened them.
So what do you do with files your own library wrote wrong?
"Be liberal in what you accept" has a bad record: RFC 9413 describes how silent tolerance entrenches bugs. But a targeted workaround for one known buggy writer is standard practice: Apache Parquet ignores the min/max statistics written by old parquet-mr versions, which it identifies by created_by, instead of rejecting the files. A checksum on metadata guards integrity, not authenticity, so accepting one exact legacy value costs nothing in security.
The policy in v0.14.3:
- old files still read, and every deviation is reported through
File.Warnings()with a typed code. A file written by the C library or by the current version produces no warnings; - everything the library writes must be valid HDF5. What can be repaired in place is repaired; what can't is refused with a clear error;
-
go.modretracts v0.13.9 and older, the versions that wrote files no HDF5 tool can open, sogo getno longer picks them andgo list -m -utells users why; - planned: a repack function to convert old files, and from v1.0 a strict default with one explicit legacy option.
Read old files, but never write a new broken one, and tell people so nobody keeps producing them.
How I check it now
- The C library is the oracle. Every format check cites the C source it follows, file and line. Where the format spec defines a value, the spec decides, not the C code.
- Every fix starts with a test that fails on the old code. Corruption tests claim megabytes against kilobyte inputs. Crafted-input runs work under a 4 GB memory cap, so a missed bound fails the test instead of the machine.
- Two reviews. Every change gets an adversarial review. Once per release, a second, independent model reviews the final security diff. I started that after the v0.14.2 advisory turned out to be incomplete. I use AI tools for this work openly. They are fast at finding the fifth path of a bug class once you've found the first, and they never replace the test that proves it.
What I'd do differently
1. Validate once, at the boundary. Every reader that re-parses the metadata is a reader that can forget a check. One validated descriptor turned "fix it in N places" into "fix it once".
2. Never trust a round trip alone. Reading your own output proves only consistency. Test against the other implementation: h5dump, h5py, the C library's bytes.
3. Unknown means error, not "skip". A filter you don't understand is a reason to stop, not to hand back bytes.
4. Your own old files are part of the format. Users keep what you wrote. Decide what you will read, what you will warn about and what you will write before the strict check ships.
5. A security fix covers a class, not a path. After the first path, look for every other path that sizes memory from file metadata. In #75 there were five.
What's next
Users drive much of this work: the crash, the h5py failure and the metadata API all started as issues from people I'd never met. Thank you, @TuSKan, @kf-domeng and @skyqvn, and @rhaist and @chiliec for merged PRs.
scigolib is meant to be a home for scientific libraries in Go. The longer plan is Go counterparts of the Python libraries people reach for, visualization included. My own main focus is GPU graphics (GoGPU) and ML (Born). So if scientific Go is your thing, I'd be glad of help: issues, PRs, or bringing your own library into scigolib. A co-maintainer for hdf5 would be welcome.
hdf5: github.com/scigolib/hdf5 · pkg.go.dev
go get github.com/scigolib/hdf5@latest
Top comments (2)
The gap between round-trip correctness and defensive parsing is a classic file-format trap. A valuable regression suite needs hostile fixtures alongside valid ones: absurd dimensions, truncated metadata, overlapping offsets, and allocation budgets. The successful parse is not enough if a tiny input can force an unbounded allocation before validation.
Agreed, and the "before validation" part is the one that bit us. Every reader had passed its own tests, because each had its own copy of the parsing, and only some copies checked sizes before allocating.
What the suite looks like now, roughly in your terms:
Allocation before validation is the bug class; one validated descriptor per dataset is how we stopped fixing it path by path.