DEV Community: SciGoLib

Read and Write MATLAB .mat Files in Go Without CGO: Files That Read as Empty, and MATLAB Files We Couldn't Open

Andrey Kolkov — Sat, 07 Nov 2026 11:57:29 +0000

The previous article was about scigolib/hdf5, a pure Go HDF5 library born out of the need to read MATLAB data in Go. This is the other half of that story: scigolib/matlab, the library that actually reads the .mat files.

It reads and writes both MATLAB formats in pure Go, with no cgo:

v5 (MATLAB v5 to v7.2): the classic binary format, including the zlib-compressed elements MATLAB writes by default;
v7.3: HDF5 underneath, through scigolib/hdf5.

What works today: every numeric class, complex numbers, multi-dimensional arrays, reading char arrays, both byte orders, 32-bit targets in CI, and a fuzz-tested v5 parser. Compressed elements are capped at 100 MB each and a 1000:1 ratio, so a zip bomb fails instead of eating memory.

What doesn't yet:

structures and cell arrays are not decoded (in v5 you get the class; a v7.3 struct comes back as one variable per field);
sparse matrices;
complex numbers in v7.3 use a real/imag group, not MATLAB's compound layout;
character arrays can be read but not written;
the writer doesn't compress;
an empty v5 cell {} makes Open fail.

Most of what I want to tell here are bugs, and nearly all of them share one cause:

Our tests mostly saw files we wrote ourselves.

Files saved with MATLAB's defaults read as empty

By default, MATLAB's save writes a Version 7 MAT file with compression on: each variable is wrapped in a miCOMPRESSED element. Up to v0.3.0, the reader skipped elements it didn't handle. So for a file saved with plain save data.mat, Open returned no error and zero variables.

Reproduced on the v0.3.0 tag, with SciPy's test file inner_outer_tbl_param.mat:

v0.3.0:   0 variables, err = nil
v0.3.19: 12 variables

Our tests never caught it. They read only our own writer's output, which isn't compressed. Three compressed SciPy files were already sitting in testdata, but no test opened them until v0.3.15.

v0.3.1 added zlib decompression with the bomb limits above. The lesson was the same one as hdf5's 116 datasets that read "fine": an empty result without an error is worse than a crash.

A bug report that waited three months

Issue #21 from @EtherHelm was the first and so far only external issue the project has had. MATLAB couldn't read files the library wrote. They had already traced the v5 cause themselves.

The array flags element has two 32-bit words. The spec puts the class in bits 0–7 of the first word and the flags in bits 8–11; the second word is nzmax. Our encodeArrayFlags put them in the wrong words. And the reader had the mirror bug: it read the class from the second word.

So the writer and the reader agreed with each other, and every round-trip test passed. Same story as the compound datatypes in hdf5: a round trip through your own code proves only that your code is consistent with itself.

What that looked like from the outside, reproduced on the v0.3.14 tag: real files saved by MATLAB and SciPy read with the correct values, but every variable's DataType came back as unknown, because the class was taken from nzmax, which is 0. A second bug sat next to it: the byte-order mapping was inverted, so the default output was big-endian.

The issue was opened on June 7 and answered and fixed on September 2. The plain reason: most of my time goes to GoGPU, Born and coregex, and a notification about a new issue here can get lost. Once those reach their releases, there will be more time to go through every library regularly. Until then, a report like this one, with the cause already traced, is the fastest way to a fix.

The fix (PR #23) put the spec layout on both sides and added two kinds of tests nobody can satisfy by agreeing with themselves:

reading SciPy's reference .mat files;
golden-byte tests that compare our output with the spec, byte by byte.

One more honest note. The v0.1.1-beta CHANGELOG said "Files now fully compatible with MATLAB/Octave". No test in the repository has ever run MATLAB or Octave. @EtherHelm's report was the first real check, and it failed.

A 136-byte file that could ask for 2 GiB

The same class of bug as hdf5's GHSA-8p88-h28q-w6m9: the v5 parser allocated each element at the size its tag claimed, before reading a byte of it, up to a 2 GiB cap. A 136-byte input that claimed 64 MiB allocated 64 MiB, then failed.

The rest turned up while bumping the hdf5 dependency:

an untyped 2^31 constant used as an int broke the build on linux/386, linux/arm and windows/386;
a claimed size of exactly 2 GiB slipped past the cap and panicked inside make on 386.

v0.3.16 validates every size against the bytes actually present, in one place, the tag reader. When the input length is unknown, it accumulates instead of allocating up front. A CI job now builds for linux/386, linux/arm and windows/386 and runs the tests on linux/386.

The fix that broke `bytes.Buffer`

The release review of v0.3.17 found that the v0.3.16 fix had broken something else.

The bound measured how much input was left after reading the 128-byte header. A bytes.Buffer reports only its unread part, so the bound came out 128 bytes short, and every valid v5 file opened from a bytes.Buffer was rejected: tag size beyond end of file.

No test had passed a bytes.Buffer to Open. Now one test opens every test file through os.File, bytes.Reader, bytes.Buffer and a reader that hides its size.

The same release also caught a stream cut off inside a skipped element being accepted, because io.CopyN errors were dropped. The review added that a file cut short gave a different error depending on whether its length was known. Every path now returns io.ErrUnexpectedEOF.

MATLAB's own v7.3 files didn't open

This bug came to light during the fact-checking for this article, and it affected every v7.3 file MATLAB writes.

MATLAB writes a v7.3 file as HDF5 behind a 512-byte user block. The first 128 bytes of that block are the usual MAT header, with version 0x0200; the HDF5 signature starts at byte 512. Our Open looked for HDF5 only at offset 0, didn't find it, and handed the file to the v5 parser:

tag size beyond end of file: 169478669 bytes at offset 512

All our v7.3 test files came from our own writer, which writes no user block. Same blind spot, third time.

There was no MATLAB-written v7.3 file at hand, so the layout was built from the references: matio (Mat_Open reads the version at bytes 124–125 and opens 0x0200 through HDF5; Mat_Create73 sets a 512-byte user block), SciPy's _get_matfile_version, and the HDF5 C library's signature search, which looks at offset 0 and then at every power of two from 512. h5jam -u produced a test file byte-identical to "block + file", and h5dump opens it. v0.3.18 detects the format by the header version, as matio and SciPy do.

The other direction took one more release. Our own v7.3 output had no MAT header either, and #21 had mentioned v7.3 too, so this may well have been that half of it. Writing the header needed user-block support in hdf5, the equivalent of the C library's H5Pset_userblock. hdf5 v0.15.0 added it, and v0.3.19 writes the header the way matio's Mat_Create73 does. h5dump opens the files. Whether MATLAB loads them, nobody has confirmed yet.

Test data and docs that claimed too much

Two smaller ones, in the same spirit.

The 8 v7.3 test files were written in November 2025 by an early hdf5 version, and h5dump can't open them; the hdf5 article tells that story from its side. They are regenerated now, and a test fails on any hdf5 warning. One old file stays in testdata/legacy to prove such files still read.

The README listed structures, cell arrays and sparse matrices as supported, and complex numbers as "proper MATLAB format for both v5 and v7.3". The reader decoded none of the three, and the v7.3 half of the complex claim wasn't true. It now says what the code does, with a Known Limitations list.

What I'd do differently

1. Your own files are not compatibility tests. Compressed files, the flags layout, the user block: each time, the tests saw little beyond what our writer produced. Test against files written by the other implementations from the start.

2. A silent empty result is worse than an error. Zero variables and nil looked like success.

3. Measure the input before you consume it, and test every kind of reader your API accepts.

4. Don't rely on email alone for issues. #21 waited three months because the notification went unseen. With many projects, look at each one's open issues directly from time to time.

What's next

MATLAB's compound layout for complex numbers, read and write;
structures, cell arrays, sparse matrices, empty cells;
writing character arrays.

If you have MATLAB or Octave, I'd like your help. Write a v5 and a v7.3 file with v0.3.19, load them there, and tell me what happens in an issue. Nobody has verified our v7.3 output in MATLAB itself, and a short report from a real MATLAB session is worth more than any test written without one.

Thanks to @EtherHelm for #21, and for doing half the work in it.

matlab: github.com/scigolib/matlab · pkg.go.dev

go get github.com/scigolib/matlab@v0.3.19

f, err := os.Open("data.mat")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

mat, err := matlab.Open(f) // v5 or v7.3, detected from the header
if err != nil {
    log.Fatal(err)
}
for _, v := range mat.Variables {
    fmt.Println(v.Name, v.DataType, v.Dimensions)
}

HDF5 in Pure Go, No CGO: a 5 KB File That Allocated 40 GB, and Round-Trip Tests That Lied

Andrey Kolkov — Sat, 07 Nov 2026 02:01:30 +0000

Update, 2026-11-07: v0.15.0 is out, and it is the version to use: go get github.com/scigolib/hdf5@v0.15.0.

Upgrade from v0.14.3. A group that switched to dense link storage while being written (at its ninth link, for example in a group that also holds an external link) could leave a file the C library rejects as truncated: end-of-file address N is beyond the file size M. v0.15.0 fixes it. An affected file is repaired by zero-extending it to N (on Linux and macOS: truncate -s N file.h5).

New: user blocks (WithUserBlock, FileWriter.WriteUserBlock, the equivalent of H5Pset_userblock). That is where a MATLAB v7.3 file keeps its MAT header.

Behaviour change: FileWriter.Delete on a non-empty group now deletes the group and everything only it links to, as H5Ldelete does. Code that relied on the old refusal should check that the group is empty first.

The follow-up article, about the .mat reader and writer built on this library: Read and Write MATLAB .mat Files in Go Without CGO.

In June 2025 I was porting SURD, a causal-discovery method from a Nature Communications paper, from its Python reference to Go (causalgo). The reference data came as MATLAB .mat files. In Python that's one scipy.io.loadmat call. In Go I found nothing I could rely on.

Those files happened to be the older v5 format. But a v7.3 .mat file, the format MATLAB needs for any variable of 2 GB or more, is an HDF5 file, so a Go reader that only handled v5 would fail on the first large dataset. And HDF5 itself is how a large part of science stores its data: climate models, telescopes, particle physics. In Go there were cgo bindings to the C library and a few pure-Go attempts, abandoned years ago and far from complete.

If I was going to write these libraries at all, it made sense to make them universal from the start. So four days later I started two repositories instead of one narrow reader: scigolib/matlab and scigolib/hdf5, a full HDF5 library, read and write, in pure Go with no cgo. It has been public for a year now, with 34 releases, and this is the first time I write about it.

Only later did I understand why the earlier attempts stopped. HDF5 is a scientific binary format with decades of history and a nuance in every structure: several versions of each header and message, variable-width fields set per file, half a dozen chunk index types, a filter pipeline, shared messages, and edge cases that only the C library's own test files show you.

What's in it today:

superblock v0, v2 and v3; object headers v1 and v2; compact and dense attributes and links;
contiguous, chunked and compact layouts; GZIP, shuffle and Fletcher32 (LZF and BZIP2 on read);
compound, array, enum, variable-length, reference and opaque types;
chunk indexes of every format version, including the ones HDF5 1.10+ writes (fixed and extensible array, B-tree v2, single chunk, implicit);
reading tested against the HDF5 C library's own test files: 379 of the 383 valid ones read;
files written by the library are checked with h5dump in CI.

One thing goes beyond the C library: when the version 2 B-trees behind dense attributes rebalance after deletions. C merges and redistributes their nodes on every delete (its older version 1 B-trees are never rebalanced). Here you choose: immediately, in batches, in a background goroutine, a strategy picked from the workload, or not at all for append-only files.

What isn't there yet. Each of these fails with an explicit error, not wrong data:

virtual datasets;
the N-bit, scale-offset and SZIP filters;
the shared-message heap.

Not everything is done, but it's close.

scigolib/matlab reads and writes .mat files from v5 to v7.3, the v7.3 ones on top of it.

Then v0.14.3 happened. It started with one report:

One changed byte in a 5,656-byte file made the library try to allocate 40.9 GB.

A 5 KB file, a 40 GB allocation

Issue #75 from @TuSKan. Take a valid file, change the datatype size in the dataset's metadata, and ask for 12 values:

vals, err := ds.ReadSlice([]uint64{0}, []uint64{12})
// v0.14.2: tries to allocate 40,873,492,480 bytes, the process dies

v0.14.2 had just shipped a fix for this class of bug, with a security advisory (GHSA-8p88-h28q-w6m9). Whole-dataset reads were bounded. The hyperslab readers weren't: they parsed the dataset metadata again on their own and skipped the check.

That is the real bug. The fix for one path didn't reach the other paths, because each path had its own copy of the parsing.

The C library doesn't work like that. When it opens a dataset, it validates the storage once (H5D__contig_check, H5D__compact_init, H5D__chunk_set_sizes), and everything after works from that. v0.14.3 does the same:

one validated dataset descriptor, built once per dataset;
every reader takes its sizes from it;
every result buffer is sized through one helper;
a guard test fails if any allocation sized from file metadata appears outside that helper.

With that in place, the review found four more paths of the same class: the B-tree walk, filter output, variable-length amplification and the compound result size. All four are closed.

One case can't be bounded by the file size. A chunked dataset may legally be larger than the file, because unwritten chunks read as the fill value. Those reads get a per-file limit: File.SetReadLimit, 4 GiB by default.

Round-trip tests that lied

Issue #65 from @kf-domeng. A compound dataset written by the library opened fine in the library. h5py and H5Web refused it: invalid number of members: 0.

My round-trip tests were green the whole time. The writer and the reader had the same misunderstanding of the compound datatype message, so they agreed with each other. A round trip through your own code proves only that your code is consistent with itself.

The fix was to stop having two opinions about the format:

one datatype codec for every class, mirroring the C library's encoder and decoder;
the test: decode and re-encode all 1554 datatype messages in the official test files and compare with C's bytes.

They match. The one deliberate difference is 8 old v1 compounds with array members, which get upgraded exactly the way C upgrades them.

Compounds were not the worst case. Dense attribute storage, which HDF5 uses once an object has many attributes, had never been C-compatible at all:

the attribute info message was written and read as message type 15, which in C is the shared-message table (the attribute info message is 21);
the attribute name index was built with link records instead of attribute records, and C checks the record type;
fractal heap objects were addressed from the end of the block header instead of the block start, so C answered "invalid fractal heap offset" for every object we wrote.

All of it passed our tests, because our reader made the same mistakes. Run against the C suite, the reader found 0 of the 9 attributes in tattr4_be.h5, and returned no error. Old files written that way still read, and their storage is moved to the C layout the first time their attributes change.

Compound types also got a public API, validated the way H5Tinsert validates members:

ct, err := hdf5.NewCompoundType(16, []hdf5.CompoundMember{
    {Name: "id", Offset: 0, Type: hdf5.Int32},
    {Name: "energy", Offset: 8, Type: hdf5.Float64},
})
if err != nil {
    return err
}
dw, err := fw.CreateCompound("/events", ct, []uint64{2})
if err != nil {
    return err
}
err = dw.Write([]map[string]any{
    {"id": int32(1), "energy": 13.6},
    {"id": int32(2), "energy": 27.2},
})

116 datasets that read "fine"

This one I found myself, and it bothered me more than the crash.

HDF5 has optional filters the library doesn't implement: N-bit, scale-offset, SZIP, blosc and others. The old code skipped an unknown filter and returned the stored bytes as if they were the values. In the C test suite, 116 datasets read without an error and returned garbage.

A crash you notice. Wrong numbers in a science pipeline you may never notice.

Now an unsupported filter returns ErrNotSupported, as the C library's filter pipeline does. Three smaller ones of the same kind were fixed along the way:

unwritten chunks read as the fill value, not as zero;
unsigned compound members above 2^31 no longer come back negative;
Fletcher32 checksums are verified on read; they used to be stripped unchecked.

Errors you can act on

All the failures above now come back as errors you can tell apart with errors.Is:

f, err := hdf5.Open("data.h5")
if err != nil {
    return err
}
defer f.Close()
f.SetReadLimit(1 << 30) // reads of data the file itself doesn't bound: at most 1 GiB

vals, err := ds.ReadSlice(start, count)
switch {
case errors.Is(err, hdf5.ErrCorrupt):      // the file contradicts itself
case errors.Is(err, hdf5.ErrLimit):        // a read above the limit
case errors.Is(err, hdf5.ErrNotSupported): // e.g. an SZIP-compressed dataset
case errors.Is(err, hdf5.ErrNoData):       // nothing stored, no fill value
}

@skyqvn asked in #67 for structured metadata instead of parsing Info() strings. Datasets now have Shape, MaxShape, Datatype and Layout.

The writer moved closer to C as well. In more places it now writes exactly what the C library writes: the fill value message, filter flags, pipeline v2, dataspace v2 and message order. Tests compare its output with C-written files.

It also got faster. Against v0.14.2, opening and walking a file takes half the time with a tenth of the memory, reading many small slices of a chunked dataset takes 58–72% less time, and across the benchmarks the geometric mean is 44% less time and 64% less memory. (benchstat, six runs of each benchmark in release_bench_test.go, run against both versions on the same laptop.) Most of it came from one fix: object headers are now read with one read per chunk instead of two per message.

Our own old files

The hardest question in this release came from our own sibling project. scigolib/matlab ran its tests against the new, stricter superblock check, and 8 of its .mat test files were rejected. They had been written by this library in November 2025.

The library was right to reject them. Versions up to v0.13.4 wrote a CRC32 checksum where HDF5 requires Jenkins lookup3, and h5dump can't open those files either. Later versions had their own gaps: v2 object headers with no checksum at all, an end-of-file address not updated on close. Files from those versions are not valid HDF5, even though our own reader always opened them.

So what do you do with files your own library wrote wrong?

"Be liberal in what you accept" has a bad record: RFC 9413 describes how silent tolerance entrenches bugs. But a targeted workaround for one known buggy writer is standard practice: Apache Parquet ignores the min/max statistics written by old parquet-mr versions, which it identifies by created_by, instead of rejecting the files. A checksum on metadata guards integrity, not authenticity, so accepting one exact legacy value costs nothing in security.

The policy in v0.14.3:

old files still read, and every deviation is reported through File.Warnings() with a typed code. A file written by the C library or by the current version produces no warnings;
everything the library writes must be valid HDF5. What can be repaired in place is repaired; what can't is refused with a clear error;
go.mod retracts v0.13.9 and older, the versions that wrote files no HDF5 tool can open, so go get no longer picks them and go list -m -u tells users why;
planned: a repack function to convert old files, and from v1.0 a strict default with one explicit legacy option.

Read old files, but never write a new broken one, and tell people so nobody keeps producing them.

How I check it now

The C library is the oracle. Every format check cites the C source it follows, file and line. Where the format spec defines a value, the spec decides, not the C code.
Every fix starts with a test that fails on the old code. Corruption tests claim megabytes against kilobyte inputs. Crafted-input runs work under a 4 GB memory cap, so a missed bound fails the test instead of the machine.
Two reviews. Every change gets an adversarial review. Once per release, a second, independent model reviews the final security diff. I started that after the v0.14.2 advisory turned out to be incomplete. I use AI tools for this work openly. They are fast at finding the fifth path of a bug class once you've found the first, and they never replace the test that proves it.

What I'd do differently

1. Validate once, at the boundary. Every reader that re-parses the metadata is a reader that can forget a check. One validated descriptor turned "fix it in N places" into "fix it once".

2. Never trust a round trip alone. Reading your own output proves only consistency. Test against the other implementation: h5dump, h5py, the C library's bytes.

3. Unknown means error, not "skip". A filter you don't understand is a reason to stop, not to hand back bytes.

4. Your own old files are part of the format. Users keep what you wrote. Decide what you will read, what you will warn about and what you will write before the strict check ships.

5. A security fix covers a class, not a path. After the first path, look for every other path that sizes memory from file metadata. In #75 there were five.

What's next

Users drive much of this work: the crash, the h5py failure and the metadata API all started as issues from people I'd never met. Thank you, @TuSKan, @kf-domeng and @skyqvn, and @rhaist and @chiliec for merged PRs.

scigolib is meant to be a home for scientific libraries in Go. The longer plan is Go counterparts of the Python libraries people reach for, visualization included. My own main focus is GPU graphics (GoGPU) and ML (Born). So if scientific Go is your thing, I'd be glad of help: issues, PRs, or bringing your own library into scigolib. A co-maintainer for hdf5 would be welcome.

hdf5: github.com/scigolib/hdf5 · pkg.go.dev

go get github.com/scigolib/hdf5@latest

DEV Community: SciGoLib

Read and Write MATLAB .mat Files in Go Without CGO: Files That Read as Empty, and MATLAB Files We Couldn't Open

Files saved with MATLAB's defaults read as empty

A bug report that waited three months

A 136-byte file that could ask for 2 GiB

The fix that broke bytes.Buffer

MATLAB's own v7.3 files didn't open

Test data and docs that claimed too much

What I'd do differently

What's next

HDF5 in Pure Go, No CGO: a 5 KB File That Allocated 40 GB, and Round-Trip Tests That Lied

A 5 KB file, a 40 GB allocation

Round-trip tests that lied

116 datasets that read "fine"

Errors you can act on

Our own old files

How I check it now

What I'd do differently

What's next

The fix that broke `bytes.Buffer`