gomarc: MARC21 for Go, 4x–11x faster than pymarc

If you work with library data, you work with MARC21 — the length-prefixed
binary record format catalogues have run on since the 1960s, complete with a
directory of field offsets, subfield delimiters, and a pre-Unicode character
encoding called MARC-8 that needs a lookup table with thousands of entries to
decode.

In Python that problem is solved: pymarc is
mature, complete, and pleasant to use. In Go it wasn't.

gomarc is a port of pymarc to Go. It
covers the binary MARC21 transmission format, MARC-8 to Unicode conversion,
MARCXML, and MARC-in-JSON — and on real catalogue exports it runs 4x to 11x
faster than the library it was ported from.

go get github.com/beyto1974/gomarc@v0.1.0

It reads like pymarc

If you know pymarc, you already know this API. Iterate records, pull the fields
you want:

reader := marc.NewReader(f)
for {
    record, err := reader.Next()
    if errors.Is(err, io.EOF) {
        break
    }
    if err != nil {
        log.Println(err) // permissive: bad records are skipped, not fatal
        continue
    }
    title, _ := record.Title()
    fmt.Println(title)
}

Title, Author, ISBN, ISSN, Subjects, Publisher, PubYear and more
are there as methods. For anything else, go at the tag and subfield directly:

value, ok := record.Get("245").Subfield("a")

for _, f := range record.GetFields("650") {
    fmt.Println(f)
}

Build records, modify them, write them back:

record.Get("245").SetSubfield("a", "The Zombie Programmer : ")

writer := marc.NewWriter(out)
writer.Write(record)

And convert to the formats the rest of your stack can actually read — both use
UTF-8 throughout instead of MARC-8, so standard tooling works:

s, err := record.AsJSON()          // MARC-in-JSON
records, err := marc.ParseXML(r)   // MARCXML

Large MARCXML files stream one record at a time via marc.NewXMLReader rather
than loading into memory.

The numbers

Two real catalogue exports — 138,076 records, 166 MB. AMD Ryzen 5 3600, Go
1.25.12, CPython 3.13.5, gomarc v0.1.0, pymarc 5.4.0, single-threaded, fastest
of 3 repetitions.

50,000 records, 58 MB

scenario	gomarc	pymarc	speedup	gomarc rec/s	pymarc rec/s
parse (MARC-8 to Unicode)	1.96 s	21.54 s	11.0x	25,545	2,321
parse (force UTF-8)	0.82 s	5.10 s	6.2x	61,096	9,813
parse + field access	2.02 s	23.01 s	11.4x	24,758	2,173
parse + write MARC21	4.69 s	26.01 s	5.5x	10,661	1,922
parse + write MARCXML	3.23 s	36.78 s	11.4x	15,481	1,360
parse + write MARC-in-JSON	8.18 s	32.22 s	3.9x	6,110	1,552

88,076 records, 108 MB

scenario	gomarc	pymarc	speedup	gomarc rec/s	pymarc rec/s
parse (MARC-8 to Unicode)	3.33 s	35.74 s	10.7x	26,459	2,464
parse (force UTF-8)	1.53 s	7.12 s	4.7x	57,662	12,370
parse + field access	3.72 s	34.55 s	9.3x	23,663	2,550
parse + write MARC21	5.48 s	38.45 s	7.0x	16,059	2,290
parse + write MARCXML	5.62 s	53.66 s	9.5x	15,674	1,641
parse + write MARC-in-JSON	7.09 s	45.67 s	6.4x	12,430	1,929

In practical terms: a full MARC-8 parse of 88,076 records drops from 36 seconds
to 3.3. A catalogue-to-MARCXML conversion drops from 54 seconds to 5.6. Peak
memory stays between 11 and 23 MB — gomarc streams, so file size doesn't drive
memory.

Where the win is biggest is the MARC-8 decode path, which is where real
catalogue data spends most of its time. Forcing UTF-8 instead narrows the gap
to roughly 5x, because the per-character table lookups that dominate the Python
run mostly disappear.

Why you can trust those numbers

Speed claims about a port are cheap. A benchmark of two libraries is really a
benchmark of two programs someone wrote, and if one parses lazily while the
other eagerly materialises everything, the ratio means nothing.

So the benchmark suite proves equivalence before it reports a single timing:

Identical record acceptance. Both libraries run permissively and report the
same counts — 50,000 and 88,076 records, zero errors — so they agree on exactly
which records are well-formed.

Identical decoded text. Summing the codepoint length of every extracted
title, author, ISBN and subject gives the same total, 671,999, from both
libraries. MARC-8 decoding agrees character for character.

Byte-identical output. Read a file with each library, write every record
back out as binary MARC21, and compare:

$ cmp py.marc go.marc && echo BYTE_IDENTICAL
BYTE_IDENTICAL

Every leader byte, every directory offset, every field terminator. That last
one is the practical point: if you're swapping gomarc into a pipeline that
pymarc currently feeds, the bytes coming out the far end don't change.

The harnesses, runner, raw per-run JSON and generated tables are all published
alongside the library, so you can re-run the whole thing:

REPS=3 ./run.sh
python report.py results.jsonl

The report includes a spread column — slowest repetition over fastest —
because a benchmark that publishes an estimator without its noise isn't worth
much.

Where it stands today

gomarc is at v0.1.0. Two things worth knowing before you adopt it:

• to_unicode=false (pymarc's RawField mode) isn't implemented yet. If you need raw undecoded bytes rather than Unicode strings, that path returns an error.
• MARC-in-JSON encoding is the least optimised corner. At 3.9x–6.4x it's still comfortably ahead of pymarc, but it allocates far more than the other paths and is the obvious next thing to tune.

Everything else — binary MARC21 read and write, MARC-8 conversion, MARCXML
read/write/stream, MARC-in-JSON read/write, field and subfield manipulation —
is covered and tested.

Try it

go get github.com/beyto1974/gomarc@v0.1.0

Repository, docs and full benchmark suite:
github.com/beyto1974/gomarc

Issues and pull requests welcome — particularly if you have MARC data that
breaks it.