I benchmarked my language against Rust and Zig, and deleted my best number

I benchmarked my language against Rust and Zig, and deleted my best number

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything?

It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had.

The benchmark was measuring the order I ran things in

machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25%. That claim also shipped inside machin guide, which is what every coding agent reads to learn the language.

When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output:

for kernel in kernels:
    for lang in [machin, rust, zig]:
        for _ in range(5):    # all 5 machin, THEN all 5 rust, THEN all 5 zig
            time(binary)

It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last. Zig always went last. Zig always looked slowest.

The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did:

intsum 10^9      before (blocked)     after (interleaved)
machin              2832 ms                3079.7 ms
rust                3764 ms                3223.8 ms
zig                 3556 ms                3189.7 ms
                 "machin +20-25%"        machin +3% = a TIE

A 20-25% win became a tie. I deleted the claim from the README and from machin guide. The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start lying in the first place.

So what does machin actually win?

Two things, and neither is raw speed. On the four kernels machin wins one clearly (recursion, 26% faster than both), ties two, loses one. Same tier as Rust and Zig — it compiles to C, so it runs about as fast as C, and so do they.

1. It tells you about the bug before you run it

A program that waits for a value that can never arrive. In machin it's a receive on a channel nothing sends to; in Rust rx.recv() with the sender alive; in Zig sem_wait on a semaphore nobody posts. Same program:

machin   DL001 at COMPILE time: "receive on channel `ch` that is never
         sent to or closed - a guaranteed deadlock"
         ...and at runtime, exit 2 with the wait-cycle:
           fatal: deadlock - all 1 goroutine(s) blocked
             goroutine 0   waiting to receive on channel #0

rust     compiles clean, no diagnostic -> HANGS FOREVER (killed at 5s)
zig      compiles clean, no diagnostic -> HANGS FOREVER (killed at 5s)

This deserves precision, because it's easy to overclaim. Rust's type system prevents data races. It has never claimed to prevent deadlocks, and that recv() is idiomatic, unsafe-free, well-typed Rust. Zig doesn't attempt either.

Same for an out-of-range index: machin falsify enumerates small concrete inputs and hands back one that breaks the function — before the program runs, on code whose only call site is in range. rustc and zig say nothing.

Neither analysis proves absence, and I don't want to imply otherwise. falsify is unsound-complete: every counterexample it reports is real, but a clean result means "no bug within the bounds", never "correct". DL001 is the opposite trade — sound and false-positive-free, so it only fires when it can prove a channel is never fed, which means a clean result isn't a proof of deadlock-freedom either. "machin found nothing" is a much weaker statement than "machin found this bug, here's the input".

2. Binary size

stripped, both dynamically linked against system libc:

  machin      14 kB
  rust       335 kB

No std runtime to link — machin's output is C, and C's runtime is already on the machine. Note the comparison I did not make: unstripped it's 17 kB vs 4291 kB, about 250x. That number is bigger and worse, because it mostly measures how much debug info each toolchain leaves in.

And here is the correction I owe that 24x, which I only caught by going back and asking what it actually measures. It compares each toolchain's fixed floor, not how they scale:

program            machin        rust
hello world       14,544 B    343,568 B
fib(40)           14,544 B    335,472 B
a JSON+HTTP app   26,840 B          -

machin's hello world and its fib are byte-identical in size; Rust's differ by 2%. Neither number measures the program — both measure the baseline each toolchain links in, and real code adds real bytes to both. So the honest form is "Rust starts about 320 kB ahead", not "machin binaries are 24x smaller". The ratio shrinks as programs grow; the offset persists.

And what it loses

A benchmark suite that only lists wins is marketing.

• Build time: Rust wins. Bare rustc -C opt-level=3 builds these kernels in ~57 ms; machin takes ~95-116 ms, because its number includes the cc -O2 backend run. I deliberately did not use cargo, which would have charged Rust for lockfile resolution and made machin look good for a bad reason.
• Default runtime safety: Rust wins. Given an out-of-range index, Rust traps (exit 101). machin's default build prints a silent wrong 0 and exits successfully — exactly like Zig's ReleaseFast, which read 281479271677952 out of adjacent memory. Both need an opt-in to trap.
• Fully static, Zig wins. 491 kB against machin's 940 kB.

That second one matters enough to repeat, because I got it wrong first. My initial version compared machin --safe against Zig ReleaseFast — machin's checked mode against Zig's unchecked one — and machin looked safe by default. It is not.

The sieve: three months of a wrong explanation

machin has always trailed on the sieve by ~1.4x, and the README explained it confidently: "its slice indexing/layout is less optimal than a Rust Vec or a Zig slice." That was wrong, and nobody checked because the conclusion sounded plausible.

Timing the phases took ten minutes:

phase                              machin      rust
build the 10M array by append     70-83 ms   27-29 ms
the sieve loop itself              110 ms     111 ms   <- dead tie
the count loop                     5-7 ms       3 ms

Slice indexing ties Rust exactly. The entire gap is append growing the array: machin's arenas free nothing mid-life, so growing a slice can only allocate a fresh block and memcpy into it, ~21 times, never releasing the old buffers. Vec::push hands the block to realloc, and glibc extends it in place via mremap.

So I wrote the obvious fix: when the block being grown is the arena's most recent allocation, hand it straight to realloc. Then I checked one thing before shipping — whether MFL slices share backing storage:

a := []int{1, 2, 3}
b := a          // b[0] changes when a[0] does -> they SHARE
mutate(c)       // params share too

They do. Today, when append abandons a block, every existing alias keeps pointing at it and keeps reading valid memory, because the arena never frees. With in-place growth those aliases become use-after-free the moment the block moves. I had written a 45 ms speedup that trades a benchmark number for silent dangling reads — in a language whose entire pitch is catching that class of bug.

I threw it away and filed issue #578 with the directions that could actually be sound.

What the claim actually is

Not "machin is faster than Rust" — same tier, and it loses a kernel. Not "machin is safer than Rust" — Rust traps by default and machin does not.

The claim is machin tells you earlier. A deadlock that Rust and Zig discover as a hung process in production, machin reports at compile time with the wait-cycle. An out-of-range index neither mentions, machin hands you with a concrete failing input. That's a real difference, worth real debugging time, and it's a different claim from Rust's rather than a bigger one.

Every number is reproducible — sources for all three languages, the harnesses, and a run.sh per benchmark. Start at docs/BENCHMARKS.md, which indexes all seven benchmarks and states the losses next to the wins.

If you re-run them and get different numbers, I'd genuinely like to know. That's rather the point of shipping the harness.