I ported BurntSushi/toml from Go to Rust: about 4,300 lines of reflection-heavy decoder behind a 775 file conformance corpus, written by the author of ripgrep. Roughly two and a half days of work. The port now passes every conformance test in its target version, produces byte-identical semantics to the original on every valid document in the corpus, and rejects nineteen documents the original wrongly accepts.
None of that is the interesting part.
The interesting part is that I added four separate layers of proof, and every single layer caught a bug that the layer beneath it could not see, including bugs in the proof itself. And the worst defect in the finished port is one that all four layers are structurally incapable of detecting. I only found it because I went looking after the tests went green.
This is the story of what each layer actually bought me.
What I picked, and why it was a good pick
Track E, Go to Rust. I wanted a target where "does it work" has an objective answer, because I did not want to spend the weekend arguing with myself about whether my output was correct.
TOML is close to perfect for this. There is a language neutral conformance suite (toml-test) that speaks TOML on stdin and tagged JSON on stdout, so a Go implementation and a Rust implementation can be compared without either knowing the other exists. The original ships that corpus in-tree. That meant I could build a differential oracle instead of trusting my own judgement, which turned out to matter more than anything else I did.
The counterintuitive consequence: I kept all the Go source in the repository. It looks like I forgot to clean up. I did not. internal/toml-test/tests/ is the corpus, cmd/toml-test-decoder is the oracle I diff against, and internal/toml-test/version.go encodes the version selection rules my Rust runner has to mirror. Deleting the original would have deleted my evidence.
Layer 0: my tests passed, and they were lying
The first version of my conformance runner reported a score. The score was fiction, in three separate ways:
- It compared the two JSON documents as text. So
3e+14and3.0e14counted as a mismatch. Three files that my parser handled correctly were being reported as failures. - It read every test file as a UTF-8 string. The nine deliberately non-UTF-8 encoding tests failed the read, and the error was swallowed, so those tests silently vanished from the denominator.
- Every
spec-1.1.0/*file got a free pass through a path that never actually compared anything.
The reference runner does none of that. internal/toml-test/json.go compares floats numerically, datetimes as instants (so .6 and .600 are the same value, and -00:00 and Z are the same value), and booleans case-insensitively, with type tags matched exactly. I had written my own comparison because it seemed obvious. It was obvious and wrong.
The lesson I would put on a poster: when you port a project that ships a test runner, port the test runner first, and port it faithfully. Your harness is production code. Mine was the buggiest component in the repository for the first day and a half, and because it was the thing measuring everything else, its bugs were invisible by construction.
Layer 1: 775 tests is not a real number
Once the harness was honest, I hit something better.
The corpus holds the TOML 1.0.0 and 1.1.0 suites side by side, and they contradict each other. valid/inline-table/newline.toml requires accepting a trailing comma in an inline table. invalid/inline-table/trailing-comma.toml requires rejecting it. No implementation on earth passes both. The reference runner resolves this with per-version exclusion lists in version.go, and a lot of "we pass N/775" claims you will read are quietly scoring against a target that cannot be hit.
So I mirrored the exclusion lists exactly and made the version selectable. Against TOML 1.1.0, which is what the Go original targets:
| Metric | Result |
|---|---|
| Valid documents | 218 / 218 |
| Invalid documents rejected | 492 / 492 |
| Encoder round trip (parse, encode, parse) | 218 / 218 |
Against the TOML 1.0.0 selection it scores 209/209 valid and 490/499 invalid, and the nine failures are exactly the 1.1.0 features the port implements (optional seconds, \x escapes, inline table newlines and trailing commas). I checked this rather than assumed it: those nine files are precisely the nine that the Go original also accepts. Two implementations failing identically on nine files is not a coincidence, it is a version mismatch, and it is the clearest evidence I have that the version scoping argument is real and not an excuse.
Reporting "710/710" without that explanation would be technically true and actively misleading. If you take one methodological thing from this post, take that.
Layer 2: the oracle, which caught a bug the corpus could not
Conformance says "you match the expected output." Differential testing says "you match the reference implementation." Those are different questions, and the second one is stronger.
I ran both decoders over all 266 valid files and compared the results under toml-test's own comparison rules. Result: 266 identical, zero divergences, zero accept/reject disagreements.
This is the layer that caught the ugliest design mistake in the project. An earlier revision of my port stored datetimes as Value::String and had the test harness recover the type by pattern-matching the text. It passed conformance. It was badly broken: the quoted string "1979-05-27" was being reported as a date-local, because the harness could not tell a datetime from a string that looks like one. The corpus happens not to contain that case.
The fix was structural rather than a patch. Value::Datetime(Datetime) became a real variant with Offset, Local, DateOnly and TimeOnly, and type_tag() moved into the library where it belongs.
Type information must never be reconstructed downstream of the thing that knew it. If your test harness is deriving facts your library already had and threw away, your library has a hole in it, and the harness is quietly papering over it.
Layer 3: the fuzzer, which found four bugs no corpus contains
775 conformance tests is a lot of tests. It is also a fixed, hand-written set that encodes the cases humans thought of. So I built a differential fuzzer: mutate TOML, feed the same bytes to both implementations, and flag any input where one accepts and the other rejects, or both accept and disagree on the value. Formatting differences are not divergences, so it reuses the same comparison module as the conformance runner.
One implementation detail that decides whether this works at all: build the Go oracle once and cache the binary. My first version shelled out to go run, which re-links on every call and caps throughput at a handful of iterations per second. Cached, it does about 450 iterations per second, which is the difference between a useful tool and a decoration.
It found three bugs during the build, none of them reachable from the corpus:
-
Float overflow silently became infinity.
3.14159265358e9793was accepted, because Rust'sstr::parse::<f64>saturates toinfwhere Go'sstrconv.ParseFloatreturns a range error. This is a genuine cross-language trap: two standard libraries, same operation, different failure mode, no compiler warning. -
A bare carriage return could open a line continuation. In
"""\<CR>T"""the backslash-newline handler treated a lone CR as a line ending. A CR only ends a line as part of CRLF. - Keys were lexed as values inside an inline table nested in an array. More on this one below, because it is the six hour story.
And while writing this post I ran the fuzzer again on a seed I had not tried, and it found a fourth in ninety seconds:
odt4 = 1979-05-27 07:32:60Z
Second 60. A leap second. My port accepts it, the Go original rejects it. I think my port is right and the original is wrong, and the evidence is sitting inside the corpus itself: the comment at the top of invalid/datetime/second-over.toml reads time-second = 2DIGIT ; 00-58, 00-59, 00-60 based on leap second rules, and the file only tests :61. RFC 3339, which TOML defers to, permits 60. Go's datetime handling rejects it anyway.
Seeds 1 through 5 were clean over about 50,000 iterations. Seed 7 found it in 40,000. "The fuzzer is clean" is a statement about the seeds you ran, not about your program. I had written the stronger claim in my README. That was overreach, and running one more seed was enough to prove it.
The edge case that ate six hours: what does a comma mean
TOML cannot be lexed context-free. A bare token such as 1.5 is one value in value position and two dotted key segments in key position, so the lexer has to know which side of an = it is on. And the token that makes this genuinely hard is the comma, because what a comma separates depends entirely on the innermost open container. Inside an array it separates values. Inside an inline table it separates key/value pairs.
Between commits 02b6660 (01:23) and 25503cf (07:15) I did nothing else. Just under six hours, one 178 line rewrite of the lexer, and 142 lines deleted from the parser as the logic moved to where it belonged.
What I shipped at 07:15 was a bracket depth counter. If the brace depth is greater than zero, we are in an inline table, so a comma means a new key is coming. It passed everything. It stayed in for the rest of the project.
It was wrong, and this input breaks it:
a = [{ b = 1 }, { c = 2 }]
An array and an inline table are open at the same time. A depth counter produces a number. A number cannot answer "which container am I innermost inside," which is the only question that matters. After the second {, the lexer read what should have been a key as a value, and let an illegal bare key through.
The fix is small and, in hindsight, was the obvious design from the start: track enclosing containers as a stack and ask the innermost one.
Token::Comma => expect_value = stack.last() == Some(&Ctx::Array),
Two things I keep turning over. First, six hours of work produced a solution that was structurally incapable of being correct, and no amount of additional testing against the conformance corpus would ever have revealed it, because the corpus does not contain that shape. Second, when you find yourself collapsing a structure into a counter, you have thrown away the answer to a question you have not been asked yet. Depth is a lossy projection of a stack. I did that at hour six and paid for it two days later.
The decision I would take back
Here is the one I did not see until after everything was green, and it is the most useful thing in this post.
That is_value_position function, the one I rewrote twice? Look at where it gets called:
// inside the main lexer loop, once per bare token
let in_value_position = is_value_position(&tokens);
And look at what it does:
fn is_value_position(tokens: &[TokenWithPos]) -> bool {
let mut stack: Vec<Ctx> = Vec::new();
let mut expect_value = false;
for t in tokens.iter() { // every token. from the beginning. every time.
...
It replays the entire token history from the start, once per token. The lexer is O(n²) and I never noticed, because the correctness work consumed all of my attention and my test corpus could not possibly have shown me.
The comment I wrote above that call site says it checks "if the previous significant token was = or ,." That is what I believed I had built. The code walks the whole vector. The comment describes an O(1) check and sits directly above an O(n) one, which is a nice illustration of how easy it is to review your own intent instead of your own code.
I measured it. Same machine, release build, median of five runs, flat documents of key/value pairs:
| Pairs | File size | As shipped | Incremental | Speedup |
|---|---|---|---|---|
| 1,000 | 10.8 KB | 4.23 ms | 0.67 ms | 6× |
| 4,000 | 49.8 KB | 61.65 ms | 2.79 ms | 22× |
| 8,000 | 101.8 KB | 395.58 ms | 5.88 ms | 67× |
| 16,000 | 217.8 KB | 2,013 ms | 13.59 ms | 148× |
| 32,000 | 447 KB | 8,698 ms | 34.06 ms | 255× |
Per doubling, the shipped version multiplies its time by about 4. The fixed version multiplies by about 2. That is the signature, unmistakable once you look for it.
A 447 KB TOML file takes my "finished" parser 8.7 seconds. The fix takes it to 34 milliseconds and is about thirty lines: keep the stack and the flag as loop state, update them as each token is pushed, instead of recomputing from scratch. Identical state machine, identical semantics. All 710 conformance tests still pass, all 218 valid documents still round trip, and the differential comparison against the Go original is still clean on all 266 files.
Now the part that actually stings. Here is why no test I wrote could ever have caught this:
The entire 266 file valid corpus is 34,104 bytes. Average file size: 128 bytes. Total parse time: 1.75 ms.
Before the fix: 1.75 ms. After the fix: 1.86 ms. The difference is noise.
My test suite is 100% green and cannot see a 255× performance cliff, because conformance corpora are made of tiny documents and real configuration files are not. A Cargo.lock for a large workspace clears 100 KB comfortably. On that input, my port is roughly 400 milliseconds of pure quadratic waste, and every green checkmark I have says it is perfect.
The decision I would take back is not the bracket depth counter. That was a bug, I found it, I fixed it. The decision I would take back is treating "passes the conformance suite" as the definition of done and letting the shape of the test corpus silently define the shape of my quality bar. I optimised for the thing being measured, which is exactly what you would expect a person to do, and the measurement had a blind spot the size of a two-orders-of-magnitude regression.
Correctness suites test correctness. They will let absolutely anything through on complexity.
What I would tell someone starting a port tomorrow
- Port the test runner before the library. It is production code, it is the thing that measures everything else, and its bugs are invisible by construction. Mine was wrong in three ways at once.
- Get an oracle, not an expectation file. Keep the original in-tree and diff against it. It is the difference between "matches what I wrote down" and "matches the thing that actually works."
- Fuzz differentially, and cache the reference binary. Four real bugs, none of them reachable from 775 hand-written tests. Then run more seeds than you think you need, because clean is a property of your seeds.
-
Check the complexity of anything you call inside a loop over the input. Especially anything that takes the accumulated output as its argument. That signature,
f(&everything_so_far)called once per item, is a quadratic tell you can grep for. - Ask what shape of input your test corpus cannot represent. For a parser conformance suite the answer is "large ones," and that answer cost me 255×.
Repository, with the decision log and every number above reproducible via make: github.com/SujalXplores/toml
Top comments (0)