What Mutation Testing Found in My Rust Framework — and the Two Holes It Couldn't See

Using Runique 3.0.0 or 3.0.1? Upgrade to 3.0.2: both vulnerabilities described here are fixed in it.

Runique is a Django-inspired web framework I've been building in Rust (Axum, SeaORM, Tera). Before tagging 3.0, I wanted to know one thing: do my ~2,300 tests actually check anything?

Green tests prove that the code runs. They don't prove that a test would fail if the code were wrong. So I ran cargo-mutants on the whole framework. It took 33 hours. Then, a few days later, a targeted security audit found two real vulnerabilities that no mutant could ever have revealed.

This post is about both halves.

What cargo-mutants does

It takes your code, changes one small thing (< becomes <=, a function returns Default::default(), a && becomes ||), and runs your tests. If the tests still pass, that "mutant" survived: your tests didn't notice the code was broken.

The full pass on Runique:

Mutants generated	4,538
Caught by the tests	2,512
Survived	728
Didn't compile (ignored)	1,296
Timeouts (infinite loops, counted as caught)	2
Duration	33 hours

728 places where I could break the code and nothing would complain.

The survivors told me uncomfortable things

Some tests asserted nothing. I had tests like this:

let _ = build_router(config);

The router was built, nothing was requested, nothing was checked. Mutants inside the router survived because the test only proved it didn't panic.

A real bug in the DSL parser. In the model DSL, [max_size: 500KB] was read as 500 MB, and 5GB as 5 MB. The unit glued to the number became the literal's suffix and was misread. No test had ever checked a unit other than MB. A mutant in the size calculation survived, and following it led straight to the bug.

A whole crate was never tested. After the run, two of my three crates showed 306 out of 306 mutants as "didn't compile". Not one mutant had been tested in them. My config file forced features that cargo refused for those packages. 100% unviable on a crate is not a result, it's a broken setup. I now check that number first.

My own verification lied to me once. To check a fix by hand, I applied a mutation, ran the test, and restored the file with a copy that preserved its old timestamp. Cargo saw an old date, considered the binary up to date, and kept testing the previous mutation. Always touch the file after restoring it.

I went through the survivors one by one, then re-checked roughly 330 mutations by hand after the second pass. Each new test was checked against its mutation: it had to fail with the mutation and pass without it. In the end, the suite went from about 2,300 to about 2,800 tests.

At that point I felt good. Every condition was tested on both sides, every survivor explained.

Then the audit

A few days later, I reviewed one flow from start to finish instead of one function at a time: what happens to a multipart form, from the first byte to the saved file?

Hole 1: the file was written before the CSRF check

Runique streams uploaded files to a staging directory while it parses the multipart body. The CSRF token was checked after parsing, before committing the files to their final place.

So a cross-site request with no valid token was refused, but its files had already been written to disk. Worse: that staging directory sat under the media root, and media was served. Combined with another detail (a file part sent under a text field's name put the staging path into a re-displayed value), an anonymous visitor could upload an HTML file and get it served from my domain. A stored XSS, with no account.

The fix is a gate: no file is written until a valid token has been seen (simplified):

let mut csrf_checked = csrf_session.is_none();
// ...
if is_file_part && !csrf_checked {
    return Err((StatusCode::FORBIDDEN, t("csrf.invalid_or_missing")).into_response());
}
// ...
if name == CSRF_TOKEN_KEY && token_matches(&text, session) {
    csrf_checked = true;
}

In practice this means the csrf_token field must come before any <input type="file"> in the form, which Runique's form renderer already does. On top of that, paths with a segment starting with . now return 404 under media, and media responses get their own CSP (script-src 'none').

Hole 2: the @ in a URL

The open-redirect protection extracted the host from a redirect target and compared it to the allowed hosts. The extraction cut at the first /, ? or #.

Now look at this URL:

https://localhost:[e-mail]/

Everything before the @ is credentials, not the host. A browser goes to evil.com. My code read localhost:[e-mail], saw localhost, and let it through. The fix is one filter:

.filter(|h| !h.is_empty() && !h.contains('@'))?;

Why no mutant could see these

This is the part I keep thinking about.

Mutation testing checks the code that exists. It can't check code that doesn't exist.

There was no line saying "check @ in the authority", so there was nothing to mutate. There was no line saying "check the token before writing", because the check simply happened later. Every line was tested. The order was wrong, and a rule was missing.

Mutants answer: "would my tests notice if this code changed?" They don't answer: "is this the right code?"

What would have found them:

• Thinking in flows, not functions. "When is the file written? When is the token checked?" is a question about order. Unit tests on each function never ask it.
• Comparing with the real parser. Browsers follow the WHATWG URL standard. A hand-written host extraction will always lag behind it. A property test that compares my extraction with the url crate on thousands of generated URLs would have caught the @ case.
• A checklist per attack type. For each feature: is the check done before the effect? The OWASP ASVS asks exactly that kind of question.

What I'm keeping

• Mutation testing is worth it. It found tests that tested nothing, a real parser bug and a broken setup. I'll run it again, incrementally with --in-diff.
• 100% of mutants caught means "my tests protect the code I wrote". It does not mean "my code is secure".
• Read your flows end to end at least once, and ask "what's missing?" instead of "is this line right?".

Both fixes are in Runique 3.0.2, with tests that fail without them. The full list is in the CHANGELOG and SECURITY.md.

If you maintain a web framework or a library that handles uploads or redirects: go check where your CSRF check happens relative to your first disk write. Then try a user:pass@ URL on your redirect guard.

---

Runique: GitHub · runique.io · crates.io