DEV Community

Cover image for How to check your Rust port against the Python original (with five real bugs it caught)
Srinathprasanna N S
Srinathprasanna N S

Posted on

How to check your Rust port against the Python original (with five real bugs it caught)

You rewrote a Python library in Rust. The benchmarks look great and the test suite is green. One question is left, and it is the one your users care about: does it still do exactly what the old code did?

Your tests can't fully answer that. They check the inputs you thought of. The differences hide in the inputs nobody thought of.

The good news is that a port comes with something most code never has: a complete specification. It's the old code. You don't need to read the Rust or reason about it. Run both versions on the same input and compare what comes back. Repeat until they disagree.

Here is how to do that in practice, and five places it found real bugs.

The idea in one function

def same(input):
    return python_version(input) == rust_version(input)
Enter fullscreen mode Exit fullscreen mode

Everything else is detail. Where do the inputs come from? How do you compare outputs fairly? What if one side raises and the other doesn't?

Animation: the same input goes to Django and to its Rust port; they agree until 'a<b', where Django says 3 and the Rust port says 6

Doing it with equiv

I use equiv, an open-source checker written in Rust. It generates inputs and runs each side in its own Python process. It stops at the first input where they disagree. Every run writes a receipt you can re-run byte for byte, which matters when you hand a bug to a maintainer.

Point it at both versions of a function:

equiv review port.py original.py parse str --errors-agree
Enter fullscreen mode Exit fullscreen mode

--errors-agree counts it as agreement when both sides raise, because a port rarely raises the exact same exception class. One side raising while the other returns is still a difference.

If the port is a new release of the same package, give each side its own environment:

equiv review new.py old.py fn str \
  --reference-python old-env/bin/python \
  --candidate-python new-env/bin/python \
  --versions-of yourpackage
Enter fullscreen mode Exit fullscreen mode

Each side reports which version it actually loaded, so you never compare the wrong build by accident.

A note on limits: this is testing, not proof. A clean run means the two versions agreed on every input it tried, and the output says how many. It works on functions that take and return plain Python values.

Five real examples

1. django-rusty-templates: a panic and a wrong count

django-rusty-templates is a Rust implementation of Django's template engine. Compared with Django's own engine across 27 checks:

  • wordwrap panicked on text starting with a non-breaking space.
  • This one is my favourite:
'a<b'|capfirst|length
Enter fullscreen mode Exit fullscreen mode

(that is, capitalise the string, then count its characters)

Django says 3. The Rust version said 6, because it escaped < to &lt; too early and then counted the extra characters.

Seven differences in all (#587, #593), all fixed in five merged pull requests.

2. LiteLLM: a regression in a release candidate

LiteLLM moved parts of its core to Rust in 1.110. Running the last Python release (1.109.1) against the release candidate, each in its own environment:

  • temperature=1 was suddenly refused for 72 of 405 Claude model names that the previous release accepted (#51394, fixed in #52341).
  • The Rust-backed tokenizer differed from tiktoken on lone surrogates and very large token IDs (#51601, fixed in #51720).

Animation: 405 Claude model names, all accepted on LiteLLM 1.109.1; 72 refused on 1.110.0rc2

3. sqlfluff: feed it SQL its tests never run

sqlfluff ships a Rust parser next to its Python one, and its CI already compares them on its own fixtures. They agree on all 16,442 statements. The trick was to give each dialect's parser SQL written for other dialects, which CI never does. Three parsers split. In one T-SQL statement, Python read the word day as a date part and Rust read it as a column name (#9095, fixed the next day).

Animation: one T-SQL statement; the Python parser reads day as a date part, the Rust parser as a column

4. ua-parser: three engines, one user agent

uap-python has three interchangeable engines, one of them in Rust. For Applebot's desktop crawler, the Python engine said "Mac" and the Rust engine said "Spider" (#334). The maintainer traced it to one rule and wrote a fix (uap-core#675). Running all three engines over the 17,820 user agents in the project's own tests: 9 disagreements before the fix, none after.

Animation: Applebot's desktop user agent gives Mac in the Python engine and Spider in the Rust engine; after the fix, 0 of 17,820 test user agents disagree

5. dulwich: a panic that except Exception can't catch

dulwich is Git in pure Python with a Rust speed-up for sorting tree entries. With names containing a slash, the Rust comparison stopped being a consistent ordering. Below 21 entries it sorted differently. From 21 entries it panicked, and a PyO3 panic is a BaseException, so ordinary except Exception handling doesn't catch it. The maintainer reproduced it and merged a fix the same day (#2582).

Animation: dulwich 1.2.21 tree entries with slashes in their names; the Rust sort gives a different order, then panics at 21 entries

What I'd do if I were porting a library today

  1. Keep the old version runnable until the new one has been compared with it. Once you delete it, you lose your specification.
  2. Test the inputs nobody chose:
    • text that isn't English, including non-breaking spaces;
    • empty values and huge values;
    • inputs from a neighbouring format (another SQL dialect, another user agent family);
    • every input form the API accepts, not just the documented one.
  3. Treat a panic in the port as a bug, even if Python raised too. Python callers can't catch it the normal way.
  4. Decide which side is right before you ship. In the ua-parser case, the Rust engine was the correct one. Either way, your users shouldn't be the ones who find out.

The full write-up, with every issue and pull request linked, is on my site: Running Rust ports against the Python they replace.

If you're porting a library and want a second pair of eyes on it, I'm happy to run it against the original. Say hello in the comments.

Top comments (0)