Can't Beat Them... Join them?

Earlier this year I started writing a de-identification tool for clinical text. The idea was simple and a bit stubborn: patient notes should never leave the machine to get scrubbed. A fast rules-and-NER pass catches the obvious identifiers, and a small local LLM handles the sentences that regexes cannot, like "the patient's sister works at the bakery on Elm Street".

I called it localscrub. Three months in, it had a benchmark, a threat model, two architecture decision records, and a v0.2.0 release. On my test set of real medical-transcription prose with planted identifiers, it fully redacted 99.9% of them, against 75% for the standard rules-only baseline. I was proud of it.

Then I ran the same benchmark against OpenMed.

The project I should have found earlier

OpenMed is a local-first healthcare AI project. Clinical NER, HIPAA PII de-identification, 2,200+ medical models, 35 languages, runtimes for Python, Android and the browser. Apache-2.0. Around 5,800 stars. One maintainer who answers issues within days.

It is, in other words, the thing I was building, except with a model zoo, a multilingual pipeline, and a community I did not have.

I found it the way you find most things in open source: by accident. I was adding a flag to localscrub so users could swap the NER checkpoint, and one of the candidate models was OpenMed's 44M-parameter clinical PII model. It was sharp on names. Sharper than the one I was shipping by default. So I went to read the rest of the project, and I kept reading for a while.

That is the moment this post is about. Not the code. The moment where you look at your three months of commits, then at someone else's 5,800 stars, and you have to decide what your project is for.

The honest comparison

My first instinct was to look for the thing I did better. That instinct is useful for about ten minutes and dangerous after that, so I made it measurable instead.

I ran OpenMed's default extract_pii() through the same harness I use for localscrub: 100 public MTSamples transcripts, 697 planted identifiers, same scorer, CPU only. The result surprised me, but not in the direction I expected.

configuration	precision	strict-span F1	NAME strict F1
OpenMed extract_pii(), default	0.62	0.34	0.00
same OpenMed model, two post-processing changes	0.91	0.91	0.96

Recall for names, addresses, dates, phones, SSNs and MRNs was 1.00 in both rows. The model was not the problem. The model was excellent. The post-processing around it was costing it almost thirty points of precision.

Three things were going on, and I only saw them because localscrub had already hit all three:

1. A SWIFT/BIC regex was matching section headers. Any 8- or 11-letter all-caps word qualified, so FINDINGS and MEDICATIONS came back as bank identifiers. In 100 clinical notes that produced 182 false spans. Most of the precision loss was this one pattern.
2. The Safe Harbor policy masked things Safe Harbor does not require. Every age, every gender, every ethnicity, plus clinical concepts like diseases and drugs. The rule only cares about ages over 89. Masking the rest removes text a clinician needs.
3. Spans stayed fragmented. First name and last name as two spans with an unmasked gap between them. Street, city, state and postcode as four. For de-identification, span completeness is the safety property. An address that is three-quarters masked is not masked.

I had fixed the same three bugs in my own tool months earlier. I knew exactly what they looked like because I had the test cases.

The decision

At that point I had two options.

Option one: publish a comparison post. "My tool scores 0.91, theirs scores 0.62." Technically true on the default settings. Also deeply unfair to a project that had a better model than mine and a much larger scope, and that was one post-processing pass away from the same number.

Option two: write up the three causes, with a minimal reproduction and the benchmark, and send it to them.

I picked option two for a boring reason. A hospital evaluating local de-identification tools does not benefit from two of them with different bugs. It benefits from one that works. If the one that works is going to be OpenMed, and realistically it is, then the most useful thing my three months could produce was a precise bug report and the fixes to go with it.

So I opened the issue, with the numbers above, the three causes, and a one-line reproduction for the regex. Then I sent three pull requests.

The maintainer's reply on the first one was short and did more for my week than it should have:

Thanks, @valbarov. Merged into prepare/v3.2.0 for #4369, retaining your attribution.

He also strengthened the regression tests around my change so that the model's own detections survive without banking context. That is the part I could not have done alone. I did not know his test suite. He did.

What I actually learned

A few things, none of them about regexes.

Your eval harness is worth more than your tool. The scrubbing code in localscrub is maybe a third of the repository. The rest is synthetic corpus generation, the MTSamples injection benchmark, and the scorer. That part transferred directly. It found three real bugs in a project with 24,000 tests, because none of those tests asked the question my harness asks: what fraction of planted identifiers come out fully redacted?

"Mine is better" is usually "mine is better on my benchmark." The moment I ran OpenMed's model through my own post-processing, it beat my default model on names. My benchmark was measuring my pipeline, not my model. Those are different things, and it took a competitor to show me which one I had actually built.

Contributing is slower than publishing and better than publishing. The issue took a day to write properly. The PRs took a week each, including rebases and a CI failure that turned out to be an unrelated dependency lockfile. A comparison post would have taken an afternoon and helped nobody.

So what is localscrub now?

This is where I have to be honest with myself in public.

localscrub still exists. It still does the one thing I started it for: the two-stage cascade, where cheap detection handles the bulk and a local LLM handles the narrative remainder that no NER model is trained on. As of the latest release it can also run OpenMed's model as its NER stage with one flag, which felt like the right way to acknowledge who has the better checkpoint.

But I have not merged a feature into it since the day I opened the OpenMed issue. My evenings go to rebasing PRs against someone else's master.

I do not think that is the wrong outcome. I am just not sure it is the right one either.

So, a real question, not a rhetorical one: when you find a bigger project doing what you set out to do, should you keep pushing your own tool? Does it still have a job, as a test harness, as a place to try the ideas the big project is not ready for? Or is that just a way of not letting go?

I would like to hear from people who have been on either side of this. Especially the maintainers.