When Fine-Tuning Stops Working: The Architecture Lesson That Took Our Healthcare AI From 60% to 92% Recall

Vaibhav Tekam — Wed, 08 Jul 2026 16:16:22 +0000

We had a healthcare claims platform stuck at 45% recall on medical coding. Not 45% on a benchmark. 45% recall on CPT and ICD code assignment, in a system whose job is getting those codes right before a claim goes out. Get it wrong, and a claim gets denied. A billing team spends an afternoon on the phone with a payer instead of doing their actual job.

This post is about the architectural decision that moved CPT recall from 45% to 92%, and why no amount of prompt engineering or fine-tuning could have produced the same result on its own.

Why fine-tuning stopped working

When a number is low, the obvious move is a bigger model. We made that move too, and it wasn't a bad instinct, at least at first.

We spent real weeks on it: sharper prompts, more examples, better retrieval so the right context sat in front of the model when it made a call. Recall climbed out of the 40s into the high 50s, low 60s. Genuine progress.

Then, somewhere in the low 60s, it stopped. More examples didn't move it. Sharper prompts didn't move it. The usual levers had stopped answering.

That's the moment worth paying attention to: not when something fails outright, but when it quietly stops responding to the thing that used to work.

Why the LLM could never solve this alone

A language model doesn't return an answer. It returns a best guess from a distribution of plausible ones. Ask it the same claim twice, worded slightly differently, and it can produce two different codes. For a chatbot, that variance is harmless. For a system deciding whether a claim gets paid, it's a serious problem, and it's not a problem a bigger model fixes, because it's not a knowledge gap. It's what a probabilistic system fundamentally is.

A large part of "getting the code right" turned out not to be a judgment call at all. It was already a solved, published, deterministic rule. NCCI, the National Correct Coding Initiative, is a set of edits from CMS stating plainly which procedure code pairs cannot be billed together, full stop, unless a specific modifier applies.

We were asking a model to guess its way toward an answer that already existed as a lookup table.

Here's a concrete version of the failure, details altered but the shape real. A clinical note describes a procedure and, in the same visit, a closely related diagnostic test. The model proposes codes for both, and it's a reasonable read of the note. A careful human coder might make the same call on a first pass.

NCCI disallows the pairing. Not because the clinical reasoning is wrong, but because CMS has already ruled that one procedure is, by definition, a component of the other when performed in the same session. The model has no way to know this from the note, because the constraint isn't in the note. It's in a separate regulatory table with no relationship to what the sentence says.

Multiply that one example by every plausible-looking pair across a real coding surface, and the plateau makes sense. The model wasn't getting worse at reading. It had run out of room to improve on a problem that was never primarily a reading problem.

Building a deterministic validation layer

Once the shape of the problem was clear, the fix stopped being about the model and became about building the layer next to it.

We loaded the full NCCI edit set. All 1.73 million pairs, not a representative sample, because the value of a deterministic check is that it has no blind spots. A rule that catches 90% of bad pairs and misses the rest hasn't solved the problem, it's relocated the failure to wherever nobody's watching.

The resulting pipeline:

The model proposes candidate codes. It's genuinely good at this: reading unstructured clinical documentation and surfacing plausible candidates is a real strength, and there's no deterministic substitute for it.
Every candidate pair is checked against the NCCI table before it reaches a claim.
Only pairs that clear the table ship. A candidate the model favored but the table rejects doesn't go through, no exceptions, because exceptions quietly rebuild the guessing problem the check was meant to remove. The model proposes. The table proves. Once we stopped treating this as one system to keep tuning and started treating it as two systems doing two different jobs, the remaining work was engineering, not research.

It didn't ship clean

The first version of the lookup was a linear scan through all 1.73 million rows per candidate pair. In a demo with one or two claims, it felt instant. In production, with a dozen candidate codes per claim and every pairwise combination needing a check, it visibly lagged. We rebuilt it as a proper index keyed on the pairs themselves. Unglamorous, and the difference between a check that's correct in principle and one that's usable in a real-time pipeline.

The second issue we found before it became an incident, not after. CMS updates the NCCI edit set every quarter. A static, load-once table is a stale table waiting to happen. Rather than wait for a batch of claims to quietly clear an outdated check, we went looking for the failure mode on our own schedule: what happens to a "deterministic" layer when the thing it's being deterministic about changes underneath it. The fix was a scheduled refresh and a version stamp on every verdict, so any decision can be traced back to exactly which edit set produced it. Finding that gap by design, rather than by incident report, is the same principle as the rest of this build: check before you're forced to.

Recall on CPT went from the low 60s to 92%. ICD landed lower, at 76.8%, and that gap is worth explaining rather than smoothing over. NCCI's rules live on the procedure-code side; that's the layer with a clean, published table. ICD diagnosis coding runs on a different kind of logic, driven by clinical specificity and payer documentation requirements, much of which is a genuine judgment call rather than a lookup. Judgment calls are exactly where a probabilistic system should be doing the work, not where a rulebook should be forced onto it that doesn't exist. Read side by side, the two numbers aren't "better here, worse there." CPT had more of the kind of problem a table solves. ICD has more of the kind that needs judgment. The recall gap is largely a measurement of that ratio.

The real lesson

Model accuracy has a ceiling, and the ceiling usually has nothing to do with the model.

Some percentage of what a system needs to get right isn't a pattern to learn. It's a rule that already exists, published and unambiguous, waiting to be wired in rather than rediscovered through enough training examples.

The climb from 45% to the low 60s was a model problem, and the standard tools solved it. The climb from the low 60s to 92% was an architecture problem, and no amount of prompting was ever going to close it. On a dashboard, the two climbs look identical. They are not the same problem, and the second one doesn't yield to more of what fixed the first.

So before reaching for a bigger model on a number that won't move: is this a guessing problem, or is it a problem that already has a correct answer sitting somewhere, waiting for someone to stop asking a probabilistic system to guess at it.

More often than expected, it's the second one.

Production AI systems become reliable when probabilistic models and deterministic systems each do the job they were designed to do.

Most Engineering Blogs Show You What Worked. We're Going to Show You What Broke.

Vaibhav Tekam — Wed, 08 Jul 2026 07:18:23 +0000

Most engineering blogs focus on the outcome. They present clean architectures, successful deployments, and polished case studies that make complex systems appear simpler than they really were.

What they rarely document is the engineering journey that produced those outcomes. They seldom discuss the migration that stalled halfway through, the production incident that forced a redesign, or the assumptions that proved to be wrong once the system met real users and real workloads.

Those experiences contain the lessons that matter most.

At Tech4Biz Solutions, we have spent years designing, building, and restoring production systems across artificial intelligence, cloud infrastructure, enterprise software, embedded hardware, industrial automation, and observability. Regardless of the technology involved, we have reached the same conclusion time and again.

Reliable systems are rarely the result of perfect technology. They are the result of thoughtful engineering, continuous iteration, careful trade-offs, and decisions made under real-world constraints.

Why We're Writing

We started writing because the most valuable engineering lessons rarely appear in traditional case studies.

Case studies usually explain what worked. They seldom explain what failed, what assumptions proved to be incorrect, what designs had to be abandoned, or why the final system looked very different from the original plan.

We're less interested in presenting finished architectures than in explaining how those architectures evolved into reliable production systems.

What We'll Write About

Every article will be grounded in engineering work we've actually done, systems we've actually built, or production problems we've actually solved.

We'll write about:

Production AI systems
Cloud architecture and distributed systems
Platform engineering and observability
Embedded systems and hardware integration
Industrial software and automation
Performance, scalability, reliability, and security
Engineering decisions made under real deadlines and real business constraints

We'll explain not only what we built, but also why we built it that way, what alternatives we considered, and what we learned in the process.

What You Won't Find

You won't find toy projects designed to demonstrate a framework.

You won't find success stories with the difficult parts edited out.

You won't find articles written simply because a technology happens to be trending.

You won't find sales pitches disguised as engineering articles.

Good engineering earns trust through transparency, not marketing.

What's Coming Next

Over the coming weeks, we'll share deep dives into production systems we've built and stabilized across cloud infrastructure, enterprise AI, observability, embedded hardware, and industrial software.

We'll also share the engineering behind Suryamon, our open-source solar plant monitoring platform, along with the architectural decisions, trade-offs, and lessons that shaped it.

The finished system is only part of the story. We're here to document everything that happened before it became reliable.

If these are the kinds of engineering problems you enjoy solving, we hope you'll follow along.

DEV Community: Vaibhav Tekam