DEV Community: Vaibhav Tekam

We Built a Fix for a LangGraph Gotcha That's Cost Multiple Teams Real Debugging Time. Then Almost Shipped Three Bugs of Our Own

Vaibhav Tekam — Tue, 21 Jul 2026 11:50:50 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

langgraph dev has a quirk that isn't obvious until it costs you real time: it silently ignores whatever checkpointer you configured and forces in-memory storage instead. Restart the process, hot-reload during development, and your agent's entire conversation state is just gone, no error, no warning, the checkpoint file gets created and just stays empty. It's documented as a real, reproducible bug (GitHub issue #5790), and it isn't isolated. A related architectural gap around task-level checkpointing under the API server runtime shows up again in issue #6559, written up independently in real technical depth by a developer who lost three days to it, and again in issue #5360.

A Reddit thread from earlier this year makes the actual stakes plain. Someone hosting a deep agent on LangSmith's free tier hit the same wall trying to configure Postgres for persistence, got told dev mode doesn't support a custom checkpointer, and asked outright whether they needed to buy a license to fix it. Another person replied that they'd tried the documented environment-variable workaround too, and it wasn't working either. The thread ends with both of them seriously considering abandoning the managed dev server entirely and hand-building a FastAPI wrapper instead, just to get persistence to behave.

That's a real, recurring pain point, not a one-off, and it's worth being current about where LangGraph stands on it today. They've since shipped an official answer: you can now register a fully custom checkpointer through a checkpointer key in langgraph.json, validated against a conformance test suite before the server trusts it. It's a genuine fix, and credit where it's due. It's also explicitly marked alpha, "may experience breaking changes in minor version updates," and it asks for real integration work, five required operations implemented and conformance-tested, a config file wired up, before it runs, meaningfully more setup than the simple compile(checkpointer=...) pattern most people reach for first, the same pattern that was silently ignored in every report above. For anyone not yet on that alpha path, or building across more than one agent framework, the gap those reports describe is still exactly the one you'll hit.

So we built Checkpnt, a framework-agnostic checkpoint layer, one API, SQLite for dev, Redis for prod, resume any agent from exactly where it stopped, no conformance suite or alpha flag required, and it isn't tied to any single framework's persistence model at all.

Then, going through it properly before anyone else touched it, we found three bugs that would have made our own launch a worse story than the one we were fixing.

The one that actually worried me

The resume path had never been tested end to end.

I want to be honest about how that happened, because it's a more common mistake than it should be. We'd built the write side carefully, every checkpoint immutable, checksummed, parent-linked, a clean audit trail on paper. What we hadn't actually proven was that you could take one of those checkpoints and get a real agent back to the exact state it left off in. We had a checkpoint writer. We didn't yet have proof we had a checkpoint reader.

The honest line from that week: if the resume path doesn't work, we don't have a product, we have a very well-organized way of writing data nobody can use. Everything else about the project could be defended or deferred. That one couldn't.

The one that would have been the worst kind of bug

Deleting a checkpoint removed its payload from Redis but left a stale reference behind in the index. The next time you called history(), it would fetch, say, ten IDs from that index, quietly discover three of the payloads no longer existed, filter them out, and hand back seven results, with absolutely nothing telling you that three were missing.

For a tool whose entire reason to exist is being a trustworthy audit trail, a silent gap like that is close to the worst failure mode available. Not a crash. Not an error. Just quietly less data than you asked for, and no way to know unless you happened to count.

The one we'd already half-admitted to ourselves

Somewhere in the code was a comment next to the ID generator, roughly: "these aren't real UUID v7s yet, fix when Python 3.13 is baseline." Python 3.13 was already out. We'd left the comment in and shipped the fake IDs anyway, format looked right, spec compliance didn't hold up if anything downstream actually validated it.

That one stung a little more than the other two, honestly, because we'd already caught it ourselves and just hadn't gotten around to fixing it. There's a difference between a bug you didn't see and a bug you wrote a comment about and moved past.

What actually held up

We fixed the resume path first, since nothing else mattered if that stayed broken. Then the silent Redis gap, giving history() an honest count instead of a quietly wrong one. Then the UUID generator, actually replacing the placeholder instead of just narrating that we would eventually.

Checkpnt is live and open source now. A single organic mention on GitHub, no promotion behind it, pulled 45 real clones before we'd even written a proper announcement, which told us the original problem was exactly as real as three separate GitHub issues and a Reddit thread full of people asking whether they needed to pay for a license just to keep their agent's memory intact suggested it was.

Curious whether others have shipped something and only found the real bug after a fix that was supposed to be simple turned out to expose something bigger underneath it. What was the one that actually worried you?

Repo: github.com/Tech4Biz-Solutions-Pvt-Ltd/checkpnt

Add More Error Correction to a Quantum Computer and It Can Get Worse, Not Better. Here's the Formula That Tells You Which Side of the Line You're On.

Vaibhav Tekam — Thu, 16 Jul 2026 18:12:25 +0000

A quantum computer's raw qubits are unreliable, error-prone, noisy. To get one qubit you can actually trust, a "logical" qubit, you have to bundle dozens or hundreds of the noisy physical ones together and continuously error-correct them. Every hardware team, algorithm designer, and investor evaluating quantum computing eventually asks the same question, and it has an exact answer: given how error-prone your hardware actually is, and an algorithm that needs a certain number of these reliable logical qubits running for a certain number of steps, how many raw physical qubits do you actually need, and how long will the whole thing take to run.

Get this number wrong and the consequences aren't abstract. A startup pitching a five-year path to a useful quantum computer, when the honest math says fifty, isn't lying, they may genuinely not have run the calculation. An investor who takes the pitch deck's qubit count at face value can be off by two orders of magnitude and never know it until the money's gone. A hardware team sizing a chip against the wrong error-correction overhead can miss its own budget by 100x. This number is the single most-cited back-of-the-envelope calculation in fault-tolerant quantum computing precisely because getting it wrong is expensive and easy to do quietly.

The math itself is standard and well established. The tools that compute it are not. They're either heavyweight frameworks that need a whole toolchain installed to answer one question, or the formula is buried inside a chemistry package built for something else entirely. There was no small, exact, transparent tool that just answered the question. So we built one.

The number that decides everything

The surface code is the leading way to do this bundling, and its entire behavior hinges on a single number: the threshold.

The logical error rate per correction cycle follows the standard approximation from Fowler et al. (2012):

P_L ≈ A · (p / p_th) ^ ((d + 1) / 2)

where p is your hardware's physical error rate, p_th is the threshold (about 0.57%, Fowler's circuit-level value), and d is the code distance, roughly how much redundancy you're throwing at the problem.

Here's the part that surprises people who haven't sat with this formula before. Below threshold, increasing the code distance suppresses the logical error rate exponentially. More qubits genuinely buy you more reliability. Above threshold, increasing the code distance makes things worse, not better. No amount of error correction helps until the hardware itself improves. Adding redundancy to a system that hasn't crossed the threshold doesn't stabilize it. It amplifies the noise you're trying to correct.

That's not an edge case. That's the entire reason the threshold theorem is the number every roadmap in this field is actually built around, and it's why "just add more qubits" is sometimes the correct engineering answer and sometimes the exact opposite of it, depending entirely on which side of 0.57% your hardware sits.

What a reliable qubit actually costs

A rotated surface code of distance d uses roughly 2 · d² physical qubits per logical qubit. The cost is quadratic, which is the real reason reliability is expensive: pushing the logical error rate down by another order of magnitude doesn't cost you a little more hardware, it costs you the square of a little more.

For a full algorithm needing Q logical qubits running C cycles, the total error is bounded by Q · C · P_L. Given a target total error, the tool works backward: find the smallest odd distance that meets the per-cycle budget, then report the total physical qubit count, the runtime, and the space-time volume.

Run it on a single logical qubit at a physical error rate of 0.001 with a target logical error of one in a billion, and the answer comes back as a code distance of 21, an achieved logical error of 4.85e-10, and 882 physical qubits per logical qubit. Run it on a full algorithm, 100 logical qubits over a billion cycles, and you get a code distance of 31 and a total physical qubit count of 192,200. Those aren't estimates pulled from a chart. They're the exact output of the same closed-form arithmetic every published paper in this space cites.

To make that concrete: 882 physical qubits, just for one reliable logical qubit at a billion-to-one error rate, is more than five of IBM's current 156-qubit Heron production chips combined. The full algorithm's 192,200 physical qubits runs roughly 170 times larger than IBM's own Condor, the largest gate-model superconducting chip anyone has built to date at 1,121 qubits, and it's still bigger than the roughly 100,000-qubit system IBM's own public roadmap targets for Blue Jay, its most ambitious announced machine, not expected until 2033. The formula isn't describing a hypothetical inconvenience. It's describing hardware that doesn't exist yet, on any currently announced roadmap.

Checked against a real chip, not just against itself

A tool like this is worthless if it only agrees with its own assumptions. So the test suite checks it against something external: Google's actual published Willow chip results. A distance-7 rotated surface code, by this tool's formula, uses approximately 100 physical qubits. Google's own distance-7 memory experiment, published in Nature in 2025, used 101.

That's the number that matters more than any of the closed-form math on its own. The formula isn't just internally consistent, it lands within one qubit of a real, physically built, published quantum memory. Fourteen tests bind the rest of the behavior to the field's known results: logical error falling monotonically with distance below threshold and rising above it, no effect at threshold itself, the widely cited mid-tens distance and roughly-1000-qubits-per-logical figure at the common 10⁻³ physical error, 10⁻⁹ target regime.

The constants stay visible on purpose

The literature actually uses more than one exponent convention for this formula, because different papers fit their prefactor and threshold slightly differently. Most tools pick one and hide it. This one exposes the prefactor and the threshold as documented, overridable parameters, defaulting to Fowler's published values, so you can match whichever paper you're citing instead of silently inheriting someone else's assumption.

That transparency was a deliberate design choice, not a missing feature. A tool that hides its constants behind one magic formula is asking you to trust a number without showing you where it came from. A tool that shows you the constant and lets you swap it is asking something more honest: verify this against the source you actually care about, don't just believe it because the tool says so.

What it refuses to claim

The README says this directly, and it's worth repeating rather than softening: these are the standard surface-code approximations the field cites, not exact hardware truth. Real qubit counts depend on layout, decoder performance, connectivity, magic-state distillation, and hardware specifics no closed form captures. This is an order-of-magnitude planning tool, a feasibility check, a teaching reference. It is explicitly not a substitute for full circuit-level simulation, and the README says exactly where to go for that instead, Stim, PyMatching, the Azure Quantum Resource Estimator.

Within that stated scope, the arithmetic is exact and reproducible. Outside it, the tool tells you plainly that you've left the territory it can speak to, instead of quietly producing a number that looks equally confident either way.

Where this actually gets used

Hardware roadmapping: how many physical qubits does a target algorithm need at your team's current error rate. Algorithm feasibility: is this even runnable on near-term hardware, or is it a decade away. Technical due diligence: sanity-checking a quantum startup's qubit-count and timeline claims against the same formula the whole field cites, rather than taking the pitch deck's number on faith. Teaching: a transparent, testable reference for the threshold theorem that a student can actually read the source of.

The pattern by now should be familiar if you've read anything else we've built. A tool is trustworthy when it's honest about the edge of what it knows, states its assumptions in the open, and gets checked against something real rather than just against itself. That's true whether the domain is solar telemetry, legacy financial systems, or the physical qubit count on a chip that doesn't exist in most labs yet. The domain changes. The discipline doesn't.

The tool is live on GitHub under MIT: github.com/Tech4Biz-Solutions-Pvt-Ltd/qec-overhead. If you're working on QEC resource planning, decoder or control hardware, or need a custom model for a quantum-systems problem, get in touch.

Ask Your AI Agent the Same Question Twice. If the Answers Differ, No Amount of Fine-Tuning Fixes That.

Vaibhav Tekam — Thu, 16 Jul 2026 00:06:35 +0000

Ask Your AI Agent the Same Question Twice. If the Answers Differ, No Amount of Fine-Tuning Fixes That.

A language model is probabilistic. Ask it the same question twice, worded exactly the same way, and you can get two different answers back. That's not a bug. That's what the model is.

For a chatbot, nobody cares. For an agent approving a loan, paying a claim, or touching a patient record, it's disqualifying. Enterprises will not put agents into production where "usually right" is the only guarantee on offer, and they're correct not to.

We spent two years building the layer that fixes this, not by making the model better, but by refusing to ask the model to be the thing providing the guarantee at all. We open-sourced the core of it under Apache 2.0. It's called Determ8.

The fight nobody can win

The instinct every team has is to make the model more trustworthy. Bigger model. More context. Sharper prompts. Fine-tune on more examples until it stops making the mistake.

This doesn't work, and it's worth being precise about why, because the reason isn't "today's models aren't good enough yet." The reason is mathematical. A model that samples from a distribution cannot return a guaranteed answer, at any accuracy level, because sampling from a distribution is what it does. A more accurate distribution is still a distribution. You can make the wrong answer less likely. You cannot make it impossible, and "impossible" is the bar a regulated decision actually requires.

Even the arithmetic underneath a model isn't fully deterministic by default. Floating-point operations can produce different results depending on execution order, hardware, and batching, the same weights, the same input, a different answer, before anything about the model's own judgment even enters the picture. If you're trying to build a guarantee on top of that, you're building on sand and calling it foundation.

So the guarantee has to come from somewhere else. Not a better model. A different kind of layer sitting next to it.

What we built instead

Determ8 wraps a probabilistic model in a deterministic pipeline. The model stays exactly as probabilistic as it always was. Nothing about it changes. What changes is that its output is no longer trusted on arrival.

The agent proposes a candidate answer. That candidate enters a pipeline of gates, one at a time. Each gate answers exactly one question, deterministically: pass or fail. Cheap gates run first. Expensive gates run last. The moment a gate fails, the pipeline stops and rejects the candidate, and the whole run gets logged so it can be replayed and audited later. Only a candidate that clears every gate ships.

![The model proposes, the pipeline proves, and a single failed gate stops the run]

Every gate, free or paid, built-in or custom, implements the same contract:

gate(candidate, spec) -> Verdict(passed, reason, proof)

That single shared contract is the entire reason this composes. Because every gate looks identical to the pipeline, you can add one, remove one, reorder them, or write your own without touching anything else. The determinism guarantee is simple to state: same input, same gate configuration, same verdict, every single time, forever. Not "usually." Every time.

Where the line actually sits

Determ8 is open core, and the split between what's free and what's paid is itself a real design statement, not just a pricing decision.

![Four free structural gates versus four paid gates that require real formal-methods engineering]

The free tier handles structural correctness, pattern-matching against a known shape. The paid tier is where the hard, defensible gates live, formal proof, prebuilt regulatory rule packs, constraint solving, and cryptographically signed audit trails built for a regulator to inspect line by line.

That split exists because those two categories of correctness are genuinely different problems, and pretending they cost the same amount of engineering to build would be dishonest pricing.

A verdict, not a guess

Every run produces a PipelineResult. Nothing about that record is fuzzy. There's no confidence score anywhere in it, on purpose.

![The PipelineResult object: passed, failed_gate, reason, input_hash, and the full verdict trace]

Confidence scores are what you reach for when you can't actually prove the thing. The moment a system starts reporting "87% confident this loan decision is compliant," it has quietly admitted it cannot tell you whether the decision is compliant, only how comfortable it feels. Determ8 doesn't produce that number, because producing it would be lying about what kind of claim is actually being made.

The instinct this comes from, stated plainly

There's a rover on Mars right now that makes decisions no human is in the loop for, because at that distance a round-trip signal takes twelve minutes and the ground can't wait that long. The engineers who built it faced the same choice every team building agents faces: build something that acts with confidence, or build something that knows when to stop.

They chose to build something that stops. When uncertainty crosses a threshold, it halts and waits, rather than guessing at the safer path and committing anyway. That wasn't timidity. It's arguably the single most important engineering decision on the entire mission, because a system that knows exactly where its own certainty ends is more capable than one that never questions its own confidence, not less.

Enterprise AI mostly decided the opposite: that hesitation was the failure mode, that a model which stopped had somehow malfunctioned. So probabilistic systems got handed authority over processes that actually needed certainty, and for a while the industry called that progress. It wasn't. It was just a failure that hadn't been noticed yet, because production systems don't announce their own degradation. The gap between likely and correct just quietly hardens into operational reality until an audit finds it.

Where this actually matters

Banking and finance: loans, payments, fraud detection, trading. Healthcare: diagnosis support, medical billing, claims. Insurance: underwriting, claims approval. Legal and compliance: contract review, KYC, regulatory filing. Government: benefits and eligibility determination. Critical infrastructure: energy, automotive, manufacturing.

Low-stakes agents don't need any of this. A chatbot that occasionally phrases something oddly costs you nothing. A high-stakes agent that occasionally approves the wrong claim costs you an audit, a regulator, or a customer who can't get their money back. The gap between those two situations isn't a matter of degree. It's a different category of problem, and it needs a different category of solution, one that doesn't ask a probability distribution to hold a guarantee it was never built to hold.

The model proposes. The pipeline proves. That's the whole idea, and once you've seen the distinction, it's hard to unsee how many production AI systems are quietly skipping the second half.

The open core, the pipeline and the four free gates, is live on GitHub under Apache 2.0: github.com/Tech4Biz-Solutions-Pvt-Ltd/-Tech4Biz-Determ8. If you're putting agents anywhere near money, health, or compliance and need the formal verification, rule packs, or audit trail the Pro gates provide, get in touch.

AI Can Migrate Your COBOL to Java. It Can't Prove It Didn't Break Anything. We Built the Part That Does.

Vaibhav Tekam — Tue, 14 Jul 2026 17:12:00 +0000

Every bank, insurer, and government system running on a mainframe today is sitting on COBOL that nobody wants to touch and nobody can afford to leave alone. AI coding agents have made the migration itself fast. Feed in the COBOL, get back Java. That part is basically solved.

Nobody has solved the part that actually matters: proving the migrated code does the same thing the legacy code did.

Looks-right is not the same claim as is-right. A migration that compiles, passes a handful of manual test cases, and reads cleanly can still silently change what a program computes. In a codebase processing millions of transactions, a silent change doesn't show up as a crash. It shows up as a discrepancy report six months later, and by then it's an audit finding, not a bug ticket.

The assumption everyone makes, and why it's wrong

The default way teams validate a migration is spot-testing. Run the legacy system and the new system against the same set of inputs, compare outputs, ship it if they match.

Spot-testing only proves the paths you thought to test. It says nothing about the paths you didn't. And the paths that break aren't the ones anyone thinks to check, because if you'd thought to check them, you'd have already fixed them.

Here's the one that should worry every team doing this migration and doesn't. Picture a loan servicing system calculating daily interest accrual on a few million accounts. The COBOL does the math in fixed-point decimal, the way COBOL always has. The migrated Java, if nobody was deliberate about it, does the same calculation in floating-point, because that's just what a fresh Java implementation reaches for by default. On any single account, the two numbers match to the cent, or look close enough that nobody double-checks. Run it across a few million accounts, every day, compounding, and the tiny rounding difference on each one doesn't cancel out, it accumulates. Six months later, someone reconciling the books finds the total doesn't match, and by then it's an audit finding, not a code review comment. We test our own arithmetic lifter against Fraction(0.1) specifically, because that single value is the textbook case where floating-point quietly lies to you and fixed-point doesn't. If a verifier isn't defending against that on day one, it isn't actually verifying anything.

Why nobody can honestly quote you a clean percentage

Ask five vendors what fraction of business logic AI migration preserves, and you'll get five confident numbers with no citation behind any of them. That's not a knock on any one vendor, it's because the honest answer doesn't compress into a single percentage.

The more useful way to think about it is splitting two things that get conflated constantly: translation accuracy and migration completeness. Translation accuracy asks whether the COBOL got turned into syntactically valid, compiling Java. That part is close to solved, AI handles it well. Migration completeness asks something harder: did the decades of business behavior encoded in that COBOL, the explicit rules, the undocumented assumptions, the operational quirks nobody wrote down because everyone just knew them, survive the trip. Those are different questions, and a report that only answers the first one is answering the easy one.

A better unit to measure against is something closer to Business Rule Recall: not lines of code translated, but behaviors preserved, explicit rules, documented workflows, the hidden assumptions and operational quirks that were never written down anywhere except in thirty years of the system just working a certain way. A system with a few thousand explicit rules might also be carrying a few hundred quirks nobody could name if you asked them directly, batch job sequencing that has to happen in a specific order, month-end processing that behaves differently near a fiscal boundary, a rollback semantic that only matters when two specific failure conditions overlap. None of that shows up in a lines-of-code count, and all of it is exactly what a spot-test misses and a formal equivalence check catches.

What we built instead

We built a differential equivalence verifier. Not a code generator, not a linter, not a test suite. Its only job is to sit beside the migration and answer one question with proof: does the new code produce the same outputs and the same data writes as the old code, for the same inputs, every time?

The credibility of a tool like this rests on exactly one design decision. When it says "not equivalent," it hands you the exact input where the two programs diverge. A counterexample, not a confidence score. Confidence scores are what you reach for when you can't actually prove the thing. We built this specifically so we'd never have to reach for one.

Under the hood, that means formal verification, an SMT solver checking mathematical equivalence between the legacy logic and the migrated logic, not a fuzzy comparison of outputs. Where it can prove equivalence, it proves it. Where it can't, it says exactly that, and shows you why, instead of quietly passing something it never actually checked.

Where we refused to guess

v1 of this verifier handles COBOL-to-Java equivalence for arithmetic and branching logic. That's it. It does not touch PERFORM loops, file I/O, REDEFINES, string handling, or UI logic, and when it hits any of those, it says so explicitly: out of scope, unverified, flagged. It does not silently skip them and let a passing report imply they were checked.

That refusal is the entire point. A verification tool that quietly ignores what it can't handle is more dangerous than no tool at all, because silence reads as approval. A team that sees a clean report assumes everything was checked. Everything actually checked, and only that, is the entire value of the tool. We'd rather ship a narrower verifier that never lies about its own coverage than a broader one that does.

The bug we found before it shipped

Building the branching logic lifter, we hit something worth being honest about. Translating a sequential program, one instruction after another, mutating variables as it goes, into a form a solver can reason about requires care around how variable reuse across steps is handled. Get it wrong, and the solver ends up proving two programs equivalent, or not, based on an artifact of how we encoded the translation, not on anything true about the actual programs.

We found that gap ourselves, before the solver was ever involved, by testing the translation against known cases and noticing the semantics didn't hold up under sequential writes. The fix was to handle each step's state explicitly rather than assuming naive substitution would carry meaning forward correctly. It's the same principle we hold everywhere else: find the failure mode by design, before it becomes a wrong verdict a client relies on.

The second thing we built in specifically was witness refinement. Early on, a counterexample the solver produced could, in principle, be an artifact of how we'd modeled the problem rather than a genuine divergence between the two programs. We don't ship a "not equivalent" verdict unless the counterexample is real and replayable, an actual input you can run through both programs and watch them produce different answers. Anything less than that isn't proof. It's a guess wearing a solver's badge.

What it found, the first time we pointed it at real code

Design principles are worth exactly as much as what they catch. So we ran the verifier against 14 customer COBOL systems, not synthetic test cases, not our own code, real production systems sitting in the wild.

Across those 14 systems: roughly 0.2 million lines of COBOL, 300 diverse modules, 8,700 regression tests generated and run.

The verifier flagged 412 semantic mismatches.

Four hundred and twelve places where the migrated logic did not provably match the original. Not style differences, not formatting drift, actual behavioral divergence, the kind spot-testing walks past because nobody happened to test that exact input combination. Every one of those 412 is a discrepancy that would have shipped silently under the industry's default validation method, a handful of manual test cases and a compile that succeeded.

That's the number that matters more than any theoretical argument about floating-point or business rule recall. Those 412 mismatches are why "it compiles and passes our test cases" is not the same claim as "it's safe to run." We didn't need to hypothesize the gap. We measured it, on real systems, and it was real.

Why this matters more than the migration itself

Every AI coding agent on the market can now translate legacy code fast. That capability is commoditizing in real time. What isn't commoditizing, what almost nobody is building, is the independent layer that proves the translation preserved the business logic, with a real counterexample when it didn't and an honest refusal when it can't tell.

If you're running a legacy modernization program, on COBOL, on any mainframe system where "probably fine" isn't a standard your auditors or your regulators will accept, this is the gap that decides whether your migration is actually safe or just looks safe until it isn't. We built the verifier because we needed it to exist. If you're facing the same problem, it already does.

If this is a problem you're facing right now, get in touch.

A Layer of Dust Can Cut Your Solar Output by 25%. Most Dashboards Won't Tell You, Because Their Number Is a Guess.

Vaibhav Tekam — Mon, 13 Jul 2026 12:39:06 +0000

Every solar plant already measures itself. The inverters report power output, temperature, lifetime energy, every few seconds, in a standard protocol most of them already speak. The data is sitting right there on the wire. Most small and mid-size operators, the rooftop plants on factories, schools, hospitals, warehouses, never read it, because the platforms that would read it for them are built for utility-scale plants with utility-scale budgets.

We built Suryamon to read it anyway. Free, open source, self-hosted. And in building it, we ran into a decision that ended up defining the whole project more than any feature did.

The number everyone gives you anyway

Performance Ratio is the single most trusted metric in solar. It answers the question that actually matters: of the energy the sun offered a plant today, how much did it actually capture. A PR of 0.80 means the sun offered 100 units and the plant delivered 80. Healthy plants sit in the 0.75 to 0.85 range, and a PR that starts drifting down is usually the earliest sign of soiling, degradation, or equipment trouble, weeks before anyone would notice it any other way.

Computing PR correctly requires irradiance data, how much sunlight actually hit the panels, measured by a pyranometer. Most small operators don't have one installed. It's an extra sensor, extra cost, extra thing to maintain, and plenty of rooftop plants skip it entirely.

Here's what nearly every monitoring platform does when the pyranometer is missing: it shows you a PR number anyway. Usually built on a modeled or assumed irradiance curve for the region and season, close enough to look plausible, precise enough to look measured. The dashboard shows 0.81 with the same confidence whether that number came from a real sensor or a guess dressed up in two decimal places.

What we refused to do

Suryamon does not do this. If there's no pyranometer, there's no PR. Full stop.

What the plant gets instead is specific yield and CUF, capacity utilization factor, both of which are real, computable, honest numbers that don't require irradiance data to mean something true. Specific yield tells you energy per unit of installed capacity, the fairest way to compare plants of different sizes. CUF tells you energy delivered against the theoretical maximum if the plant ran at full power around the clock. Neither one requires guessing what the sun did.

This was a genuinely uncomfortable decision to make, because a dashboard with a PR field that's sometimes blank looks less complete than one that always has a number in it. We made it anyway, because a fabricated PR is worse than no PR. A plant owner making a decision off a guessed number doesn't know they're looking at a guess. A blank field at least tells the truth about what you don't know.

Where the same instinct shows up again

The same discipline shows up in how Suryamon scores string-level anomalies. Every string's current gets compared against the plant-wide value to detect underperformance, soiling, a failing connector, a dead string. The obvious way to do that comparison is against the mean. We use the median instead.

The reason is specific and worth stating plainly: a single dead string, current near zero, will drag the mean down with it. Compare every string against a mean that a dead string has already corrupted, and the dead string itself can end up looking only mildly underperforming relative to a lowered average, exactly the failure mode you built the check to catch, quietly defeating itself. The median doesn't move the same way. A handful of healthy strings around a median stays a stable reference point even when one string goes to zero, so the dead string still reads as dead instead of borderline.

It's a small implementation choice. It's also the same principle as the PR decision: don't let a metric look more trustworthy than the math underneath it actually is.

How we know the math itself is right

None of this matters if the KPI calculations are wrong, so every one of them is tested against IEC 61724-1 reference cases, the actual published standard, not our own assumptions about what the numbers should be. A 100 kWp plant receiving 5.5 kWh/m2 of irradiation and producing 440 kWh of output has to produce a PR of exactly 0.80, that's asserted in the test suite, not eyeballed.

The collector side gets the same treatment. The simulator's SunSpec Modbus server and the collector's reader talk over a real TCP connection inside the test suite, register encoding, model-chain walking, scale factors, all exercised on the actual wire protocol, not mocked. Every PromQL expression in every alert rule and every dashboard panel gets parsed by a real PromQL parser as part of CI, so a broken query fails the build instead of failing silently in front of a plant owner months later.

The dependency decision we made and documented

Suryamon pins pymodbus to the stable 3.6 line rather than tracking the latest release. The 3.7+ series is mid-migration to a new server API with breaking changes, and we didn't want Suryamon's reliability riding on a dependency that was still finding its own footing. This is written down directly in the codebase as a deliberate, documented choice, not something we're hoping nobody asks about. We'll move when the newer line actually stabilizes. Until then, the older, proven version is the correct choice, even though it's the less exciting one to ship.

The actual thesis

A monitoring tool's job is to tell you the truth about a plant, including the parts of the truth that are inconvenient, a metric it can't compute, a string that's failing, a dependency that isn't ready yet. The moment a tool starts filling gaps with plausible-looking numbers instead of admitting the gap, it has stopped being a monitoring tool and started being a guess with better formatting.

We'd rather ship the honest blank field than the confident wrong number. That's not a limitation of Suryamon. That's the whole point of it.

If this is a problem you're facing right now, get in touch.

The Product Works. That's Not the Question Due Diligence Is Asking.

Vaibhav Tekam — Thu, 09 Jul 2026 11:08:52 +0000

We had a healthcare claims platform stuck at 45% recall on medical coding. Not 45% on a benchmark. 45% recall on CPT and ICD code assignment, in a system whose job is getting those codes right before a claim goes out. Get it wrong, and a claim gets denied. A billing team spends an afternoon on the phone with a payer instead of doing their actual job.

This post is about the architectural decision that moved CPT recall from 45% to 92%, and why no amount of prompt engineering or fine-tuning could have produced the same result on its own.

Why fine-tuning stopped working

When a number is low, the obvious move is a bigger model. We made that move too, and it wasn't a bad instinct, at least at first.

We spent real weeks on it: sharper prompts, more examples, better retrieval so the right context sat in front of the model when it made a call. Recall climbed out of the 40s into the high 50s, low 60s. Genuine progress.

Then, somewhere in the low 60s, it stopped. More examples didn't move it. Sharper prompts didn't move it. The usual levers had stopped answering.

That's the moment worth paying attention to: not when something fails outright, but when it quietly stops responding to the thing that used to work.

Why the LLM could never solve this alone

A language model doesn't return an answer. It returns a best guess from a distribution of plausible ones. Ask it the same claim twice, worded slightly differently, and it can produce two different codes. For a chatbot, that variance is harmless. For a system deciding whether a claim gets paid, it's a serious problem, and it's not a problem a bigger model fixes, because it's not a knowledge gap. It's what a probabilistic system fundamentally is.

A large part of "getting the code right" turned out not to be a judgment call at all. It was already a solved, published, deterministic rule. NCCI, the National Correct Coding Initiative, is a set of edits from CMS stating plainly which procedure code pairs cannot be billed together, full stop, unless a specific modifier applies.

We were asking a model to guess its way toward an answer that already existed as a lookup table.

Here's a concrete version of the failure, details altered but the shape real. A clinical note describes a procedure and, in the same visit, a closely related diagnostic test. The model proposes codes for both, and it's a reasonable read of the note. A careful human coder might make the same call on a first pass.

NCCI disallows the pairing. Not because the clinical reasoning is wrong, but because CMS has already ruled that one procedure is, by definition, a component of the other when performed in the same session. The model has no way to know this from the note, because the constraint isn't in the note. It's in a separate regulatory table with no relationship to what the sentence says.

Multiply that one example by every plausible-looking pair across a real coding surface, and the plateau makes sense. The model wasn't getting worse at reading. It had run out of room to improve on a problem that was never primarily a reading problem.

Building a deterministic validation layer

Once the shape of the problem was clear, the fix stopped being about the model and became about building the layer next to it.

We loaded the full NCCI edit set. All 1.73 million pairs, not a representative sample, because the value of a deterministic check is that it has no blind spots. A rule that catches 90% of bad pairs and misses the rest hasn't solved the problem, it's relocated the failure to wherever nobody's watching.

The resulting pipeline:

The model proposes candidate codes. It's genuinely good at this: reading unstructured clinical documentation and surfacing plausible candidates is a real strength, and there's no deterministic substitute for it.
Every candidate pair is checked against the NCCI table before it reaches a claim.
Only pairs that clear the table ship. A candidate the model favored but the table rejects doesn't go through, no exceptions, because exceptions quietly rebuild the guessing problem the check was meant to remove. The model proposes. The table proves. Once we stopped treating this as one system to keep tuning and started treating it as two systems doing two different jobs, the remaining work was engineering, not research.

It didn't ship clean

The first version of the lookup was a linear scan through all 1.73 million rows per candidate pair. In a demo with one or two claims, it felt instant. In production, with a dozen candidate codes per claim and every pairwise combination needing a check, it visibly lagged. We rebuilt it as a proper index keyed on the pairs themselves. Unglamorous, and the difference between a check that's correct in principle and one that's usable in a real-time pipeline.

The second issue we found before it became an incident, not after. CMS updates the NCCI edit set every quarter. A static, load-once table is a stale table waiting to happen. Rather than wait for a batch of claims to quietly clear an outdated check, we went looking for the failure mode on our own schedule: what happens to a "deterministic" layer when the thing it's being deterministic about changes underneath it. The fix was a scheduled refresh and a version stamp on every verdict, so any decision can be traced back to exactly which edit set produced it. Finding that gap by design, rather than by incident report, is the same principle as the rest of this build: check before you're forced to.

Recall on CPT went from the low 60s to 92%. ICD landed lower, at 76.8%, and that gap is worth explaining rather than smoothing over. NCCI's rules live on the procedure-code side; that's the layer with a clean, published table. ICD diagnosis coding runs on a different kind of logic, driven by clinical specificity and payer documentation requirements, much of which is a genuine judgment call rather than a lookup. Judgment calls are exactly where a probabilistic system should be doing the work, not where a rulebook should be forced onto it that doesn't exist. Read side by side, the two numbers aren't "better here, worse there." CPT had more of the kind of problem a table solves. ICD has more of the kind that needs judgment. The recall gap is largely a measurement of that ratio.

The real lesson

Model accuracy has a ceiling, and the ceiling usually has nothing to do with the model.

Some percentage of what a system needs to get right isn't a pattern to learn. It's a rule that already exists, published and unambiguous, waiting to be wired in rather than rediscovered through enough training examples.

The climb from 45% to the low 60s was a model problem, and the standard tools solved it. The climb from the low 60s to 92% was an architecture problem, and no amount of prompting was ever going to close it. On a dashboard, the two climbs look identical. They are not the same problem, and the second one doesn't yield to more of what fixed the first.

So before reaching for a bigger model on a number that won't move: is this a guessing problem, or is it a problem that already has a correct answer sitting somewhere, waiting for someone to stop asking a probabilistic system to guess at it.

More often than expected, it's the second one.

Production AI systems become reliable when probabilistic models and deterministic systems each do the job they were designed to do.

If this is a problem you're facing right now, get in touch.

When Fine-Tuning Stops Working: The Architecture Lesson That Took Our Healthcare AI From 60% to 92% Recall

Vaibhav Tekam — Wed, 08 Jul 2026 16:16:22 +0000

This post is about the architectural decision that moved CPT recall from 45% to 92%, and why no amount of prompt engineering or fine-tuning could have produced the same result on its own.

Why fine-tuning stopped working

When a number is low, the obvious move is a bigger model. We made that move too, and it wasn't a bad instinct, at least at first.

Then, somewhere in the low 60s, it stopped. More examples didn't move it. Sharper prompts didn't move it. The usual levers had stopped answering.

That's the moment worth paying attention to: not when something fails outright, but when it quietly stops responding to the thing that used to work.

Why the LLM could never solve this alone

We were asking a model to guess its way toward an answer that already existed as a lookup table.

Building a deterministic validation layer

Once the shape of the problem was clear, the fix stopped being about the model and became about building the layer next to it.

The resulting pipeline:

The model proposes candidate codes. It's genuinely good at this: reading unstructured clinical documentation and surfacing plausible candidates is a real strength, and there's no deterministic substitute for it.
Every candidate pair is checked against the NCCI table before it reaches a claim.
Only pairs that clear the table ship. A candidate the model favored but the table rejects doesn't go through, no exceptions, because exceptions quietly rebuild the guessing problem the check was meant to remove. The model proposes. The table proves. Once we stopped treating this as one system to keep tuning and started treating it as two systems doing two different jobs, the remaining work was engineering, not research.

It didn't ship clean

The real lesson

Model accuracy has a ceiling, and the ceiling usually has nothing to do with the model.

More often than expected, it's the second one.

Production AI systems become reliable when probabilistic models and deterministic systems each do the job they were designed to do.

If you like to know more, let's discuss.

Most Engineering Blogs Show You What Worked. We're Going to Show You What Broke.

Vaibhav Tekam — Wed, 08 Jul 2026 07:18:23 +0000

Most engineering blogs focus on the outcome. They present clean architectures, successful deployments, and polished case studies that make complex systems appear simpler than they really were.

What they rarely document is the engineering journey that produced those outcomes. They seldom discuss the migration that stalled halfway through, the production incident that forced a redesign, or the assumptions that proved to be wrong once the system met real users and real workloads.

Those experiences contain the lessons that matter most.

At Tech4Biz Solutions, we have spent years designing, building, and restoring production systems across artificial intelligence, cloud infrastructure, enterprise software, embedded hardware, industrial automation, and observability. Regardless of the technology involved, we have reached the same conclusion time and again.

Reliable systems are rarely the result of perfect technology. They are the result of thoughtful engineering, continuous iteration, careful trade-offs, and decisions made under real-world constraints.

Why We're Writing

We started writing because the most valuable engineering lessons rarely appear in traditional case studies.

Case studies usually explain what worked. They seldom explain what failed, what assumptions proved to be incorrect, what designs had to be abandoned, or why the final system looked very different from the original plan.

We're less interested in presenting finished architectures than in explaining how those architectures evolved into reliable production systems.

What We'll Write About

Every article will be grounded in engineering work we've actually done, systems we've actually built, or production problems we've actually solved.

We'll write about:

Production AI systems
Cloud architecture and distributed systems
Platform engineering and observability
Embedded systems and hardware integration
Industrial software and automation
Performance, scalability, reliability, and security
Engineering decisions made under real deadlines and real business constraints

We'll explain not only what we built, but also why we built it that way, what alternatives we considered, and what we learned in the process.

What You Won't Find

You won't find toy projects designed to demonstrate a framework.

You won't find success stories with the difficult parts edited out.

You won't find articles written simply because a technology happens to be trending.

You won't find sales pitches disguised as engineering articles.

Good engineering earns trust through transparency, not marketing.

What's Coming Next

Over the coming weeks, we'll share deep dives into production systems we've built and stabilized across cloud infrastructure, enterprise AI, observability, embedded hardware, and industrial software.

We'll also share the engineering behind Suryamon, our open-source solar plant monitoring platform, along with the architectural decisions, trade-offs, and lessons that shaped it.

The finished system is only part of the story. We're here to document everything that happened before it became reliable.

If these are the kinds of engineering problems you enjoy solving, we hope you'll follow along.