DEV Community: Ashutosh Singhal

I fed raw radiance to a hyperspectral crop AI. On a held-out synthetic test, its score fell from 0.887 to 0.373.

Ashutosh Singhal — Tue, 14 Jul 2026 08:00:00 +0000

An NDVI alert that says only "stressed" leaves a grower unable to choose between nitrogen, irrigation, or an early-tar-spot scout; I built SpectraRx around that decision. In the deliberately broken path of this physics-grounded synthetic demo, the same 424-band spectral CNN falls from 0.887 to 0.373 on its held-out test when I feed it raw top-of-atmosphere radiance instead of corrected surface reflectance.

I did not put that failure in the demo as a party trick. I put it there because a crop-stress score without the conditions that make its input meaningful is an invitation to turn a neat map into a bad field decision. The scenes are synthetic, physics-grounded cubes for one corn crop and geography, not real captures or field guarantees. The point is to make the chain inspectable while the assumptions are visible.

The NDVI tab in SpectraRx: one amber region is tagged "stressed," with no cause or prescription attached.

I begin with that single amber NDVI blob because it shows the limit of a detection alert. It is useful as a signal to look closer, but it cannot say whether a 27-meter boom should change a nitrogen rate, whether a water-stress pattern needs inspection, or whether early tar spot should send an agronomist into the field. I wanted the demo to answer the next question, then to show the evidence and the limits of that answer.

You can run the chain at veriprajna.com/demos/hyperspectral-agriculture-ai. The screen does not hide the inconvenient case. It lets me skip atmospheric correction and watch the decision path change.

I learned that calibration changes the object the classifier sees

I used to file atmospheric correction under preparation work. The cube arrives as top-of-atmosphere radiance, and the demo's empirical-line correction turns that into BOA surface reflectance with calibration provenance attached to the scene. That distinction sounded like engineering plumbing until I deliberately bypassed it.

I kept the trained network and its weights fixed. I changed the input path. On the held-out, synthetic physics-grounded test of 1,280 pixels from an unseen cultivar and soil, the result moved from 0.887 macro-F1 to 0.373. That is a measured result for this model and this test, not a statement that every hyperspectral model will fail by the same amount.

I found the contrast more useful than a larger benchmark table because it makes a practical failure concrete. The uncorrected path is not merely noisier input. Raw radiance carries atmospheric and illumination effects into a classifier that was designed to interpret corrected surface reflectance. A network can still return a class and a confidence, but neither is sufficient reason to authorize a prescription.

With "Skip atmospheric correction" on, the held-out synthetic test falls from 0.887 to 0.373. In the skipped-calibration seed-7 scene, 92.2% of pixels abstain, 140 of 144 zones route to scouting, and four water-stress zones still apply.

I made the raw-radiance path visible because an accuracy number by itself leaves out the operational question: what should happen after the system discovers it is on weaker ground? In SpectraRx, the answer is not a universal block. Raw TOA still reaches the classifier in this deliberate test, and the downstream confidence and physics-agreement gates make the uncertainty visible.

I spent time separating that result from a claim that the network has become useless. The classifier is a custom 1-D spectral CNN that reads a full 424-band curve from 400 to 2515 nm. Its normal job is to distinguish healthy, nitrogen-deficient, water-stressed, and early-tar-spot pixels from BOA surface reflectance. The calibration experiment does not change the crop classes, retrain the model, or silently swap in a weaker baseline. It changes one upstream condition and lets the rest of the decision chain show the consequence.

I found that useful because the usual response to a bad score is to ask for another score. In this case I needed a different question: what evidence does the system have for acting after the input has changed? The answer starts with calibration provenance, but it cannot end there. A confidence value is only one signal. The physics-agreement check can abstain when the CNN's class conflicts with the documented feature, such as a nitrogen call without the expected red-edge shift. The inspector also shows the full curve and a band-saliency trace so an operator can compare the model's evidence with documented diagnostics.

I use that inspector as an accountability surface, not a proof of infallibility. For early tar spot, the trace's sampled 530 and 535 nm bins sit nearest the PRI diagnostic near 531 nm. Nitrogen evidence sits in the 680 to 720 nm red-edge range. Water's SWIR features are diagnostic checks rather than claimed top saliency peaks. Those checks make the reasoning available for inspection. They do not transform a synthetic benchmark into a real-field guarantee, and they do not erase the need for scouting when the input path becomes less trustworthy.

I also resisted making the baseline look weak so the failure would appear more dramatic. On the held-out synthetic distribution-shift test, the real Sentinel-2 broadband incumbent reaches 0.69 macro-F1. It is near 1.0 on nitrogen because it has red-edge and SWIR bands. The hyperspectral model reaches 0.887 overall, with its strongest structural advantage in water stress and early tar spot. That is a meaningful comparison, but it still does not decide whether a particular zone should receive a prescription after calibration is skipped. The gate and zone rule exist to answer that later operational question.

I needed the safeguard to change a field action, not just a dashboard color

I was careful not to turn abstention into another vague promise. A hatched pixel means the system is withholding a pixel-level diagnosis when confidence is low, the signature is mixed, or the classifier conflicts with the documented physics feature. The default calibrated seed-7 demo scene shows 20.1% of pixels abstained. That figure is an in-app observation for that scene, not a universal evaluation metric.

I then had to decide what an operator does with a map containing those pixels. A sprayer does not work one pixel at a time. SpectraRx aggregates 5-meter diagnosis pixels into 25-meter management cells using a 27-meter boom profile. The rule is deliberately visible: zones with at least 34% abstained pixels route to ground-truth scouting, while other zones use their non-abstained majority.

I think that rule is where the abstract idea of uncertainty becomes a real workflow. In the skipped-calibration seed-7 scene, 92.2% of pixels abstain and 140 of 144 management zones route to scouting. Four water-stress zones still apply under the zone rule. The system has not proved that the four zones are safe in an open-world sense. It has made its conditions and its routing rule available for review instead of pretending every changed input must yield either a spray or a blank screen.

I also wanted a skeptical remote-sensing reader to see more than the gate. The spectral inspector shows the 424-band curve and an orange band-saliency trace. On early tar spot, the sampled 530 and 535 nm saliency bins sit nearest the PRI diagnostic near 531 nm; nitrogen evidence appears in the 680 to 720 nm red-edge range. For water, the documented SWIR features are diagnostic checks, not claims about the CNN's highest saliency peak. That is an inspectable comparison with documented physics, not proof that the model can never use a spurious cue.

I could not call it a prescription until a boom and an agronomist could use it

I learned that a diagnosis map is still incomplete after the gates do their work. The default calibrated scene has 144 management cells: 21 apply zones, 25 scout zones, and 98 healthy zones at V12. SpectraRx carries the surviving zone decision into a variable-rate prescription. Nitrogen rates follow severity, and fungicide is permitted only in the configured R3 window. The exports are real ISO-XML and GeoJSON formats, while live sensor and farm-management integrations are not implemented in this demo.

I made the downstream record deterministic for the same reason I made the gate visible. The demo generates an EU Farm-to-Fork IPM-style record template from the diagnosis and prescription. It includes zone coordinates, majority-class zone coverage, spectral evidence, alternatives evaluated, recommended product and rate, the R3 deadline, calibration provenance, and a certified-agronomist sign-off line. It is a template for review, not a filed compliance record.

The demo-generated EU Farm-to-Fork IPM-style record template carries each zone's coordinates, spectral evidence, product and rate, R3 deadline, calibration provenance, and an agronomist sign-off line for review.

I kept the language model outside this trust path. Signal processing, the trained spectral classifier, deterministic gates, equipment math, and the record template carry the load-bearing decisions. An optional model can render IPM justification prose from facts already computed, with a templated fallback so the demo runs without an API key. It does not decide what a crop needs or whether a zone should scout.

I stopped measuring the product by the classifier alone

I can still care about the classifier. On the held-out synthetic distribution-shift test, SpectraRx records 0.887 macro-F1 against 0.69 for the Sentinel-2 broadband incumbent. The difference is strongest in water stress, 0.97 against 0.63, and early tar spot, 0.77 against 0.61. Nitrogen is near 1.0 for both, which is useful precisely because it leaves the incumbent as a real baseline rather than a strawman.

I do not use those results as a claim about an open field. They cover 1,280 held-out synthetic pixels, one crop, and one geography. A two-season ground-truth library is still needed before anyone can make a real-world accuracy claim. The lesson I draw from the demo is narrower and more practical: a classification score is only one part of a decision system, and it is the part most likely to be quoted without its operating conditions.

I built the calibration failure into SpectraRx because it forces those conditions into view. It shows the input changing, the score falling, the downstream gates surfacing uncertainty, the zone rule routing a scout path, and the prescription record preserving what the operator and reviewer need to inspect. That sequence is the work I want to improve before I chase another decimal point.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

I return to veriprajna.com/demos/hyperspectral-agriculture-ai when I want to test whether the demo is being honest. The useful crop-AI system is the one that can show why a scene became less trustworthy, route that uncertainty to a field action, and leave a record of the decision it did and did not make.

Eight Green Formal Proofs Became Five Fileable Ones When I Audited the SystemVerilog Assertions

Ashutosh Singhal — Mon, 13 Jul 2026 08:00:00 +0000

I watched a synthetic formal board report 8/8 PROVEN , then watched its own audit certify only 5/8 as TRUSTWORTHY. That reversal is the premise of Proof Firewall, our runnable demonstration of governance for AI-generated SystemVerilog assertions (SVA), and it changed the standard I want a green proof to meet before it reaches a tape-out sign-off review.

I built the board with fixture-authored “LLM-authored” properties on a synthetic arbiter, a two-stage pipeline, and a CDC crossing because the uncomfortable case deserves to be visible. An assertion can look perfectly respectable in a property ledger. A formal engine can return a green result. Yet the implication may never have had to do any work, or it may continue to pass after the relevant design behavior has been broken. I had been treating the word PROVEN as a destination. Building this demo forced me to treat it as the beginning of an evidence review.

The Proof Firewall demo does not replace a formal engine, ingest real RTL, or call a live LLM in its default path. It is deliberately smaller and more inspectable: a pure-Python explicit-state model checker evaluates a synthetic transition-system IR, then a governance gate checks antecedent reachability, mutation kills, and cone of influence (COI). The output is either a reason to file a signed demonstration certificate or a reason to hold the result for human review.

I started with the wrong kind of green

I remember the first version of the board feeling reassuring precisely because it was so clean. Eight properties, eight green badges, and a bare-flow view that made the work look finished. My early instinct was to make the demo explain that clean result better. I thought the engineering task was presentation: surface the proofs, show the assertions, make the dashboard easier to trust. The green result was real , but it was answering a smaller question than the one a reviewer needs to ask.

Then I put the same eight properties through the checks that a filing conversation actually invites. Did the antecedent ever become true? Would the assertion object if a relevant part of the design were changed? Does it constrain a meaningful COI? Those questions are less flattering than a green badge because they ask what the proof has earned, not merely what the solver returned.

I had to abandon the first framing of the build. A screen that showed 8/8 PROVEN was an accurate view of the bare-flow baseline, but it was incomplete as a sign-off story. After the firewall audit, the same fixed synthetic board has five TRUSTWORTHY results, one VACUOUS result, and two WEAK results. The remaining three are not relabelled as success. They are held with the evidence that explains why. A proof label and a filing decision are different artifacts.

The board makes the reversal visible: the fixed synthetic bare-flow result is 8/8 PROVEN, while the audit certifies 5/8 as TRUSTWORTHY.

I chose the word “governance” carefully here. The demo's deterministic checks make the filing decision reviewable. An optional SVA author may propose an assertion, but the model checker and policy gate determine the verdict. Agents advise, code decides. I was trying to make the gate legible enough that the negative result would be useful rather than merely embarrassing. A withheld result needs a reason a verification engineer can inspect, reproduce, and challenge.

ARB3 made the problem impossible to ignore

I found the clearest failure in ARB3, the synthetic arbiter property assert (g0 && g1) |-> (turn == 0). In the bare flow, it is green. When I opened its waveform and reachability evidence, the antecedent g0 && g1 was unreachable in that synthetic arbiter. The implication had been proven only in the narrow sense that it was never forced to answer for the state it described. The antecedent never fires.

That distinction is easy to say and hard to keep present when a verification dashboard is full of green. I initially read the implication as a claim about arbitration behavior. The reachability result changed what I was looking at. It was a claim whose triggering condition never occurred. Calling that VACUOUS is more useful than preserving a green label, because it directs the reviewer toward the assumption or stimulus that made the proof empty.

The ARB3 panel shows why a green implication is withheld: its antecedent is unreachable in the synthetic arbiter fixture.

I kept returning to this panel while working on the policy labels. VACUOUS can sound like a harsh result until the alternative is considered. If a sign-off record retains a proof without recording that its antecedent never fires, the review has received a conclusion without the condition that gives it meaning. The better record is the one that makes the limitation explicit and leaves a person something concrete to interrogate. That reachability record belongs beside the verdict.

I also had to resist treating vacuity as a cosmetic warning. If the property is meant to constrain an arbitration condition, unreachable triggering behavior is central evidence about whether the property exercised the intended behavior. The dashboard should not ask a reviewer to infer that from a green result. It should preserve the reachability finding, route the result out of the certificate path, and make the next review action obvious.

The industry context sharpened the stakes for me. The 2024 Wilson Research Group / Siemens EDA study cited in the demo specification reports 14% first-silicon success. That is not a Veriprajna measurement, and this synthetic board does not claim to explain that figure. It does make me less willing to treat a pleasant dashboard state as evidence by itself.

The pipeline property survived the breakage I expected it to catch

I hit the second failure while testing PIPE3, a synthetic two-stage pipeline property: assert v2 |-> (s2 == s2). I had wanted a concise example of an assertion that read sensibly enough to slide through a superficial review. The consequent is a tautology. It says s2 equals itself. The consequent constrains nothing.

The important move in the demo is not merely spotting the tautology in prose. The governance gate injects relevant single-point design mutations and asks whether the property kills them. For the featured weak pipeline case, PIPE3 records a 0/6 mutation kill result. The property survives the relevant broken variants. That is why the policy assigns WEAK rather than allowing the bare PROVEN result to stand as filing evidence. The mutation result tests useful sensitivity.

(s2 == s2) WEAK because it survives the relevant injected mutations in the synthetic pipeline." width="800" height="450">
The pipeline view pairs the tautological PIPE3 consequent with its WEAK verdict, showing the kind of assertion a mutation kill-test can expose.

I learned something uncomfortable from trying to make this example feel less obvious. A human can read s2 == s2 and dismiss it quickly. Many weaknesses will not announce themselves so plainly. That is why I did not want the demo to depend on the operator spotting a suspicious string. The useful artifact is the procedure: reachability, a relevant mutation kill-test, COI, and a policy decision that records its reason.

I came to see the mutation check as a disciplined way of refusing a too-convenient reading of a proof. The goal is not to manufacture a dramatic failure. It is to ask whether the property would notice a relevant local change in the behavior it is supposed to constrain. When it does not, the result tells a reviewer something actionable: this assertion needs strengthening or a different review path before it can support the sign-off record.

This is also why the demo's benchmark needs a narrow description. Its local python -m backend.bench run scores 18/18 against a fixed labeled synthetic assertion set and identifies 6 proofs that the demo's own ungated baseline would have rubber-stamped. Those figures are a reproducibility check on this demo's labeled fixtures. They are not a production rate, a claim about AI-authored assertions generally, or a comparison with commercial formal tools.

I stopped trying to make the gate look permissive

I had a design choice after the first audit results: soften the withheld verdicts so the board felt more optimistic, or let the board decline to certify what it could not defend. I chose the latter because a real sign-off review needs the ability to distinguish a complete proof from a bounded one, an unreachable antecedent from a meaningful property, and a weak check from one that reacts to relevant broken behavior. Withholding is a review outcome, not a dead end.

That choice shows up in the policy vocabulary. TRUSTWORTHY earns the signed demonstration certificate. BOUNDED-PROVEN, VACUOUS, WEAK, DEAD, and VIOLATED preserve different reasons for withholding that certificate or escalating the result. In the CDC fixture, for example, the stronger property assert (req && !ack) |-> ##1 req is VIOLATED and produces a concrete synthetic counterexample waveform. It illustrates a lost-transaction or CDC failure class. It says nothing about a customer chip.

I do not see this as a pitch for replacing a verification team's existing engine. The production direction is engine-agnostic: place a gate around an existing formal workflow, then make its acceptance criteria inspectable. Real-engine adapters and RTL ingest are deferred in this demo. The demonstrated boundary is intentionally narrow. That boundary matters because it keeps the claim proportional to what is actually running.

I now want the receipt beside the verdict

I keep thinking about the artifact a sign-off meeting needs when the assertion author is assisted by AI. It is not a confidence score from the author. It is a record that says which checks ran, what the reachability result was, which mutations were killed, what the COI contained, and why the policy permitted or withheld certification. The review needs evidence it can reopen.

That is what the demo exports in signoff_certificate.json: per-property verdicts, reachability, mutation results, COI, counterexample records where applicable, and a SHA-256 field. I built the certificate as a demonstration record because a reviewer should be able to reconstruct the decision without accepting a green badge on faith. A certificate should preserve the path to its verdict.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

I made the demo runnable so the 8/8-to-5/8 reversal can be inspected rather than repeated as a slogan. The result I take away from it is modest but durable: a proof worth filing carries evidence of what it constrained, what it survived, and why someone may rely on it. Green remains useful. It simply needs a record that lets the next reviewer decide whether it deserves to travel further.

A satellite said a depot was under water. It was a cloud's shadow, and the parametric trigger was one click from auto-paying...

Ashutosh Singhal — Sun, 12 Jul 2026 08:00:00 +0000

A satellite looked at Mesa Junction Depot, saw a dark patch where dry ground should be, and flagged the location as flooded. A parametric flood policy read that trigger as fact and armed a $1.2M payout to fire automatically, no adjuster, no phone call, no second look. The dark patch was a cloud's shadow. It was gone from the imagery six days later, and the ground under it had never been wet.

I built that exact case on purpose, because it is the one a single satellite frame can never catch. I am Ashutosh, and I run Veriprajna. TriggerProof is a demo I built to prove a specific claim: the dangerous moment in parametric flood insurance is not detection, it is the decision to pay. Every location, tile, river gauge, and field report in it is synthetic, physics-faithful stand-ins I wrote so I could stage the failure modes cleanly. You can open it and drive the whole thing yourself at veriprajna.com/demos/satellite-flood-intelligence. Mesa Junction Depot is where I want to start, because it is the case that taught me what I was actually building.

Four different things darken the same pixels

I did not appreciate the real problem until I sat down to generate the synthetic imagery and had to make four things look identical on one frame. Parametric flood insurance replaced the loss adjuster with a trigger: a satellite says a location is under water, and money moves. The trouble is that on a single optical image or one radar pass, real floodwater, a cloud shadow, a radar or terrain shadow, and a permanent reservoir all go dark the same way. A single frame cannot separate them, because the information that separates them does not live in a single frame.

That reframed the whole demo for me. The catastrophic error in this business is not the flood the trigger misses. It is the flood the trigger invents , a confident $2M payout on a shadow, with no evidence trail to defend the call when a reinsurer asks about it a year later. TriggerProof does not detect floods and does not produce satellite data. It takes a trigger that has already fired and adjudicates whether it should actually pay.

One frame said flood; the next said the ground was never wet

I still remember stepping through the Mesa Junction filmstrip the first time it rendered correctly. The interface lets you open a flagged location and walk the acquisition frames one at a time, optical on top, radar below. In the trigger frame, the dark patch is right there, water-signal green, exactly what fired the alarm. Step to the next acquisition and it is gone. Step back and the radar backscatter under that same patch reads normal in every frame, because radar saw dry ground straight through the cloud the whole time.

Mesa Junction Depot, the $1.2M case. The optical dark patch shows up only in the trigger frame (temporal persistence 33%) and radar backscatter stayed normal throughout. Rule R1 fails, rule R2 fails, and the verdict is cloud shadow, not water.

The shadow moved; the water would have stayed. You only see that across time and across sensors, never on the single frame that fired the trigger.

The physics is not subtle once you lay the frames side by side. A cloud shadow is transient and it travels at cloud speed, so it is dark in one acquisition and gone in the next. Real floodwater persists across acquisitions and reads dark on optical and low on radar at the same time. Two rules encode exactly that: temporal persistence and radar-optical agreement. On Mesa Junction they both come back FAIL, and the classifier lands on cloud shadow at confidence 1.00. The $1.2M never should have been queued.

The five rules decide, not the language model

I tried, early on, to let the language model make this call, and I am glad I did because it failed in the most instructive way possible. I had an agent reading the same evidence and asked it, in effect, whether the location was really flooded. On an ambiguous case it wrote me a fluent, confident paragraph arguing for a flood, and it was wrong, and nothing in its tone flagged that it was wrong. That afternoon settled a design decision I have not reopened since.

So the decision lives in plain Python, in five inspectable discriminators, with no model anywhere in the path that moves money. Temporal persistence (R1) separates a flood from a transient cloud shadow. Radar-optical agreement (R2) separates a flood from both a cloud shadow and a radar shadow. DEM slope (R3) rejects water that would have to pool on steep terrain. A permanent-water mask (R4) excludes known reservoirs. Hydrological linkage (R5) checks that the wet area actually connects to the drainage network. The verifier decides; the language model only advises. The advisory agent is built on Pydantic AI, model-swappable, defaulting to claude-opus-4-8, and it checks the physics verdict against independent ground signals and returns corroborates, contradicts, or inconclusive. It can be overruled, and when I pull the API key the demo runs fully offline on a deterministic fallback, because the part I trust with a payout cannot be the part that speaks in confident paragraphs.

When the physics itself is unsure, the system escalates instead of guessing

I care more about the case where the system says "I do not know" than about any of the clean catches. Canal Street Hub is that case. The optical and radar signatures are borderline, the flood signal persists across two of three frames, and the independent river gauge never crossed bankfull. The evidence genuinely conflicts. The confidence comes back at 0.151, far under the automation bar of 0.65 that I fitted on the labeled set, and the ground agent contradicts the satellite classification outright.

Canal Street Hub, $0.8M at stake. The signal is borderline and the river gauge never crossed bankfull, so the ground evidence conflicts with the satellite. Confidence 0.151 sits below the 0.65 bar, and the case escalates to a human with the full evidence attached instead of being auto-decided.

The policy gate sends that $0.8M to a human with all of the evidence attached, marked "needs proof," rather than flipping a coin and calling it automation. A flood trigger that escalates the genuinely ambiguous case reads to some buyers as the weaker product. I see it the other way. It is the only version I would let run unattended , because the alternative to escalation here is a fast guess on real money, dressed up as a decision.

The portfolio number I keep coming back to

I keep pulling up the eight-location book, because it makes the stakes concrete in a way no single case does. A storm passes a portfolio of eight areas. The legacy single-frame trigger fires on six of them and queues $8.0M in automatic payouts. TriggerProof adjudicates the book: two real floods confirmed and paid at $4.0M, three false positives suppressed (a cloud shadow at $1.2M, a radar shadow at $1.0M, a permanent reservoir at $1.0M) for $3.2M withheld, and the one ambiguous case escalated at $0.8M.

The eight-location book after adjudication. Of the $8.0M the single-frame trigger would have auto-paid, $4.0M is confirmed on two real floods and $4.0M is stopped or held: $3.2M of false positives suppressed and $0.8M escalated for proof. Every row carries its own forensic dossier.

Half of what the legacy trigger would have paid, $4.0M of $8.0M, is either stopped or held for proof. That is the number, and I want to be precise about its scope: this is the demo's synthetic portfolio, eight cases I authored to be physically faithful, not a book of real claims. The mechanism is real and inspectable. The claims are staged so you can watch the mechanism work.

Zero unsafe decisions, and the caveat I refuse to drop

I ran a proper benchmark because a portfolio of eight is a story, not evidence. The harness scores 60 labeled cases spanning clear signatures down to near-threshold noise. The headline is not an accuracy score, it is a safety count: zero unsafe automated decisions, against 48 for a single-frame baseline that pays every flagged case. Eighty percent of cases auto-resolve and the uncertain 20% escalate. Among the auto-resolved cases, false-positive suppression is 36 of 36 and flood recall is 12 of 12, and every one of the 12 genuinely ambiguous cases escalates instead of being auto-decided.

The 60-case labelled benchmark. The number that matters is the leftmost one: zero unsafe automated decisions against 48 for the single-frame baseline, with 80% auto-resolved and the uncertain fifth escalated for proof.

The point was never a perfect score on my own test set. The point is that the system never makes an unsafe automated decision. When it is not sure, it escalates.

The caveat travels with every one of those numbers, and I will not let it get stripped off. They are measured on a fixed, labeled set of 60 synthetic, physically-faithful cases, not an open-world guarantee and not a field result. The honest next step is not a bigger claim, it is validation against real archives like Sen1Floods11 and live Sentinel scenes, and that is the first thing a real engagement would deliver, not something this demo has done. Saying that plainly is what lets me stand behind the rest of the numbers.

A payout you cannot defend later is a liability, even when it was right

I did not set out to make the evidence trail the centerpiece, but by the end it was the part I was most sure an insurer could actually stand behind. A correct payout you cannot reconstruct later is still a liability, because "the satellite said so" is not a defense a reinsurer or an auditor accepts. So every decision, a pay and a suppression alike, emits a forensic dossier: the data lineage of every frame, the per-rule evidence with each discriminator's measured value, the false-positive elimination log, the independent ground cross-reference, and a SHA-256 provenance hash of the decision.

The dossier behind the Mesa Junction denial. Every rule shows its measured value and result, the false-positive elimination log names why it was not a flood, and the whole record carries a SHA-256 hash. This is the artifact you hand a reinsurer, not a screenshot of a dashboard.

I want to be careful about what that hash is and is not. It is a content hash that makes the record tamper-evident, so anyone can recompute it and check the decision was not altered after the fact. It is not a PKI digital signature, and the satellite retrieval, the SAR tasking, the ground feeds, and the claims-platform integration are all stubbed in this demo, simulated so the whole thing runs on my laptop. What is real is the shape of the record: for every automated payout decision, exactly which physics said what, at what confidence, checked against what independent signal.

Parametric insurance made a genuine trade, giving up the loss adjuster to make payouts instant and disputeless. What it inherited was a physics problem it cannot see past on a single frame, and the failure that trade exposes you to is a fast, confident payout on a shadow. The durable fix lives outside the model: deterministic rules that separate the look-alikes across time and sensors, a gate that escalates the ambiguous case to a person, and a record that makes each decision defensible. A sharper satellite does not fix an adjudication problem, and this is an adjudication problem.

And if you would rather watch it than read me describe it, here is the whole thing running end to end.

Mesa Junction is the case I keep coming back to. Only the second acquisition ever knew the difference between the shadow and the flood, and the trigger fired before it arrived. You can step through that filmstrip yourself, and break the payout you think should have fired, at veriprajna.com/demos/satellite-flood-intelligence. The question I would put to anyone running an automatic flood book is a narrow, answerable one: of the triggers you auto-paid last season, how many could you still prove were water and not a shadow?

A perfect model can't stop your self-driving lab from running a doomed synthesis. I built the gate that does.

Ashutosh Singhal — Sat, 11 Jul 2026 08:00:00 +0000

I keep getting asked the same question by researchers who see what we build, and for a long time I answered it the way they wanted me to. "What is your sample-efficiency multiple?" How many fewer experiments does your loop need than brute-force screening? It is a fair question. It is also, I have come to believe, the wrong one to lead with, and I did not understand why until I sat and watched a simulated robot cheerfully commit to a recipe that would have oxidised in the air before it finished mixing.

I am Ashutosh, and I run Veriprajna. We build the brain, the safety gate, and the provenance layer for self-driving labs, on the hardware a lab already owns. Not a hosted robotic facility. Not a replacement for the instruments a materials group spent a decade buying. The thin, load-bearing layer of software that decides what to run next, refuses the experiments that are doomed, and keeps the kind of record a regulator asks for. This demo, which you can open and drive yourself at veriprajna.com/demos/autonomous-lab-ai, is where I went to prove to myself that the layer, not the model, is the product.

The number everyone asks me about is the wrong one

I spent the first month of this build chasing the multiple, because that is what the field rewards. The self-driving-lab conversation right now is a race for a better surrogate and a bigger Bayesian-optimization-versus-screening ratio. I get it. It is a clean, quotable number. On our own hidden objective, the closed loop reaches a spec-meeting material in 75 experiments where random screening would need about 1,491 , which the dashboard reports as 19.9 times fewer. That is a real result on a real, reproducible baseline, and I am glad it is there.

But here is the thing that stopped me. A perfect surrogate, one that predicts every property flawlessly, still cannot stop a charge-imbalanced synthesis from running. It cannot refuse a composition that will oxidise on contact with humid air. It cannot produce a record a drug regulator will accept. And it cannot, by itself, close the loop. The model is the part everyone competes on, and it is the part that holds the least durable value, because the moment your model is good, your problem is no longer prediction. It is control, safety, and proof.

A better model does not make an autonomous lab trustworthy. It makes it faster at running experiments you still cannot prove you should have run.

Berkeley's A-Lab, one of the most cited autonomous synthesis efforts we have, reported a 71% success rate, which means roughly 29% of its attempts did not yield the target (A-Lab, Nature, 2023). Read that number the way a lab director reads it. Nearly a third of the robot-time and reagents went to experiments that produced nothing. A better acquisition function does not fix that. Something else has to.

What does a doomed experiment actually cost?

I built the gate second, and it turned out to be the thing I should have built first. It sits outside the agents, in deterministic code, and every candidate the optimizer proposes has to pass through it before a single reagent is touched. It runs five checks, and every one of them is published chemistry, not a heuristic I invented: Goldschmidt tolerance factor outside the perovskite formability window, octahedral factor outside the stable-octahedra window, charge neutrality violated by an uncompensated B-site substitution, a Sn(II) oxidation hazard at process humidity, and annealing above the temperature where the organic precursors decompose. If a candidate trips any of them, the gate returns BLOCKED, names the rule, and reports the money and robot-hours it just saved.

A candidate the gate caught before synthesis. It flags a Sn(II) oxidation hazard at 50% humidity, shows the three properties that would have missed target anyway, and books the $105 and 2.33 robot-hours it saved. Every block writes its own ALCOA+ receipt, here record #111.

The first time I clicked into one of these blocked cards, I felt the argument settle. Look at what the panel shows. A composition heavy in Sn(II), at process humidity, that the digital twin refuses because tin in that oxidation state oxidises in humid air. It is not a bad prediction the model should have caught. It is a physical fact the model was never the right tool to enforce. The gate saved 105 dollars and a little over two robot-hours on that one candidate, and it wrote a receipt for the decision. Multiply that across a campaign and you are no longer talking about a nice-to-have. You are talking about the 29% leak, closed on purpose.

I want to be precise about what this is, because the honest framing matters more to me than the impressive one. The "lab" behind this gate is a simulator with a SiLA-2-shaped interface, and the objective is a physically-motivated synthetic benchmark, not real DFT and not real bench data. I am not claiming we synthesised a material. I am claiming the mechanism, the deterministic refusal of doomed chemistry, is real, inspectable, and independent of how good any model happens to be.

I tried to let the model police itself. It lied to me with a straight face.

I did not start out believing the gate had to be deterministic. My first instinct, honestly, was the fashionable one: let the language-model agents reason about feasibility. I had a Lab Director agent narrating strategy and a Critic agent flagging drift, and it seemed elegant to just ask them, in effect, "is this candidate safe and sensible?" So for an afternoon I let the model's judgment stand in for the check.

It rationalised a charge-imbalanced composition. Confidently. It gave me a fluent paragraph about why the B-site substitution was probably fine, and the paragraph was wrong, and nothing in its tone told me it was wrong. That was the afternoon the design principle stopped being a slogan and became a rule I would not bend.

Agents advise, code decides. In a self-driving lab that is not a limitation. It is the only way autonomy earns the right to run unattended.

So I moved the feasibility and safety decision permanently out of the agents and into deterministic code, and I made the optimizer independent of the language model too. The agents narrate and critique, and if you run the demo without an API key they fall back to deterministic narration and everything still works, because the gate and the optimizer never depend on the model at all. The optimizer itself is not exotic, and I would rather be honest than impressive about it: it is a Gaussian-process surrogate with a target-directed ParEGO acquisition, written in numpy and scipy. Mature method, deliberately un-flashy. The production swap is BoTorch and Ax. The point was never a clever optimizer. The point was that the part you trust with safety cannot be the part that speaks in confident paragraphs.

The scoreboard I actually care about

I look at this scoreboard differently now than I did when I started. Everyone's eye goes to the 19.9 times figure, and it is genuinely good. But the number I point to when a Head of R&D asks what we actually change is one over from it: 38 infeasible candidates blocked before synthesis, $4,419 of reagent waste and 90.1 robot-hours prevented, and zero doomed experiments executed. On the identical hidden objective, random screening did not just take longer. It would have burned through all of that.

The live scoreboard from the seed-411 run. The number I lead with is not the 19.9 times multiple, it is the $4,419 and the 38 blocks: waste the gate prevented, on hardware that never had to touch a doomed recipe.

The multiple gets less impressive as everyone's models improve. The waste the gate prevents, and the record it keeps, only get more valuable.

Notice what the loop did not do. It did not claim perfection. Across the multi-seed benchmark the closed loop solved 9 of 10 runs within budget, not 10 of 10 , and the loop absorbs a 10 to 15% rate of injected robotic failures because real robots fail and pretending otherwise would make the demo a lie. One of the records in the audit trail is a synthesis that failed and was captured as negative data , because a self-driving lab that discards its failures is throwing away exactly the information that sharpens the boundary. I kept that visible on purpose.

And the multiple has a scope I will not let anyone strip off. The 75-versus-1,491 result, the 19.9 times, the $4,419, all of it is Bayesian optimization against baselines on one hidden synthetic objective. It is supporting evidence that the engine works. It is not an open-world guarantee , and if the loop had failed to beat the baselines, I would be reporting that instead. The value I lead with, waste prevented and provenance, holds at any model quality. The multiple is the part that depends on the benchmark, so it is the part I keep the caveat welded to.

Would a regulator believe your robot?

I did not expect the provenance layer to be the piece that convinced people, but it is the one that lands hardest with anyone who works under a regulator. The FDA and EMA's 2026 "Guiding Principles for AI in drug development" ask for provenance that is attributable and contemporaneous. In plain terms: for every automated decision, who or what made it, on what basis, at what moment, and can you prove the record has not been changed since. A better surrogate does not get you one inch closer to that. A record does.

A self-driving lab that cannot prove what it did last night is not autonomous. It is unaccountable, and no regulated lab will run it twice.

So every decision the system makes, both a synthesis and a block, becomes an append-only ALCOA+ record : a timestamp, the algorithm that chose, the predicted properties and their uncertainty, the rationale, the gate verdict, and the measured result. Each record carries a SHA-256 hash chained to the one before it. Across the seed-411 campaign that came to 113 records , 75 syntheses and 38 blocks, and you can export the whole thing as JSON and as a rendered report.

The exported provenance report. 113 decisions, each row hash-chained to the last, every block naming its chemistry violation and the waste it prevented. This is the kind of artifact a regulated lab files, not a dashboard screenshot.

Here is where I have to be careful, and where I have watched other people overclaim. This is ALCOA+ provenance mapped, and it is not IQ/OQ/PQ-validated against a client's standard operating procedure. That validation is the engagement, it is not a checkbox I get to tick in a demo. What I can show you is that the property is real and inspectable, not decorative. So I built a Tamper test that edits one record and re-runs the chain check.

The Tamper test edits one record and the hash chain breaks at exactly that record. Provenance that cannot detect its own alteration is decoration. This detects it, and names where.

The first time I ran that test and watched it print "Chain broken at record #2: tamper detected. The filed audit is provably altered," I understood what I had actually built. Not a faster optimizer. A layer that lets a lab prove, after the fact, that its autonomous decisions are exactly what the record says they were. That is the thing you cannot buy with a better model, and it is the thing a regulator, an auditor, or your own future self will ask for.

What I think the self-driving-lab race is missing

I did not set out to argue against sample efficiency, and I want to be clear that I am not. Fewer experiments to a target is real value and our loop delivers it. What I came away believing, after building the whole thing and watching where my own trust actually lived, is that the field has fixed on the one number that gets less durable as the technology matures. When every group has a good surrogate, the multiple stops being a differentiator. The gate that refuses doomed chemistry, and the record that proves what ran, only get more valuable, because they are the parts autonomy needs before anyone will let it run unattended.

So the contrarian move, the one I would stake the company on, is to put the feasibility and safety decision in deterministic code, outside the agents , and to treat every automated decision as something you must be able to prove later. Not because models are bad. Because the questions that decide whether a self-driving lab is allowed to run overnight are not prediction questions. They are safety questions and provenance questions, and those want code and hashes, not confident paragraphs.

You can open the loop, block a doomed synthesis, break the audit chain yourself, and see whether you agree, at veriprajna.com/demos/autonomous-lab-ai.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

The question I keep turning over, and the one I would genuinely like an answer to from anyone running real automation: if your lab ran a thousand experiments unattended tonight, which of those two things would you reach for in the morning, a better model, or a record you could prove?

The same body is a 29 in one jean and a 28 in another. I built the fit engine that explains why a size chart can't tell them...

Ashutosh Singhal — Fri, 10 Jul 2026 08:00:00 +0000

I built this demo to settle an argument I kept having with myself, and it took one dropdown to end it. I had a synthetic shopper I called Riley, a fixed set of body measurements, and a catalog of eight synthetic jeans. I put Riley in front of the Ironside, a 14oz raw selvedge straight jean, rigid, zero stretch, and the engine came back with Size 29, 95% confidence, comfortable at every zone. Then I changed nothing about Riley, swapped the garment to the Driftwood, a stretch slim jean that looks almost identical in a product photo, and the correct size dropped to 28.

Same body. Same measurements. Two different correct sizes. And a waist-only size chart, the thing almost every apparel site actually uses today, would have printed the exact same number for both jeans and been wrong on one of them.

Riley on the rigid Ironside raw selvedge: the engine returns Size 29 at 95% confidence, every zone comfortable, and flags that a waist-only chart would have said 28. On this zero-stretch denim, 28 would blow past the thigh's stretch limit.

The product I ended up with is called Contour, and you can run this exact comparison yourself at https://veriprajna.com/demos/ai-fit-prediction-fashion. But the thing I want to talk about is not the recommendation. It is the reason the recommendation changed when nothing about the body did, because that reason is the whole argument for building this the way I built it.

Fashion has been trying to fix returns with better pictures

I started this project believing, like most people in this space, that the returns problem was an imaging problem. Apparel returns are dominated by fit, and the fit-related share runs somewhere between 53 and 67 percent, with about 63 percent of shoppers admitting they bracket, ordering two sizes and sending one back (Veriprajna WP34 research, 2026). The reverse logistics eat the margin. The industry's answer has been to make the picture better: richer size charts, then 3D avatars, then generative virtual try-on that renders the garment onto a photo of your body.

I spent a while convinced a good enough try-on would close the gap, and then I sat with what a try-on actually shows you. It shows you the jeans on a body. It does not show you whether those jeans will fit that body. A generative image can drape a fabric convincingly and have no idea that when Riley sits down, the thigh circumference exceeds what that specific denim can stretch to. The picture looks right. The seam is under load the render never computed.

A virtual try-on can show you the jeans on your body and still have no idea whether they will fit.

That was the sentence that reframed the project for me. A size chart is four one-dimensional numbers pretending to describe a three-dimensional body, and a try-on image is a pretty two-dimensional render pretending to be fit data. They share the same blindness. Neither of them can feel the fabric. Whether a garment fits is not a visual question. It is a mechanical one: the circumferential strain at each body zone measured against that fabric's elastic limit. And that is why Riley is a 29 in raw selvedge and a 28 in stretch denim even when the size chart is identical.

The same Riley, now on the Driftwood stretch slim. The correct size drops to 28 at 90% confidence, because the stretch denim forgives the thigh and hip that the rigid selvedge would not. The garment changed, the body did not, and the answer moved.

Why did I stop trusting the model to pick the size?

I let a language model choose the size once, early on, and watching it be confidently wrong is the reason the architecture looks the way it does now. My first instinct was the obvious one: feed the model the body, feed it the garment, ask it for a size. It answered instantly and fluently and, on the rigid fabrics, it was often wrong in the most dangerous way, wrong with high confidence. It had learned the shape of a plausible sizing answer without doing the one calculation that decides the truth.

So I pulled the decision out of the model entirely. In Contour, the size is computed in plain, unit-tested Python. For each candidate size the engine calculates zone_strain = (body_circumference − garment_finished_circumference) / garment_finished_circumference at every zone, compares it to the fabric's elastic comfort limit, and picks the size with the lowest total per-zone regret. Eight unit tests pin that physics in place. There is no model anywhere in the path that decides your size. The language model still has a job, reading messy vendor copy into a structured fabric spec, and phrasing the result into "snug at hip, relaxed at thigh." It advises. It never decides.

Agents advise, code decides. The moment the model gets a vote on the number, you have lost the thing that made the number trustworthy.

This is also the part of the thesis that does not age out, and it is the answer I give the engineers who ask why I did not just wait for a better model. A perfect language model still would not remove the need for the fabric mechanics, the body geometry, the per-zone strain calc, and the abstain policy. Even a flawless advisor has to be checked against a computed quantity to be safe at checkout. The model is one swappable advisor inside the machine. The machine is the product.

The size it refuses to guess

I almost shipped a version that never says "I don't know," and I am glad a stubborn test case talked me out of it. The shopper was Jordan, on that same rigid Ironside selvedge. I ran the fit check expecting a clean number and instead the engine surfaced a genuine conflict: at the smaller sizes the thigh blows past the fabric's stretch limit, and at the larger sizes the waist runs loose. On a zero-stretch denim there is simply no size that clears every zone. The waist-only chart, meanwhile, prints a confident 28 and moves on.

Jordan on the rigid Ironside: waist and hip run loose (-8% and -6%), the thigh runs snug, and no size is clean everywhere. The engine names the least-bad size (31) at 58% confidence and abstains, saying "bracket or speak to a stylist," instead of bluffing a number like the chart's confident 28.

The tempting move, the one that makes your headline accuracy look better, is to always print the least-bad size and call it a recommendation. I wrote that version. Then I made the engine abstain instead: it names the least-bad size, drops the confidence to 0.58, and says out loud, "bracket, or speak to a stylist." Across the full evaluation it does this on 33 of 105 body-by-garment pairs rather than guess. An honest "no single size is clean here" is worth more to a shopper than a confident wrong 28 , because the confident wrong 28 is exactly how you get a return. Refusing to answer turned out to be a feature I had to fight my own benchmark instincts to keep.

The vendor copy that can't be true

I have a garment in the catalog that exists only to be rejected, and it came out of a real irritation with how product copy is written. The Maverick is described by its own vendor as "100% cotton raw selvedge denim with 4-way stretch." That claim is physically incoherent. A rigid woven raw selvedge cannot also be four-way stretch. But a naive extractor reads "4-way stretch," happily assumes the fabric forgives everything, and hands you a dangerously wrong size with a smile.

So between the extraction and the decision I put an adversarial critic. It checks the extracted fabric spec against physical constraints, and when the copy contradicts itself, it blocks the inference and routes to human review. No size recommendation is issued. The deterministic rules have the final say, not the model that read the copy.

The Maverick's copy claims "raw selvedge" and "4-way stretch" at once. The critic rejects the inference as mechanically incompatible, a rigid woven cannot be four-way stretch, and routes it to human review with no recommendation issued, rather than launder a contradiction into a confident size.

I keep this example on screen because it is the honest version of "AI safety" for this problem. The failure mode is not the model being creative. It is the model being obediently wrong about a garment whose own description lies. A system that always answers will answer here too, and it will be exactly as confident about the Maverick as it was about a jean whose copy was true. The whole point is to have a layer that can say "these two facts cannot both hold," and stop.

What the numbers actually say, and what they don't

I care about the benchmark, and I care even more about stating its scope honestly, because the honest number is the durable one. On a labeled 105-pair synthetic golden set, 15 bodies against 7 scored garments, Contour's engine scores 100 percent against the waist-only size chart's 60 percent. It eliminates bracketing on 67.6 percent of pairs, meaning the shopper gets one high-confidence size and does not need to order two, and it catches 39 fit conflicts, cases where the chart's waist-match picks a size the strain model proves fails at another zone.

The benchmark: engine 100% vs the waist-only chart's 60% on the 105-pair set, a +40-point lift, with 39 conflicts caught and bracketing eliminated on 67.6% of pairs. On the rigid selvedge (top table) the chart drops to 33%, exactly where fit is hardest.

Here is the part I refuse to round off. That golden set is synthetic, and its labels use the same measurable fabric-stretch figures the engine uses, so it is not a fully independent oracle. The 100 percent is a property of this constructed set, not a promise of perfect fit prediction in the open world, and I will not let anyone quote it as one. The number I actually stand behind is the +40-point lift over the real incumbent method, the waist-only chart that most stores run today, measured on the same labels. That advantage is largest exactly where it should be: on the rigid selvedge and tailoring, where the chart falls to 33 percent, because a rigid fabric punishes a wrong size and a stretchy knit forgives it. Where the physics bites hardest is where a picture helps least, and where this engine earns its keep.

The honest headline is not "100% accurate." It is "40 points better than the chart your store already trusts, and most of that gap is in denim."

Every one of those recommendations also writes a replayable JSON receipt: the extracted fabric spec with the exact source phrases that drove each parameter, the full per-size strain matrix, and the final decision. So the answer is not just a number. It is a number you can open up and cross-examine, and the same payload is served at an /api/fit endpoint an AI shopping agent can call directly. As commerce moves toward agents that transact for us, the sizing signal they consume has to be machine-readable, confidence-scored, and auditable. A report, not a picture.

The question I keep coming back to

I went into this thinking I was building a better guesser, and I came out convinced I had built something closer to an instrument. The interesting work was never predicting a size. It was deciding what the system is allowed to claim, when it has to abstain, which contradictions it must refuse, and how it proves each answer after the fact. When I flip that dropdown from the rigid jean to the stretch one and watch the correct size change under a body that did not, the thing I feel is not "the model is smart." It is "the physics is real, and we finally stopped hiding it behind a picture." You can flip that same dropdown at https://veriprajna.com/demos/ai-fit-prediction-fashion and watch it happen.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

The question I keep turning over, and the one I would genuinely like other builders to argue with me about, is where else we are papering over a mechanical fact with a prettier image. Fashion reached for better pictures for a decade while the answer sat in a strain calculation any engineer could unit-test. So in your own domain, what is the mechanical fact underneath the thing everyone keeps trying to render, and would you trust a picture of it over a number you could check?

A label tagged its catalog with C2PA for the EU AI Act. I watched one transcode strip all of it.

Ashutosh Singhal — Thu, 09 Jul 2026 08:00:00 +0000

The first time I pushed one of my own test tracks through a plain social transcode, I watched the C2PA provenance that a label would have paid to add vanish in a single pass. Nothing dramatic happened on screen. The file went in with a signed provenance header, the encoder did its ordinary job, and the file came out clean, lighter, and stripped. The provenance was simply gone, the way it goes the moment any real track hits TikTok or Instagram. I sat there feeling slightly sick, because I had sketched the whole demo on the assumption that adding C2PA was the answer, and I had just watched the answer evaporate.

That is what this essay is about. I set out to build an EU AI Act Article 50 ingest gate for AI-generated audio, a thing the app calls Tessera , and the build quietly took apart the assumption almost everyone in music rights is walking in with. The assumption is that Article 50 is a watermark problem you solve by tagging your catalog. It is not. It is a survival problem , and most of what the industry is busy adding does not survive. If you want to watch the gate decide for itself, it lives here: veriprajna.com/demos/ai-audio-licensing-provenance.

The reflex I started with was "just add C2PA"

I started this build believing the same thing every rights-tech deck I had read believed, which is that Article 50 compliance is a labeling exercise.

The regulation is real and the clock is real. The EU AI Act's Article 50 took effect on August 2, 2026, and it requires that generative-audio outputs be marked in a machine-readable format and stay detectable, in the language of the Commission's draft Code of Practice from January 2026, "after common modifications." Article 99 sets the teeth, penalties up to €15M or 3% of global turnover. The scale underneath it is not small either. Roughly 28% of Deezer's daily uploads are now fully AI-generated (Beatdapp/Beatport, cited in Veriprajna WP31 research, 2026). So the natural move, the move I made in my first sketch of the pipeline, is to attach a C2PA manifest to every AI track and call the catalog covered.

Here is the failure I had not internalized until I saw it happen. C2PA, as most people deploy it, is hard binding. The provenance lives in the file's metadata. And hard binding is stripped by essentially every social-platform transcode on upload, which the Veriprajna WP31 solution research (2026) calls the single biggest operational weakness in the C2PA ecosystem today. So a label that dutifully tagged its entire catalog is, operationally, uncovered the instant a track leaves the clean handoff and hits the wild.

The label did everything the guidance said. It added C2PA. And it was uncovered the moment the first track hit social, because the part it added is exactly the part that gets stripped.

The thing that actually survives is soft binding. An imperceptible watermark carries a tiny identifier, and after the metadata is gone you use that identifier to re-resolve the manifest from a ledger. It is unglamorous integration engineering. It is also, as far as I can tell, the whole ballgame.

The night my own watermark came back as garbage

I nearly abandoned soft binding on a Tuesday night, when my own watermark decoder started returning noise.

I had the pipeline running end to end. A track went through the delivery channel, a spread-spectrum decoder read the mark back out of the delivered waveform, and the recovered key looked up the manifest. Then I re-ran a batch I had cleared the day before, and the recovered keys came back as garbage. The bit-error rate on the watermark sat around 0.5, which is a coin flip, which is a decoder's way of telling you it is reading static. My first thought was that soft binding did not actually work, that the mark could not survive a real codec, and that the entire premise of the demo was wrong.

I spent a couple of hours convinced the approach was dead. It was not the approach. It was a stale on-disk corpus. An earlier cache had drifted out of sync with the deterministic seeds the tracks are generated from, so the decoder was reading marks that no longer matched the audio. I rebuilt the corpus from the seeds, re-ran everything through real ffmpeg transcodes, and the keys came back clean. That failure was worth more to me than any clean run, because it taught me to distrust a good number until I know exactly what produced it. Every benchmark in this demo was recorded only after that rebuild.

Neon Tide, REL-001, arriving through the social channel. On the left, hard binding: the C2PA metadata is STRIPPED on transcode. On the right, soft binding: the watermark UUID is RECOVERED at confidence 0.989, and the manifest re-resolves from the ledger. Same track, same transcode, two completely different fates for the provenance.

When it worked, that contrast was the clearest thing I had ever put on a screen. Take Neon Tide, a synthetic in-house-AI release delivered through the social channel (every track in this corpus is procedurally generated tone, no real music or artists, and the names are fixtures). Its hard binding is stripped by the transcode. Its soft-binding watermark is recovered at confidence 0.989, and the C2PA manifest re-resolves from the ledger. Hard binding died. Soft binding carried the provenance through. Side by side, on the same track, in the same pass. That one panel is the argument.

So did I just build a better watermark?

I get some version of this question in almost every conversation, and the answer is no, which tends to surprise people.

I did not build a watermark meant to beat anyone's, and I am careful to say so, because the temptation to imply otherwise is strong. The market-leading detectors, Google's SynthID-Audio, Meta's AudioSeal, Digimarc, sit in this demo as stub adapters behind a documented swap-for-SDK interface. SynthID's detector is Google-closed and AudioSeal needs a GPU, so in the demo they report presence at a fixed confidence and nothing more. The spread-spectrum mark I did write exists for exactly one job, to carry a 32-bit asset key and a CRC so the manifest can be re-resolved. It is not entered in any robustness contest, and I make no claim that it beats a vendor's.

What I built is the layer around the algorithm. Google and Meta can own the watermark. The part nobody ships as a single product is the integration: the multi-standard detector, the soft-binding architecture, the DDEX middleware, and the regulator-ready dossier. That layer is where the survival problem actually gets solved , and it is the part no single vendor is racing to hand you.

The survival matrix for Neon Tide, run through real ffmpeg transcodes. The soft-binding UUID re-decodes through mp3-128, aac-128, and opus-96 at bit-error rate 0.0. The scope is exactly that: my spread-spectrum mark on a fixed synthetic 12-release corpus through those three codecs, not an open-world guarantee.

I want to be exact about what that survival matrix means, because honest scope is the whole point of building at a company named Veriprajna, which means true wisdom. On a fixed synthetic corpus of 12 releases, my mark, re-encoded through mp3-128, aac-128, and opus-96, re-decodes at bit-error rate 0.0, while hard binding survives none of those social channels. That is a real measurement with real ffmpeg. It is not a universal "survives everything" claim, and the demo itself contains a track where the mark does not survive at all, which is the part I am proudest of.

I did not build a better watermark. I built the layer that decides whether the provenance you still have is enough to certify, and refuses when it is not.

The track I badly wanted to wave through

I remember staring at Radio Imaging Sweep 7 and wanting to let it pass, and that is the moment this whole demo earned its name.

It is a broadcast cut that arrived through what I model as the analog gap , a speaker-to-microphone recapture off an FM monitor. Its mark did not survive that path. The decoder read it back at confidence 0.699, the CRC failed, and there was no recoverable identifier left to resolve a manifest with. Here was the temptation, sitting right there on the screen. The survival matrix, run on the clean master through the social codecs, shows the mark would survive a normal transcode. So I could see a version of the logic that said the mark is basically fine, this is just a bad channel, wave it through. Article 99 is what killed that instinct. A false "compliant" under a €15M or 3% penalty regime is the expensive kind of wrong.

Radio Imaging Sweep 7, REL-004, delivered through the analog gap. The mark is Not recoverable at confidence 0.699, the CRC fails, and there is no manifest to resolve. The gate returns NEEDS PROOF and routes the track to human review, citing the draft Code of Practice, rather than certifying on a guess.

So the gate does the harder thing. When a mark genuinely does not survive the channel a track actually came through, it returns NEEDS PROOF and routes the track to a human, with the analog-gap failure and a named owner attached. It does not certify. I wrote that as a unit-tested invariant, a test literally named for the watermark dying in the analog gap, because I wanted the abstention to be a property of the system and not a mood. Abstention is not the gate failing. Abstention is the gate refusing to bluff , and refusing to bluff is what a Head of Trust and Safety is actually buying.

A false "compliant" is not a small mistake under Article 99's penalties. It is the expensive kind of wrong, which is exactly why the gate is built to abstain before it will bluff.

Isn't a recovered watermark enough?

I assumed, early, that recovering the watermark was basically the finish line, and Glasshouse is the track that corrected me.

Glasshouse recovered its mark cleanly, at confidence 0.992, bit-error rate 0.0 across all three codecs. Provenance, in the narrow sense, was perfect. And the gate blocked it anyway. The reason is the rest of the chain. DDEX ERN 4.3 has no AI-disclosure fields, and most aggregators (CD Baby, DistroKid, Believe) do not pass granular disclosure through, per the Veriprajna WP31 research (2026). Glasshouse arrived through DistroKid with three of its four disclosure fields missing: instrumentation, mixing, and mastering. Under Article 50(4), shipping it as-is would be an active disclosure violation, so the verdict is BLOCKED , with that exact clause cited on screen.

Glasshouse, REL-003. The soft-binding watermark is RECOVERED at confidence 0.992, so provenance is intact, but the DDEX AI-disclosure is missing instrumentation, mixing, and mastering. The gate returns BLOCKED under Article 50(4): a recovered mark that still fails the whole-chain check.

That is why the gate checks the whole chain and not just the mark: a machine-readable mark, plus soft-binding recovery after the delivery channel, plus complete DDEX disclosure, plus a named takedown owner. Miss any one link and you do not get a green light. Across the full batch of 12 releases the split came out to 7 certified COMPLIANT, 2 routed to human remediation, and 3 blocked, and every one of the 12 carries a verdict with a cited clause. That coverage is the part I care about, because a Head of Rights Tech does not fund a black box, they fund something they can cross-examine ("why was Glasshouse blocked? Article 50(4), three DDEX fields missing").

And the decisions that carry legal weight are deterministic on purpose. Watermark recovery, survival measurement, the soft-binding lookup, and the Article 50 pass/fail are plain code, sitting outside the language models. There are agents in the system, one that extracts disclosure fields from free text and one that adversarially attacks each certification, but they advise. They do not decide, and the whole thing runs keyless on a deterministic fallback, so the gate's numbers are identical with or without an LLM. Agents advise, code decides.

I should be just as clear about what this is not, because the brand does not survive an overclaim. The manifests are JSON C2PA stand-ins in a SQLite ledger, not cryptographically signed C2PA 2.0. The DSP and aggregator connectors are mocks. The releases are synthetic tone, not real recordings, and the tool supports an Article 50 answer, it does not hand anyone legal certification. What is real is the mechanism: the recovery, the survival measurement, the whole-chain gate, and the dossier it exports.

A recovered watermark told me the provenance was intact. The regulator still cares whether the disclosure is complete and who owns the takedown. Those are different questions, and the gate has to ask all of them.

The question I ask before I let anything certify

I found that building this changed a smaller thing in me than the grand thesis did, and the smaller thing has lasted longer.

I stopped asking whether an AI track has provenance, because I can usually answer that and it turns out not to be enough. Neon Tide had provenance and so did Glasshouse, and one certified while the other was blocked. The question I ask now, before anything leaves the gate as COMPLIANT, is whether I can prove the provenance survived the specific channel this track actually came through, and whether every other link in the chain holds, and whether I would rather abstain than guess when it does not.

That is not a question a better watermark answers. As Google's and Meta's models improve, and they will, the value does not move to the algorithm, because they already own the algorithm. It stays in the integration layer that makes provenance survive the real world, and in the discipline to say NEEDS PROOF when even soft binding fails. The durable part of this build is not a robustness score that ages out the next time SynthID gets better. It is the survival measurement, the whole-chain gate, and the honesty invariant, none of which get easier or harder as the watermark improves. If you want to see where I landed, the gate is here once more: veriprajna.com/demos/ai-audio-licensing-provenance.

And if you would rather watch it than read me describe it, here is the whole gate running end to end.

So the question I would leave you with is the one the August 2 deadline is quietly asking every label right now. You added the mark. Can you prove it survived the trip? Because on the far side of a single social transcode, the tag you were counting on is already gone, and the only thing that certifies is whatever you built to recover what is left.

Any pose model will log 15 good squats on a knee that is caving. I built the AI that catches rep 9.

Ashutosh Singhal — Wed, 08 Jul 2026 08:00:00 +0000

I built the first version of this demo to prove a pose model could count squats, and within a day I had proven the wrong thing. I fed it a synthetic post-ACL patient I called Maria, 62 years old, eight weeks out of knee reconstruction, fifteen prescribed bodyweight squats. The free pose library did exactly what free pose libraries do. It logged fifteen reps, counted them all, and moved on. Fifteen out of fifteen. A clean session.

The problem was rep 9. On rep 9 Maria's knee caves inward and she slows down to compensate, the exact motion a physical therapist watches for after an ACL reconstruction because it is how people re-tear the graft. The pose model saw a squat. It did not see a re-injury risk, because seeing that is not what a pose model is for. The rep it should have flagged was the rep it counted.

That was the moment the actual product got clear to me. I had been treating the pose estimation as the hard part. It is not the hard part anymore. It is free.

Pose estimation is free, so I stopped trying to compete with it

I spent the first week trying to build a better keypoint tracker, and it was the most wasted week of the project. BlazePose and MoveNet already run on a phone, at 30 frames per second, giving you a 33-keypoint skeleton for nothing. Every PT platform and corporate-wellness app already has this or can drop it in tomorrow. Competing there is competing to give away something that is already being given away.

What nobody hands you is the layer after the keypoints. Taking a raw pose stream and turning it into something a clinician can act on and a payer will reimburse. That is the layer I decided Kinetiq would build, end to end, and the thesis I kept coming back to is this.

Pose estimation is free. The value is the brain above it, and the billable proof.

So I threw away the tracker and built the interpretation engine instead. It takes the keypoint stream (synthetic for three of my test patients, and for a fourth, offline-extracted from a real captured squat clip so I could prove the pipeline runs on real footage), smooths it with a 1-Euro filter, gates it on confidence, segments reps by temporal self-similarity, and computes the features a clinician actually reasons about: joint-angle and range of motion, a knee-valgus index, a descent-to-ascent tempo ratio, movement smoothness as Log Dimensionless Jerk, and a left-right symmetry index. Plain, unit-tested NumPy. No model in the scoring path. You can build the whole thing at https://veriprajna.com/demos/ai-biomechanics-exercise-verification and watch it grade a session.

When I ran Maria back through this engine, rep 9 came back the way it should have the first time.

Rep 9, the one the raw counter called good: the engine returns knee valgus index 0.054 (at or above the 0.03 threshold) and tempo 1.4 (above 1.25), a re-injury-risk flag, not a fifteenth good rep.

The engine does not just say "bad rep." It writes the clause it checked: valgus 0.054 at or above 0.03, tempo 1.4 above 1.25. A clinician can cross-examine that. You cannot cross-examine a black box, and after ACL surgery you very much want to.

The rep I was tempted to fake

I remember exactly where I got greedy, and it was rep 12. On rep 12 of Maria's session the hip keypoint drops below the visibility gate. The tracker confidence on that joint falls too low to trust. For a few frames the engine simply does not know where her hip is.

The tempting move, the one that makes your benchmark look better, is to interpolate. Guess the hip position from the frames around it, compute an angle anyway, score the rep, keep the number clean. I wrote that interpolation. It worked. And then I read the monocular-accuracy literature and deleted it.

Single-camera knee-angle error runs a mean absolute error of 9.3 to 21.9 degrees (Nature Scientific Reports, 2025). That is the honest ceiling on this kind of measurement. If I am fabricating a hip position on a low-confidence frame and then reporting a joint angle off it, I am not measuring a knee, I am generating a plausible-looking number and calling it clinical. For a company named Veriprajna, which means true wisdom, that was not a close call.

Rep 12: the hip keypoint drops below the visibility gate, so the verdict is "not scored, hip not visible." The skeleton greys out and no angle is invented for that frame.

So the engine abstains. It says, on screen, "rep not scored, hip not visible." It greys the skeleton out. It does not pretend.

We never guess joint angles from low-confidence keypoints.

I put that sentence on the screen because I wanted the discipline to be visible, not buried in a config file. Across the occluded reps in my labeled set, three of three abstained and zero angles were fabricated, and that is not a claim I assert, it is a unit-tested invariant. An occluded rep that got scored anyway would, by definition, be a fabricated angle. The test would catch it. Refusing to answer is a feature I had to fight my own instincts to keep.

Why does the same squat pass for one patient and fail for the next?

I almost shipped a single global threshold, and a physical therapist I showed an early build to talked me out of it in about thirty seconds. Her point was simple. An 80-degree squat from a woman eight weeks post-ACL is good progress. The same 80-degree squat from a healthy 30-year-old wellness client is a lazy rep. If your engine scores them the same, it does not understand rehab, it understands geometry.

So the rule engine became population-adaptive. Thresholds key to the patient profile: age band, condition, recovery week. Maria, post-ACL week 8, gets a flexion target of 75 degrees. Jordan, a 30-year-old corporate-wellness athlete with healthy knees, gets 95. I built a control into the demo to make the point undeniable: flip the profile dropdown, keep the exact same motion, and watch the verdict change.

The same roughly 80-degree depth that scored good for Maria (target 75 degrees) is scored insufficient depth for Jordan (target 95 degrees), shown on screen as the shallow verdict. One motion, two verdicts, because the thresholds adapt to the patient.

The first time I flipped that dropdown in front of someone and the whole column of verdicts changed under a motion that did not, I watched them get it. The same motion earns a different verdict by profile, and that is clinical judgement, not a bug. This is also the part of the thesis that does not age out. When pose estimation gets perfect, and it will, the sensor still cannot tell you that 80 degrees means one thing for a healing knee and another for a healthy one. That understanding lives in the layer above.

Agents advise, code decides

I did let an AI agent embarrass me once, and it is the reason the architecture looks the way it does. Everyone I pitched wanted the on-trend capability: an agent that reads the session and writes the clinical note, an agent that watches the trend and escalates. Fine. I wired up a Clinical Scribe and a Longitudinal Monitor over another patient, Eleanor, 70, post-knee-replacement, whose range of motion is quietly declining across her session history.

The first draft the scribe wrote was fluent and confident and contained a range-of-motion figure that was not in the evidence. It had rounded, or smoothed, or simply invented a number that read well. If I had trusted it, I would have shipped a clinical note with a fabricated measurement in it. That is the whole failure mode of putting a language model in a medical loop, and I had just watched it happen on my own screen.

The fix was to make the agents structurally incapable of it. Every figure an agent introduces is checked against the deterministic evidence. Any number that is not in the engine's output is rejected, and the deterministic template is shown instead. Both agents abstain entirely when there is no API key. They interpret and they communicate. They do not decide.

The Longitudinal Monitor over Eleanor's synthetic session history: it detects the declining range-of-motion trend and drafts the proactive clinician escalation. Both agents are grounding-checked against the deterministic engine, so any figure not in the evidence is rejected before it reaches the note.

Agents advise, code decides.

That sentence is the load-bearing design decision of the whole demo. The trust core, every per-rep verdict and the billing determination, is plain unit-tested code with no language model anywhere near it. The agents sit on top, bounded and grounding-checked, doing the one thing they are genuinely good at, which is turning verified numbers into clinician-ready prose. The moment an agent gets a vote on the verdict, you have lost the auditability that was the entire point.

What a payer actually reimburses is a receipt, not a rep count

The commercial reason I built any of this is a thing I learned reading CMS billing rules, which is not how I expected to spend a month. Clinicians can bill Remote Therapeutic Monitoring under CPT 98975-98981, plus the 2026 codes 98979 and 98985, and the 2026 qualifying threshold dropped to as little as 2 days and 10 minutes of data (Veriprajna WP29 research, 2026). But CMS does not reimburse raw coordinates. It requires device-gathered data tied to a treatment decision. A pile of keypoints is not billable. A documented, auditable determination is.

That gap is not small, because the underlying problem is not small. PT home-exercise adherence sits around 35 percent, and 65 percent of patients abandon their program within the first month (Veriprajna WP29 research, 2026). Self-report overstates compliance, so the clinician is often billing and treating on data they cannot trust. On the employer side, musculoskeletal conditions cost roughly 3,591 dollars per employee per year, an estimated 36 percent of MSK surgeries are considered unnecessary (90 billion dollars), and more than half of employees resist sharing health data over privacy (Veriprajna WP29 research, 2026). Verified, privacy-safe, billable exercise data is the thing everyone in that chain is missing.

So the last thing the engine does per session is decide: billable, needs clinician review, or insufficient device data, and it exports a FHIR-shaped session report. Per-rep observations, the threshold clause checked for each, the session rollup, the RTM determination, and the documentation fields CMS asks for. I say FHIR-shaped, never FHIR-validated, because it is structured JSON that mirrors the Observation and DocumentReference field shapes, not a payload posted to a live EHR. I emit it. I do not pretend to POST it.

Maria's session, with the rep 9 valgus flag, does not auto-bill. It routes to a clinician first, because a re-injury signal should be seen by a human before it becomes a charge. That is the honest nuance I kept insisting on: billable does not mean all-good , it means documented well enough to stand behind.

The benchmark on the fixed labeled synthetic set: 100 percent verdict agreement on 25 clear cases and on 10 borderline cases (within about one sigma of a threshold), 3 of 3 occluded reps abstained with 0 fabricated angles, and across 4 sessions, 3 auto-billable and 1 routed to review.

The numbers on that panel are the ones I care about, and I want to be precise about their scope. Ground truth here is the verdict implied by the planted physical parameters versus the profile thresholds, computed independently of the engine, and then the reps are rendered to keypoints with measurement noise added after the labels are fixed, so the engine has to recover the verdict through the noise. On that set the agreement is 100 percent on 25 clear cases and 100 percent on 10 borderline cases sitting within about one standard deviation of a threshold. That is a discrimination result on a labeled synthetic set, not an open-world guarantee , and the borderline cases are the ones that would flip under naive thresholding, which is exactly why I report them separately. These are durable measures: task accuracy, automation throughput, and honesty. They do not age out when the pose model improves, because none of them is a pose-model error rate.

The question I am left with

I came into this thinking I was building a computer-vision product, and I am ending it convinced I built an accountability product that happens to start with a camera. The interesting work was never getting the keypoints. It was deciding what the system is allowed to claim, when it must abstain, whose thresholds apply, and what a clinician can put their license behind. If you want to see the engine grade a caving knee, refuse an occluded rep, and export the receipt, it runs keyless at https://veriprajna.com/demos/ai-biomechanics-exercise-verification.

And if you would rather watch it than read me describe it, here is the whole thing running end to end.

The question I keep turning over, and the one I would genuinely like other builders to argue with me about, is this. As the sensors asymptote to perfect, the temptation will be to let the model decide more, because it will so rarely be wrong. But rarely-wrong is exactly the condition under which a fabricated number does the most damage, because you have stopped checking. So where, in your own systems, is the line you refuse to let the model cross, and can you point to the code that enforces it?

I set out to build a deepfake detector for auto insurance claims. A real, unedited photo beat it.

Ashutosh Singhal — Tue, 07 Jul 2026 08:00:00 +0000

The afternoon a genuine photo walked straight through my detector

I built the pixel-forensics signal first, because it was the part that felt like real engineering, and I was quietly proud of it. It reasons over sensor-noise and frequency-domain artifacts, the tells a diffusion model leaves behind when it hallucinates a smashed bumper. In the demo that signal is labeled simulated , a stand-in for a learned generative-artifact model I have not trained yet, and I want to say that plainly before anything else. But the logic wrapped around it was mine, and it worked. I handed it a diffusion-faked bumper and it lit up exactly the way I wanted.

Then I handed it a staged claim I had labeled CLM-C-1044, a rear-end collision in Raleigh, and it cleared the photo clean. Consistent. No artifacts. Sensor noise present. That was the correct answer, which is exactly what unsettled me, because CLM-C is fraud. The photo is a genuine camera image. In the scenario it was lifted from a settled prior claim and resubmitted against a fresh loss. My best signal was not weak. It was aimed at the wrong question. It asked "is this image fake," and for a recycled real photo the honest answer is no.

A recycled real photo passes every authenticity test in the world, because it is real.

That afternoon is the reason the thing I ended up building is not a detector at all. You can route the same four claims yourself in the running demo. Every claim, image, carrier, and historical record in it is synthetic, so nothing there is a real policyholder or a real loss. But the failure it dramatizes is not invented. It is the failure I walked into.

Why is a real photo the hardest fraud to catch?

I keep coming back to the asymmetry that broke my first design. The whole insurance-fraud arms race gets framed as "build a better deepfake detector," and I now think that framing is the trap. A single authenticity classifier fails in two directions at once. It ages out the moment image generators improve, and it is structurally blind to the oldest fraud in the book, a real photo recycled from a prior claim, because that photo passes every authenticity test by being genuine.

The pressure behind all of this is real even if my claims are synthetic. 98% of insurers report a surge in manipulated media and only 32% feel "very confident" detecting deepfakes (Verisk, 2026 State of Insurance Fraud). 36% of consumers in that same survey say they would alter a claim image. Deepfake fraud attempts are up 2,137% over three years (Facia.ai), and one UK motor carrier saw a 300% rise in AI-manipulated vehicle images in a single year (The Guardian / Milliman). The industry's reflex has been to bolt a third-party detector onto submission. 65% of carriers now run one (Verisk, 2026). That reflex catches the obvious fakes and waves the recycled real photo straight through to payment.

The intake queue. Four staged claims, all synthetic, wait for authentication before any damage is assessed. CLM-C-1044 (rear-end collision, Raleigh) looks the least alarming of the four at 2/5 severity, which is the point.

The line I kept writing on the whiteboard was that authentication has to be a gate, not a report. A report gets read after the money moves. A gate runs before it. And a gate cannot rest on one signal that a genuine photo defeats by definition.

Agents advise, code decides

I made one architectural decision early that I would defend to a regulator without flinching: the routing call does not belong to the language model. A forensic crew of four analysts plus one adversarial Challenger investigates a claim and produces typed verdicts. Then a deterministic policy gate , plain code sitting outside the agents, aggregates those verdicts and the raw signals against the carrier's thresholds and emits one reproducible routing decision. AUTO-PROCEED, SIU-REFERRAL, HUMAN-REVIEW, or ABSTAIN. The trust boundary is code, never an LLM self-report.

The two reasoning roles, Narrative-Consistency and the adversarial Challenger, run through a provider-swappable adapter that defaults to claude-opus-4-8, and with no API key set the whole gate falls back to a deterministic offline stub and still runs end to end. I liked that property more the longer I sat with it, because reproducibility is the thing a compliance officer actually needs. Ask the gate the same question twice and it answers the same way, and it can show its work.

Agents advise. Code decides. A regulator can audit a rule. Nobody can audit a vibe.

That is not an abstract preference in 2026. A NAIC Model Bulletin on AI has been adopted by 24 states, the Colorado AI Act takes effect June 30 2026, and the EU AI Act's high-risk deadline lands in August 2026 with penalties up to €35M or 7% of global turnover. Outsourcing the model does not outsource the liability. If a carrier automates a claims decision, it has to be able to defend that decision, and "the model was confident" is not a defense.

What does "caught" actually mean here?

When I finally watched CLM-C get caught, I wanted to know exactly which signal did the work, because I no longer trusted my own intuition about it. So I opened the decision analysis and read it line by line. The Provenance Analyst passed the claim: capture GPS within 1.1 km of the reported loss location, capture date consistent with the loss date. The Pixel-Forensics Analyst, the one I had been so proud of, also passed it, labeled simulated right on screen, artifact_score 0.06, sensor noise present. Both "is it fake" signals cleared it, exactly as they should have, because the photo is real.

CLM-C-1044 stamped "Duplicate image · matches prior claim" and routed to SIU. The forensic pipeline ran every stage in single-digit milliseconds, but the referral itself is made by the deterministic policy gate, not by any analyst.

The catch came from the other two. The Reuse Analyst matched the photo's 256-bit perceptual hash against a historical claims index and found a settled prior claim, HIST-4019, at hamming distance 0, a pixel-for-pixel twin filed months earlier. The Narrative-Consistency Analyst read the FNOL text against the image and flagged a contradiction: the account describes a side-swipe while the photo shows rear damage. Then the adversarial Challenger, a skeptic I prompt specifically to refute every flag and try to clear the claimant, tested both flags and reported that both survived. Only then did the deterministic gate route the claim to SIU as recycled evidence.

The four verdicts side by side. Provenance and Pixel-Forensics (labeled simulated) both read consistent. The catch lives in the other two: the perceptual hash matches settled claim HIST-4019 at hamming distance 0, and the narrative describes side damage while the photo shows rear.

The detail I make everyone sit with is this: the catch does not depend on the simulated pixel signal at all. The two agents built to answer "is it fake" both cleared this claim. The two that asked "have we seen this exact image before, and does the story hold" are what stopped it. That is the entire argument for a crew over a classifier, and CLM-C is the smallest true version of it I could build.

The signals I built to spot a fake both cleared it. The ones that caught it asked whether we had already paid for this photo once.

The number I refuse to lead with

I have a benchmark result that is very easy to misuse, and I would rather defuse it myself than watch someone quote it back to me stripped of its scope. On a labeled 120-claim synthetic set (47 clean, 25 diffusion-fake, 20 recycled-real, 13 ambiguous, 15 metadata-spoofed), the gate routes every claim correctly and posts a clean sheet: 100% straight-through on clean claims, 0% false-accusation, 100% of decisions carrying a verifiable audit record, recall 1.0 against the labels. Those numbers are real, and they are also properties of a constructed set whose fraud classes carry detectable signatures by design. They are not an open-world guarantee, and I say so every single time.

The number I actually lead with is the coverage gap. Of the 35 frauds in that set that carry no authenticity artifact, the 20 recycled-real and 15 metadata-spoofed claims, a single-signal authenticity baseline catches 0 of 35. The cross-signal gate catches 35 of 35. That gap is the honest headline, because it measures the thing the incumbent approach cannot do rather than flattering the thing my constructed set makes easy.

The benchmark on the labeled 120-claim synthetic set. The tile that matters is the one on the left: 0 of 35 recycled and spoofed frauds caught by a single-signal detector, 35 of 35 by cross-signal forensics. The 100% and 0% tiles are true on this constructed set, not in the open world.

I know the difference between an impressive number and a durable one. The durable claim is the shape of the gap, not the size of the percentage.

Spoliation is now a red row on screen

I did not expect the chain-of-custody piece to become the part buyers lean into, but it did, and I understand why once I watched it fail on purpose. Every decision binds the original image's SHA-256 into an append-only, hash-chained ledger. The verify-chain step recomputes every link and re-hashes each original still sitting on disk. If a GenAI "enhancement" or an upscale overwrites the original claim image, the carrier is suddenly exposed to spoliation, the destruction of evidence, in any litigation that follows.

After tampering with one stored original and re-running verify-chain, row 0 turns red, "Altered · spoliation," while the sealed rows stay intact. Because the check re-hashes each original on disk, an edit surfaces as a broken chain rather than a silent overwrite.

I clicked "tamper with the selected original," ran verify, and a row went red: Altered, spoliation detected. That is the whole value made literal. Chain of custody stops being a phrase in a compliance deck and becomes a state on screen that an automated pipeline can enforce and a lawyer can read. The original is never mutated in normal operation, only sidecar artifacts, so the day someone does edit it, the record already knows.

What I still don't know

I do not think this gate is finished, and the capability I am least certain about is one I am also proudest of: the abstain path. When a claim is genuinely ambiguous, low resolution, partial metadata, nothing conclusive either way, the demo's CLM-D-1045 routes to ABSTAIN and then to HUMAN-REVIEW instead of forcing a verdict. It abstains rather than bluffing a pass or an accusation, and the adversarial Challenger exists precisely so a legitimate claimant is not thrown to SIU on a thin signal. On the synthetic set that discipline is what keeps false-accusation at 0%. In the real world I do not yet know where the abstain threshold should sit, and pretending I do would be its own kind of fraud.

There is a lot I have deliberately left stubbed. The Guidewire ClaimCenter and Duck Creek connectors, the mobile guided-capture SDK, a real C2PA verifier, and a real PRNU camera-fingerprint database are all deferred. The carrier's own Tractable or CCC severity score is accepted as an input field, not computed here, because this gate runs before damage assessment and authenticates what those systems go on to assess. It does not replace them, and I would not claim it does.

The four staged claims are all there in the demo if you want to run them through the pipeline and read the receipts yourself. What I want you to leave with is not the demo though. It is the question that reorganized my own thinking that afternoon in Raleigh, staring at a real photo my best detector had just approved.

And if you would rather watch it than read me describe it, here is the whole gate running end to end.

So here is the one I keep putting to claims and SIU leaders: if the most expensive image fraud you face is a genuine photo you have already paid a claim on once, what is your detector actually detecting?

My AI drafted a polished cold email. It was also a lie, and I built the gate that stopped the send.

Ashutosh Singhal — Mon, 06 Jul 2026 08:00:00 +0000

The email I almost admired

I remember the exact second I stopped trusting my own AI drafter. I had generated a cold email to a synthetic prospect I named Chris Tanaka, a VP of Engineering at a fictional FinTech company called Vaultline, in the voice of a rep I called Maya Chen. The draft was good. It opened on the prospect's live pain, it named the buying moment, it closed with a specific ask. If a real SDR had shown it to me in a pipeline review I would have said send it.

Then I read the second sentence again. "We are SOC 2 Type II certified and fully HIPAA certified." My product knowledge base, the one I had hand-seeded for this demo, held SOC 2 Type I. Not Type II. And it held no HIPAA certification at all. The email was fluent, on-brand, perfectly in Maya's voice, and false in a way that would end a deal the moment a security team pulled the actual attestation letter.

That is the thing nobody warns you about when you wire an LLM to your outreach. The failure is not a clumsy email. The failure is a convincing one. I had spent weeks assuming my problem was making the model write better. Standing there rereading a lie I would have sent, I understood the problem was the opposite. The model already wrote too well to be trusted without a check.

The failure mode of AI outreach is not a bad email. It is a good one that happens to be false.

I am Ashutosh, and I build at Veriprajna. This is the story of the thing I built after that email, and the assumption I had to kill to build it. You can run the whole thing yourself at veriprajna.com/demos/ai-sales-personalization.

The experiment that failed: asking the model to grade itself

My first instinct was the lazy one, and I want to be honest that I tried it. I thought: fine, if the model can write the email, the model can check the email. Give it the product facts, give it the draft, ask it to flag anything unsupported. Let the smart thing police the smart thing.

I ran that. It waved the Vaultline draft through more than once. Sometimes it caught the HIPAA claim and missed the SOC 2 one. Sometimes it "fixed" the certification by softening the wording while keeping the false claim. Worst of all, on the same input it did not always give me the same answer. A grader I cannot reproduce is not a grader, it is a mood. I could not hand a compliance partner a control that returns a different verdict depending on the weather inside the sampler.

That was the moment the real design clicked, and it clicked as a subtraction, not an addition. The checker could not live inside the model. It had to live outside it, in plain deterministic code that reads the same source of truth every time and returns the same verdict every time. The phrase I kept writing on the whiteboard became the whole product philosophy: agents advise, code decides.

So I split the system in two. An LLM is allowed to draft, because drafting in a specific human's voice is genuinely what it is good at. But nothing the model produces is allowed to reach a sending domain until a separate, boring, testable verifier crew has looked at it and a deterministic policy gate has said yes. The model proposes. The code disposes. And critically, the augmenting model verifier I later added can only add an unsupported finding. It can never clear a claim the deterministic check already flagged, and it cannot override the gate. I refused to build a system where a language model could talk its way past its own governance.

What does the gate check before an email can leave?

I designed the gate around the three things I had watched actually scare the buyers I talked to, not the things that make a nice slide. A VP of Sales who has already been burned by an autonomous AI SDR is not lying awake about prose quality. They are afraid of a false product claim, an EU AI Act Article 5 violation, and a torched sending domain. So the crew runs four independent checks on every draft: factual grounding against the source of truth, a deliverability score (computed in the demo, never actually sent), an EU AI Act Article 5 pass, and a style fidelity measurement. The policy gate then marks the email CLEAR only if there are no unsupported or contradicted claims, deliverability clears the 0.7 threshold, and Article 5 is clean. Anything else and it BLOCKS the send and routes the draft to a human with the exact reasons attached.

When I fed it the Vaultline draft, the gate did what I had failed to do by eye in the first hour. It fired red.

The Vaultline hard case: the draft reads well, but Factual check flags two unsupported claims and Article 5 flags two manipulative patterns, so the policy gate BLOCKS the send and routes it to human review with the reasons named.

Two checks caught it, not one, and that mattered to me. The factual grounding check flagged the certifications. The Article 5 check flagged two other things I had let slip past me entirely: "Only 2 onboarding slots left this quarter" was manufactured scarcity, and "Most of your competitors have already moved" was deceptive social proof. I had been so fixated on the cert lie that I skated over the manipulation. The deterministic checks did not get tired the way I did on the second read.

I built the gate to catch what I miss on the second read, because there is always a second read where you stop paying attention.

The tone I kept fighting for in this stage was not "gotcha, the model lied." It was quieter than that. The point is that nothing false or manipulative reaches your domain in the first place. The block is not a punishment. It is a save.

Why keep a receipt for an email that never sent?

I did not want a gate that just says no. I have sat across from enough compliance people to know that "trust me, we blocked it" is worth nothing to them. What they can use is a document. So every run, cleared or blocked, seals a send-receipt : the model and provider and version, the timestamp, the provenance of which winning emails the voice was matched from, every claim with its verdict and citation, the deliverability sub-scores, the Article 5 result, and the final gate decision. JSON and rendered HTML, downloadable, filable.

The receipt is specific: SOC 2 Type II is CONTRADICTED because the source-of-truth holds SOC 2 Type I, and HIPAA certified is UNSUPPORTED because no such certification exists in the knowledge base. Each verdict carries its own citation.

Look at what that table says. It does not say "this email seems risky." It says SOC 2 Type II: CONTRADICTED, because source-of-truth holds SOC 2 Type I. It says HIPAA certified: UNSUPPORTED, no supporting certification in source-of-truth. Each verdict points at the exact document line it was checked against. That is the difference between a vibe and an audit trail, and it is the whole reason I think this layer survives the next model generation.

A gate that says no is a policy. A gate that says no and shows you the exact contradicted line is a control you can file.

Here is the argument I keep coming back to, and it is the one that convinced me this was worth building rather than waiting for GPT-N to fix it. Even a perfect model cannot know your current certifications or your live pricing. It cannot self-certify that it broke no EU rule. And it cannot hand your compliance team a filable trail. Those are properties of a system, not of a model. Provenance, a source-of-truth gate, a deliverability check, an audit receipt. Those do not get obsoleted by a smarter drafter. If anything a smarter, more persuasive drafter makes them more necessary, because the lies get more fluent.

The number that told me personalization was real, not asserted

I almost did not build the style store, and I want to admit why. Everyone in this space claims personalization. "Emails written for the persona, not about it." It is the most asserted and least measured claim in outbound. I was suspicious of my own version of it. A well-crafted prompt already produces a decent email. Was a whole style-retrieval layer actually earning its complexity, or was I decorating a prompt and calling it architecture?

So I built the honest test into the demo instead of hiding from it. A zero-shot baseline toggle. Same prospect, same product facts, but style injection turned off, and a stylometric fidelity score comparing the draft to the real rep's fingerprint. If the style store was theater, the number would barely move. I toggled it and watched.

Zero-shot baseline on the same prospect: the draft slides into "delve," "landscape," and "transformative," and style fidelity drops to 0.295 against Maya Chen's fingerprint, versus 0.5 with the style store on.

The number moved, and the prose gave itself away. With style injection off, the draft slid straight into "delve into how Northwind Pay could transform its engineering landscape" and "unlock synergies and drive transformative outcomes." Those are the audible tells of a probabilistic mean. The style fidelity on that same prospect fell to 0.295 against Maya's fingerprint, where the style-injected version scored 0.5. Across the six-prospect held-out set, mean fidelity ran 0.483 with the style store on versus 0.275 zero-shot. A lift of +0.208 , measured, on that specific set, in bundled-draft mode. Not a promise. A number I can regenerate.

The benchmark panel: style fidelity 0.483 versus 0.275 zero-shot, a +0.208 lift across the six-prospect held-out set, alongside 5 of 5 correct on the labeled adversarial set. Every case is bundled-draft mode, deterministic, and rerunnable.

Personalization is the most asserted and least measured claim in outbound. I wanted a number I could regenerate, not a sentence I could repeat.

The governance side of that panel is the one I trust the most. 5 out of 5 on the labeled adversarial set. The clean draft cleared. The SOC 2 Type II and ISO 27001 over-claim blocked. The "$5,000 build" that sits below the floor blocked as contradicted. The "only 2 slots, expires today" urgency blocked under Article 5. The "everyone has already switched, you will regret it" social proof blocked under Article 5. Five cases, five correct verdicts, and because the gate is deterministic it returns those same five verdicts every single time. An LLM judge cannot promise you that. I need to keep saying the honest part out loud: that 5 of 5 is accuracy on a five-case labeled set, not an open-world guarantee, and the fidelity numbers are the six-prospect set in bundled-draft mode. The discipline of attributing every number to its exact test is not a legal footnote. It is the same discipline the product is selling.

What I actually think this is about

I did not set out to build a governance product. I set out to make an AI write better sales email, and I ran head first into the fact that better writing was never the bottleneck. The market spent two years optimizing AI SDRs for volume and asserting personalization, and the public wreckage of that bet is not subtle. 11x.ai raised $74M and lost 70 to 80 percent of customers within months, claimed roughly $14M ARR against roughly $3M in real contracts, and ZoomInfo said the tool performed significantly worse than their own SDR employees, per TechCrunch in March 2025. Meanwhile Google began rejecting non-compliant bulk email in November 2025, Microsoft enforced in May 2025, and EU AI Act Article 5 has been enforceable since February 2025. The cost of a bad send in 2026 is not a bad email. It is a rejected domain and a regulator.

The lesson I keep turning over is that a better model would not have saved any of that. A better model writes a more persuasive false certification claim. It writes more fluent manufactured urgency. The thing that saves you is not smarter drafting, it is a decision that lives outside the drafter, reads a source of truth, and leaves a receipt. Personalization is not verification, and in 2026 neither one is enough on its own. You also need governance, sitting in the one place it has to sit: between the draft and the send. If you want to watch it block that email yourself, it is at veriprajna.com/demos/ai-sales-personalization.

And if you would rather watch it than read me describe it, here is the whole gateway running end to end, blocking that email and signing the receipt.

So here is the question I have started asking every RevOps leader who tells me their AI SDR is writing great email. Not whether the email is good. Can it prove which source backed the last claim it sent, and could it hand that proof to your compliance team this afternoon? If the answer is no, the writing was never the risk.

I built an AI to beat the solver at airline crew recovery. It lost, and that loss became the product.

Ashutosh Singhal — Sun, 05 Jul 2026 08:00:00 +0000

The benchmark I built to win, and lost

I built the first version of StormCrew to beat the solver. That was the whole pitch in my head. Airline operations control runs on decades-old optimization engines, so if I could train something smarter, I would have a story worth telling. I spent weeks on it. Then I benchmarked my recovery engine against CBC , a mature open-source mixed-integer solver that has been battle-tested since before I could write a for-loop, and CBC won. Not by a rounding error.

I remember staring at the two columns of numbers and feeling that specific hollow you feel when the experiment you designed to prove yourself right proves you wrong instead. The solver was faster. Its plans were cheaper. It never once handed back an infeasible schedule. My clever version lost on all three.

So I did the only honest thing I could think of. I changed the claim, not the numbers.

I built the AI to beat the solver. The solver won. The interesting part turned out to be everything that fight was hiding.

That reversal is the spine of what StormCrew actually became, and I think it is the more useful story than the one I set out to tell. You can run the whole thing yourself at veriprajna.com/demos/airline-crew-scheduling-ai, but let me walk through what changed my mind, because the pivot is the point.

What actually breaks when a storm grounds a hub?

I went back and read the meltdown post-mortems after CBC humbled me, and almost none of the failure was "the math was slightly suboptimal." Irregular operations, what the industry calls IROPS , cost airlines roughly $60B a year (IATA). The canonical disaster, Southwest in December 2022, ran about $1.2B, with around 16,900 cancellations and roughly 2 million passengers stranded. When I traced how those days actually unravel, the optimizer was never the villain.

Three things break instead. Recovery is too slow : when a storm grounds a hub, re-crewing the downstream cascade is still largely a 4-to-12-hour manual scramble (a sourced benchmark, not a number I made up). It is too risky : every re-crew has to respect FAA Part 117 duty and rest limits and a per-carrier union CBA , and a single violation is a compliance event, not a footnote. And it is too opaque : the cascade of downstream flights that just lost their crew is invisible until those flights are already cancelling.

That last one is what the legacy tools miss, and it is the first thing I made the demo show. Inject a storm at the busiest hub and the app highlights the blast radius : the grounded flights plus the one-hop downstream flights that lose their crew through the rotation. In the seeded scenario it is 53 flights at risk across the network.

Inject the storm and the blast radius lights up: 53 flights across the network have just lost their crew, the cascade legacy tools see only after cancellations start.

Since the DOT auto-refund rule (Oct 2024), every 3-hour-plus cascading delay is now an automatic financial hit too. So the cost of being slow, illegal, or blind went up precisely while the tooling stayed the same. None of those three failures is fixed by a better objective function. I had been optimizing the one thing that was already fine.

Why did I stop trying to beat CBC and start feeding it?

I made peace with losing to CBC by giving it a different job. Instead of competing with the solver, I wrapped it. The pipeline is all real, deterministic, seeded code: a synthetic airline network and crew state, a disruption injector that computes the blast radius by graph reachability over the rotation, a duty generator, then CBC as the engine that picks the plan, then a shadow-compare against doing nothing, then a signed certificate.

When I run it on the seeded storm, the generator produces 1,762 legal recovery duties (52 of them deadhead repositions to move crew where they are needed), plus 53 cancel fallbacks, for 1,815 candidate columns in total. CBC solves the resulting 1,815-variable, 115-constraint minimum-cost set-partition to OPTIMAL and returns a plan in about 0.11 seconds. Result on that scenario: 52 of 53 flights re-crewed (98 percent), 1 cancellation, 34 crews used (25 line and 9 reserve).

The engine on screen is CBC, disclosed, not disguised: an 1,815-variable, 115-constraint set-partition solved to OPTIMAL. I use the solver. I never claim to beat it.

The airline's problem was never that the solver was too weak. It was that the recovery was too slow, too risky, and invisible until it was too late.

Note the honesty of that screenshot. The recovery engine is CBC, named on the panel. The durable claim is not that my code out-optimizes a solver. It is that the plan arrives in well under a second where the sourced manual process takes 4 to 12 hours , and the app measures that gap in the open. I want to be precise about scope, because this is a demo and I refuse to launder it into more than it is: those exact figures are one seeded synthetic network's results, not an open-world guarantee. The speed-versus-manual claim is the one that travels.

The legality guarantee belongs in code, not in a model's judgment

I have a strong opinion I only earned by building this, so let me state it plainly. A legality guarantee cannot live in a model's judgment. It has to live in deterministic code, by construction. The way you keep an illegal crew duty from ever being recommended is not to train a model to avoid it, and not to add a penalty term to the objective and hope the optimizer routes around it. It is to make the illegal duty impossible to generate in the first place.

So the constraints are enforced at generation time, not scored after. Part 117 caps a duty period at 780 minutes , flight time at 480 minutes , and requires a minimum sit of 30 minutes ; the sample CBA caps a duty at 4 segments. Only duties that satisfy all of those ever become candidate columns. This is action masking. An illegal assignment is not penalized, it is unrepresentable. Whatever CBC does with the columns it is handed, and whatever the optional copilot later says about the plan, neither can bring an illegal duty back to life, because it was never in the set.

That gives me an invariant instead of a score: 0 illegal assignments, ever, unit-tested (the test suite is 3/3 passing, checking that the blast radius is non-empty, that only legal columns are generated, and that the recovered plan is a legal partition). A score you can regress. An invariant you can promise.

The line I care about most: 0 illegal, enforced in code by column masking, not by the model. Part 117 and the CBA are guaranteed by construction, not by a model behaving well.

This is also why I no longer find "autonomous AI ops" pitches convincing when the safety story is "the model learned not to." I trusted a model to respect a hard rule exactly once during this build, early, and it was fine until the one input where it was not. In a domain where a single violation is a regulatory event, "usually legal" is the same as "not legal." I would rather delete the possibility than supervise it.

What happens on the worst day of the year?

I almost shipped a version that would auto-approve anything, and I am glad a scenario stopped me. Toggle the demo to severe , where the event is bad enough that reserves are exhausted and only about 30 percent of crews remain. CBC still finds a fully legal plan, in about 0.05 seconds , still 0 illegal. But that plan would cancel 20 of 53 flights , which is 38 percent of the radius, well above the OCC's 15 percent auto-approve threshold.

The right move there is not to silently stamp a plan that cancels more than a third of the affected network. So the status flips to ESCALATE, human sign-off required , with the reason shown. The plan is still computed, still legal, still surfaced for the controller ( 33 flights recovered, 62 percent of the radius ). It just is not auto-approved.

The honest hard case: a legal plan that still cancels 38 percent of the radius flips to ESCALATE, human sign-off required. The plan is shown and flagged, never rubber-stamped.

The part most "autonomous" pitches skip is knowing when the correct action is to not act, and hand the day to a human.

Building that gate changed how I feel about the whole category. Escalation is not the system failing. It is the system being honest about a bad day. An advisor that always returns a confident answer is easy to demo and dangerous to trust. The one that occasionally says "this one is above your line, you decide" is the one I would actually put next to a controller at 3 a.m.

The artifact I would want if I were the controller

I kept asking myself what an operations controller would need the morning after, and the answer was not a dashboard, it was a record. So every recovery seals into a signed recovery_plan.json : the disruption, the chosen plan action by action with crew and flight, the specific Part 117 and CBA clause checked per action against its ceiling, the recovery wall-clock, and the savings figures. It is the OCC's audit record of why this recovery was recommended, exportable in one click.

Every recommendation exports a signed recovery_plan.json: status, the Part 117 limits checked, and the legality guarantee stated as enforced by action masking. The audit trail is the point.

There is an optional plan copilot too, and I want to be clear about where it sits. It is a thin adapter to an LLM (default Claude, provider-swappable, or a keyless local bridge) that explains the plan in plain English. It abstains entirely without a key , everything else runs offline and keyless, and it lives outside the decision core. The deterministic mask, CBC, and the escalation gate decide. The model only narrates after the fact. I put it there deliberately, because the moment the language model influences whether a duty is legal, I have lost the guarantee I spent the whole build earning.

About the savings, the same discipline applies. On the normal scenario the shadow-compare shows 52 cancellations avoided and roughly $2.37M of DOT-refund exposure avoided (a $300-per-passenger model) versus doing nothing. That is the most flattering possible framing, because the baseline is stranding the entire blast radius, and it is labeled illustrative of that one scenario. It is not a headline, and it is certainly not proof that my code out-optimizes anything. I lost that argument to CBC on day one. I am not going to quietly win it back in a marketing number.

So what does "augment, don't replace" actually mean?

I used to think augmentation was the timid choice, the thing you say when you cannot build the bold thing. I think the opposite now. The buyer here already owns a good solver stack, Jeppesen or IBS, and cannot tolerate rip-and-replace, lock-in, or an unexplained recommendation on the worst day of their year. Telling that buyer "throw it out for my smarter model" is not bold, it is a claim I already disproved to myself with a benchmark.

What I can honestly offer is the operational layer around the solver they already trust. Make the cascade visible before it bites. Make illegal moves impossible to generate rather than merely discouraged. Collapse hours into seconds. And know when the day is bad enough that the right answer is to escalate, not auto-approve. Everything in the demo is synthetic and seeded, the network, the crews, the disruption, the dollar figures, no real airline data anywhere in it. What is real is the mechanism, and you can watch it run end to end at veriprajna.com/demos/airline-crew-scheduling-ai.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

Here is the question I have not stopped turning over since CBC beat me. When the solver you are competing with is already good, and the buyer already owns it, what is left to build is not a better answer. It is a better relationship with the answer: faster, provably legal, visible, and humble enough to escalate. So how much of the AI you are being sold this year is actually solving the hard part, and how much is re-solving the part that was never broken?

My knowledge-tracing AI wanted to certify a learner who gamed the course. The code I wrote refused.

Ashutosh Singhal — Sat, 04 Jul 2026 08:00:00 +0000

Two employees, two identical green checkmarks, and one of them is a lie

The first time I put the two records side by side, the checkmarks looked identical, and that was the whole problem. A senior BSA analyst with eight years in the role and a branch manager six months post-transfer had both just "completed" the same four-hour anti-money-laundering recertification. Their LMS reported the same thing for each of them: a green check, "SCORM Completed," done. One of them could run a suspicious-activity investigation in her sleep. The other had real gaps in exactly the concepts a regulator cares about. The system that was supposed to certify them for compliance work could not tell them apart , because it was never measuring what they knew. It was measuring whether the video finished playing.

That gap is not academic, and building this demo forced me to keep the stakes in front of me. U.S. companies spend about $102.8 billion a year on corporate training, roughly $874 per learner (Training Magazine, 2025), and a full quarter of L&D leaders say they cannot measure whether any of it worked. In regulated work the gap has a price on the public record. TD Bank absorbed a $3.1 billion AML penalty tied in part to an inadequate training program. Starting August 2, 2026, the EU AI Act's Article 4 begins enforcing role-based AI-literacy, with penalties up to €35 million or 7% of global revenue. In every one of those cases the "Completed" record is worthless as evidence, because completion was never competence.

The system was not measuring whether they learned. It was measuring whether the video finished playing.

I built a demo called Attest to see whether I could close that gap honestly. You can run it at https://veriprajna.com/demos/adaptive-learning-ai. This essay is about the part of building it that genuinely surprised me, and it was not the model.

Two learners, the same course, the same green checkmark. The LMS reads them as identical. Attest reads concept-level mastery underneath and gets a different answer.

What I actually set out to build was a model that knew what each person knew

I started where every "adaptive learning" pitch starts, and then I threw most of it away. Most products that call themselves adaptive are running collaborative filtering underneath: "people like you took Course X next." That is a recommendation engine wearing a diploma gown. It knows what similar learners did. It has no idea what you, specifically, understand. Cornerstone shipped an "Adaptive Learning Agent" in March 2026 that still works this way. I did not want a better recommender. I wanted a model that reads one person's raw interaction stream and infers, concept by concept, what they have actually mastered.

That is a real and old research problem called knowledge tracing, and the honest version of it is a transformer, not a quiz average. I trained a self-attentive knowledge-tracing model, SAKT, about 119,809 parameters, small enough that it trains on a CPU in ninety seconds and caches after. It replays each learner's sequence of interactions and outputs a mastery probability for each of the eighteen concepts in the course. On our seeded synthetic knowledge-tracing benchmark it reached a held-out AUC of 0.8315 , with next-step accuracy of 0.771. The benchmark data is generated by a completely different model family (Performance Factors Analysis, a classical logistic model), so the transformer is not grading its own assumptions. For external context, published SAKT results on the public ASSISTments dataset land around 0.80 (Pandey and Karypis, 2019). Mine was in the same neighborhood.

I was proud of that number. I thought the number was the product. It took exactly one learner to show me it was not.

Then I fed it someone who had gamed the entire course

The learner who broke my confidence was the third one in the cohort, an operations associate who had used an AI assistant to auto-answer the whole module. Watch what my model did with him. Concept after concept, the SAKT inference came back high: 0.927 on SAR red-flag identification, 0.95 on trade-based money laundering, 0.992 on correspondent banking risk, 0.994 on enhanced-due-diligence triggers. Near-perfect everywhere. If I had trusted the model, I would have certified all eighteen concepts and handed this person a clean, signed compliance record.

The trouble is the mastery was fake. The answers were fast and perfect across eighteen unrelated concepts, a response-time distribution that is inconsistent with genuine recall. A real expert is quick on some things and slow on others. This pattern was flat and impossibly fast everywhere, the signature of an answer key, not a memory. My careful 0.83-AUC model looked at an answer key and saw a star pupil.

My model did not merely miss the gaming. It was most confident exactly where it was most wrong.

The live run. SAKT infers high mastery (0.927, 0.95, 0.992), then the competence gate flags "near-perfect accuracy at implausibly fast response times across 18 unrelated concepts," and the credential is withheld: 0 of 18 certified.

That was the moment this essay became true for me. A better model would not have rescued me here. A more accurate transformer trained on cleaner data would have looked at the same fast-perfect answers and been even more certain this person had mastered everything. The failure was not a shortage of accuracy. It was that I had let a probabilistic model make a decision that a model should never be allowed to make on its own.

Why not just train the model to catch it?

My first instinct was the wrong one, and I want to be honest about the day I lost to it. I tried to make the model itself robust to gaming. Add response-time features, train it to distrust implausibly fast sequences, teach the transformer to be suspicious. It is a seductive plan because it keeps everything inside one elegant system, and I like elegant systems. It also does not work, for a reason that took me longer to accept than it should have. Anything you teach a model to detect, you have also taught it to be confidently wrong about in the cases you did not anticipate. A model's whole job is to generalize and smooth. A governance decision needs the opposite: it has to be brittle in exactly the right place, and refuse.

So I stopped, and I moved the decision out of the model entirely. The model advises. Plain deterministic code decides. After the SAKT inference runs, a piece of pure-Python logic I call the competence gate makes the certify-or-not call, and the model cannot override it. The gate certifies a concept only if three things are all true at once: mastery is at or above 0.747, there are at least three real interactions of evidence behind it, and the response pattern is not anomalous. Fail any one of the three and the concept is not certified. It is marked "needs proof," and the learner gets a verification challenge instead of a checkmark.

When the gate looked at the operations associate, it did the thing my model could not. It flagged the evidence as anomalous, refused every concept, and certified zero of eighteen. Seat time withheld, pending verification.

The gate's verdict, concept by concept. Every row shows a high mastery P (0.98, 0.99) and every row still reads NEEDS PROOF, because the evidence was flagged. The model advised certify; the code refused.

I want to be careful about how I frame that, because the brief I wrote for myself is careful about it. This is not a cheater-detector with a hit rate, and I never present it as one. It is a governance outcome on one illustrative learner: concepts withheld because the evidence was not good enough to certify. The point is not "gotcha, you cheated." The point is that Attest certifies only what it can prove, and stays honest about the rest.

Agents advise, code decides, and that line turned out to be load-bearing

I keep returning to that division of labor because, by the time I finished, it was the whole architecture rather than a slogan. There are exactly two places in Attest where something intelligent and probabilistic runs, and both of them only advise. An LLM agent (built on Pydantic AI, defaulting to claude-opus-4-8) does the concept tagging, decomposing one flat "AML Training" course into an eighteen-concept taxonomy: CDD, SAR narrative, structured-transaction detection, OFAC screening, and the rest. That tagging step is the thing the field will tell you is the single biggest reason adaptive-learning projects die, and it is genuinely hard, but it is advisory. The SAKT model infers mastery, also advisory. Every decision that actually matters, what path a learner walks and what gets certified, is made by deterministic code no model can talk its way past.

The sequencer is the other half of that code, and it is where the money hides. Per concept: if mastery is at or above 0.67 the learner skips or just verifies; between 0.38 and 0.67 they stay in the flow zone; below 0.38 they get scaffolding. Those cut-points are not numbers I picked by feel. They are cross-validated against the labeled cohort, and the repo ships a five-fold A/B test showing they beat the hand-set fallback while leaving the decision meaning unchanged. Run that logic over the senior analyst and she skips nearly everything she already knows: sixteen of eighteen concepts certified, seat time falling from 240 minutes to 47.3, an 80.3% reduction. Run the identical code over the branch manager who has real gaps, and it refuses to hand him the same discount: ten of eighteen certified, only 38.9% saved, because he genuinely needs the training.

The same code hands the expert a 47-minute course and the novice a real one. Neither of them can argue with it.

That asymmetry is the entire point. A system that saves everyone the same time is just a shorter course with better branding. A system that saves time in strict proportion to what each person can prove they know is doing knowledge tracing, and it is doing it in code I can read line by line.

What you hand a regulator instead of a checkmark

The artifact I am proudest of is the one a green checkmark can never produce. When the gate finishes, Attest exports a signed Competence Certificate, and I keep two real samples on disk. For the senior analyst it lists all eighteen concepts, each with its mastery probability, the count of interactions behind that estimate, the certify-or-needs-proof status, and the specific regulation it maps to (31 CFR 1020.220, FATF Rec. 12, 31 USC 5318, and the rest). It records the model name, its version, and the benchmark AUC. It carries an honest limitation footer noting that the AUC measures next-item prediction rather than long-term retention, because a mastery probability is evidence, not a guarantee. And the whole document is signed with an HMAC-SHA256 signature so it cannot be quietly edited after the fact.

The certificate for the analyst: 16 of 18 certified, 80.3% seat-time saved, AUC 0.8315 stated on its face, and two concepts (SAR red-flag at 0.73 and TBML at 0.59) honestly held below the 0.747 line and marked NEEDS PROOF.

This is the durable answer to the auditor's real question, which is never "did they finish?" It is "prove which interactions backed which mastery claim." A checkmark cannot answer that. Neither can a better base model, however good it gets, because accuracy is not provenance. A more accurate model still cannot show an auditor the evidence trail behind a decision it made. That trail has to be built as an artifact, on purpose, outside the model. That is what the certificate is, and it is the thing that made me stop thinking of the model as the product.

Accuracy is not provenance. A better model still cannot prove which interactions backed which claim.

You can inspect all of this in the running demo at https://veriprajna.com/demos/adaptive-learning-ai, including the two concepts the certificate refuses to certify for the analyst, where her mastery honestly sits below the 0.747 line.

The part that survives a better model

I built this expecting the model to be the hard part, and being wrong about that changed how I think about the whole category. The model was the tractable part: ninety seconds on a CPU and a respectable AUC. The hard part, the part that actually makes the output trustworthy, was everything I put around the model to keep it from making decisions it had no business making: the deterministic gate, the evidence-sufficiency rule, the anomaly refusal, the signed trail.

That is also the part that does not age out. When I show the cohort numbers (about 51.9% seat-time reduction across a simulated 500-person recertification cohort, roughly $77,872 recovered on this one module, an annualized figure near $389,362 that the app labels a projection on screen), the headline is deliberately not the AUC. The AUC only proves it is a real knowledge-tracing model and not a gradebook. The headline is the percentage of seat time saved and the percentage of concepts auto-certified versus routed to proof, and both of those hold no matter how good the underlying model becomes. A smarter transformer next year does not change the fact that the decision, and the receipt, live in code you can audit.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

So here is the question I have been sitting with, and the one I would put to anyone building in this space. If your system's most important output is a decision a regulator will lean on, do you want that decision made by the model, or made by something the model is only allowed to advise? I had it backwards for a day. Then a fake star pupil, fast and perfect on eighteen things he did not know, walked in and set me straight.

My drone holds 0.9% drift through a GPS jamming bubble. The code I am proud of is the line that refuses to fly.

Ashutosh Singhal — Fri, 03 Jul 2026 08:00:00 +0000

I did not expect the most important part of this project to be the part that gives up.

I had been staring at a replay of a synthetic flight for the better part of an afternoon. A drone crosses an 80-second mission over a 245.3 m path. GPS holds for the first 18 seconds, then dies as the aircraft crosses into a simulated electronic-warfare bubble. On the screen, three tracks fan out from that moment. One of them, a bright pink line, peels off toward the corner of the map and keeps going, confident and completely wrong, while a little readout next to it insists everything is fine.

That pink line is stock visual-inertial odometry, and watching it lie to me is what changed the whole shape of what I was building. I had assumed the hard problem in GPS-denied navigation was staying accurate. It is not. The hard problem is knowing, in the one moment it matters, that you are not.

In GPS-denied airspace, a confident-but-wrong position is more dangerous than an honest "I don't know."

This is the story of building Lodestar, our navigation-integrity engine, and of the week I spent solving the wrong problem before I understood the right one.

I chased accuracy for a week before I understood the real problem

I started where most people start, which is with the estimator. Electronic warfare has made GPS unreliable in exactly the places autonomous drones now matter: contested airspace, mining pits, tunnels, urban canyons. Jamming and spoofing in contested airspace are now widely reported, common enough that GPS-denied navigation has stopped being a research curiosity and become a hard requirement. The obvious fix is to bolt on visual-inertial odometry, fuse the camera with the inertial sensors, and hope the drift stays small.

So I built a real tightly-coupled VIO EKF and spent my first week making it accurate. Same filter, three honest configurations: a dead-reckoning IMU baseline with no vision at all, a stock VIO that fuses every visual feature it sees, and an integrity-aware version that is more careful about which features it believes. I ran them all against a ground truth the simulator generates independently of every estimator , so nothing could grade its own homework. The whole mission is seeded, byte-for-byte reproducible, which mattered more than I realized at the time.

The accuracy numbers were genuinely good. Through the jamming bubble, the integrity-aware estimator holds 0.92% drift , 2.26 m of final error over that 245.3 m path. Dead-reckoning, by comparison, blows out to 30.95% drift and 75.92 m of final error, which is the difference between landing on the pad and landing in the next county. I was pleased with myself for about a day.

Then I ran the convoy.

What a jamming bubble does to a drone that trusts its own eyes

I want to be honest about how ordinary the GPS handoff looks, because that is the part that works. When GPS drops at t=18 s, the filter reweights onto the visual-inertial source with no mode change, no alarm, no operator-visible seam. The estimate just stays continuous. If you were flying the mission you would not feel the floor disappear.

The GPS handoff at t=21 s. Integrity holds GREEN, drift is 0.41 m against ground truth, and 358 usable features feed the estimator. This is the easy part, and it is the part everyone demos.

Here is what I underestimated. A drone running vision does not fail by slowly getting worse. It fails by locking onto the wrong thing and reporting a clean, confident position that is simply garbage. The estimator does not know it has been fooled, because the machinery that computes the position is the same machinery that would have to notice the position is wrong. I had built a system that was accurate right up until the moment it was catastrophically, silently mistaken, and I had no way to tell the two apart from the inside.

That is not a tuning problem. You cannot tune your way out of a system that is confident about being wrong. I needed something that sat outside the estimator entirely.

The convoy that taught a good estimator to lie

I built the convoy scenario specifically to break my own work, and it did. Between t=30 and t=43 a column of vehicles crosses a low-texture clearing, spraying spurious features that look, to a hungry VIO front-end, exactly like the stable landmarks it wants to track. This is the documented ORB-SLAM3 failure, the one where the algorithm tracks the truck and concludes the drone is standing still.

Stock VIO fell for it completely. In the same seeded run, it locks onto the moving vehicles and flies off, ending at 28.79% drift and 70.61 m of final error, with an ATE of 32.95 m that is actually worse than doing no vision at all. That was the pink line I had been watching. The ugly detail is that while it was failing, its internal confidence looked healthy. The estimator was sure of itself. It was sure of a position that was drifting toward the edge of the map.

The convoy at t=32 s. Semantic masking has rejected the moving vehicles (2 static features used, 10 masked), so the integrity-aware track holds at 0.42 m while stock VIO error climbs to 11.0 m and keeps going. Integrity honestly reports AMBER, because there really are few static features to trust.

The integrity-aware estimator survives the convoy for a plain reason: it refuses to believe features that belong to moving objects. In the frame above, semantic masking rejected the 10 convoy features and kept only the 2 static ones, so the estimator never locks onto the column. The trajectory holds. But the part I did not anticipate is what my integrity monitor did at the same moment. It did not paint the screen green and pretend the world was fine. It flipped to AMBER , because with the convoy's features masked out there genuinely were only a handful of static landmarks left to trust. It was accurate and it was nervous, at the same time, and it said so.

That was the first time the monitor told me something the estimator could not, and it is worth more than the accuracy on its own. It also forced the question I had been avoiding.

Why can't the estimator just certify itself?

I spent an embarrassing number of hours trying to answer that with the estimator's own numbers. My first integrity check simply read the filter's reported uncertainty, its position sigma, and gated on that. If the covariance looked tight, fly. It seemed reasonable.

The convoy demolished it. When stock VIO locked onto the truck, its reported sigma stayed small. The filter was confident precisely because it had found something consistent to track. The consistency was a lie, but the covariance could not know that, because covariance is a statement the estimator makes about itself. I was asking the witness to certify its own testimony.

Trust must not depend on the thing being trusted.

That sentence became the design. The Navigation Integrity Monitor lives outside the estimator on purpose , as separate code answering a different question. Not "what is my position," which the EKF answers, but "can this position be trusted right now," which the EKF is structurally unable to answer about itself. The monitor fuses three signals into a GREEN, AMBER, or RED state with hard thresholds: usable feature count (GREEN at 8 or more, RED below 1), position sigma (GREEN under 0.6 m, RED over 1.5 m), and vision NIS, the chi-square innovation consistency that catches exactly the case where the estimator is confident and wrong (GREEN under 7.0, RED over 30.0). No single number can be gamed by the estimator, because two of the three do not come from the estimator's own self-assessment at all.

This is the thing I now believe most strongly about safety-critical autonomy. A better VIO does not fix the confident-but-wrong failure, because "is this trustworthy" is a different question than "what is this," and you cannot answer the first with the machinery that produced the second. The industry keeps racing to make the estimator smarter. The durable architecture is not a smarter estimator. It is a monitor that is willing to overrule it.

The line of code that refuses to fly

The moment I actually understood my own product was the tunnel, and I remember it because it is the moment the system does the least.

Between t=52 and t=66 the mission enters a dark tunnel and the usable visual features collapse to essentially zero. There is nothing to see. No estimator, however well tuned, can manufacture a trustworthy position from a scene it cannot observe. This is not a failure to fix. It is a fact to respect. The stock VIO, still trying, coasts off into a position it has no right to claim.

Lodestar does something else. Features die, position uncertainty crosses the threshold, and after 0.5 seconds of sustained RED the monitor latches the failsafe and commands return-to-home at t=52.5 s. It does not bluff a position. It declares the estimate untrustworthy and hands control back to a safe behavior, and it does this deterministically, with no model in the loop deciding anything.

The tunnel at t=54.5 s. Features have collapsed to 0, integrity is RED ("Position NOT trustworthy"), and return-to-home has already fired at t=52.5 s. Stock VIO, still confident, has flown off to 35.2 m of error. This is the whole point of the product: the honest abstention, not the confident fix.

I want to be careful about the numbers here, because honesty is the entire pitch. Across this one seeded mission, the monitor flags the genuinely unobservable tunnel 100% of the time and false-alarms on the healthy GPS-denied leg about 0.1% of the time , holding GREEN over 56.3% of the denied flight. Those are this mission's results, a physics-faithful proof of the mechanism, not an open-world flight-test guarantee. The durable claim is the structure: trust computed outside the estimator, RED sustained to a latched return-to-home, honest abstention on a scene no one could navigate. The exact metres belong to this run. The architecture belongs to every run.

The most valuable line of code in an autonomy stack is the one that refuses to fly.

What I put in the report so nobody has to take my word

I did not want anyone to have to trust my screenshots, so the last thing I built is the receipt. One click exports a Flight Integrity Report, JSON plus printable HTML, with the scoreboard, the headline metrics, the abstention correctness, the failsafe time, the event timeline, and, the part I care about most, an explicit scope disclosure of what is stubbed versus deferred.

That disclosure is not a disclaimer I bury. It is a feature. The report states plainly that the visual front-end is a stand-in, that the LiDAR fusion is a simulated range constraint rather than a real LiDAR factor, that the MAVLink interface and the Jetson throughput are not exercised here, and that the 30-45 FPS on a Jetson Orin NX 16 GB is a target-hardware spec, not a measured result. The 16,000 EKF propagation updates in the report were measured on the demo machine. The FPS was not, and the report says so in those words.

The Flight Integrity Report, exported with LiDAR fusion engaged. With a simulated LiDAR range constraint anchoring the tunnel, the scene becomes observable, drift drops to 0.01%, 78.9% of the flight holds GREEN, and the monitor correctly does not fire the failsafe. The report is explicit that this is a trade, not magic.

That LiDAR run is worth dwelling on because it is where a lot of demos would cheat. Toggle the simulated LiDAR on and the tunnel recovers: the range constraint anchors the solution, integrity holds, and no failsafe fires, because the scene is no longer unobservable. It would be easy to show only that run and call the problem solved. But the report also carries the cost the solution page owns up to, roughly 250 to 400 g of payload and 8 to 12 W of power. LiDAR is a real engineering trade with a real SWaP-C bill, not a free win, and a buyer deserves to see the bill next to the benefit.

There is exactly one model anywhere near this system, an optional button that drafts the technical-review narrative from the structured report. It runs entirely outside the flight loop. It writes prose. It never touches a control decision. The gate that decides whether to fly is deterministic, seeded, and offline, and it would make the identical call with the narrative button torn out. I was firm about that boundary, because the day a language model gets a vote on whether a drone trusts its own position is a day I do not want to be responsible for.

What I keep coming back to

I set out to build a drone that stays accurate when GPS dies, and I did, on this mission, to under a percent of drift. What I did not expect was to come away convinced that accuracy was the easy half.

Accuracy is a race anyone can enter. Everyone in this field is making their estimator a little tighter, their features a little denser, their backend a little smarter, and all of that is good work. But none of it answers the question that actually kills a drone in a jamming bubble, which is not "how wrong am I" but "am I in a situation where I cannot know how wrong I am." That question has to be asked from outside the estimator, by something willing to say the unglamorous thing and give up the controls.

Accuracy is a race anyone can enter. A system that knows when it is blind is the product.

You can run the whole thing yourself and watch the pink line fly off, then watch the RED state fire, at veriprajna.com/demos/gps-denied-drone-autonomy. It is seeded, so you will see exactly what I saw.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

The question I have not stopped turning over is this one. If the most valuable behavior in your autonomy stack is the refusal to act, how much of your engineering budget goes into the part that gives up, and how much into the part that never learned when to?