DEV Community: Nabbil Khan

Zero Is Not a Score

Nabbil Khan — Sun, 19 Jul 2026 00:20:18 +0000

The evals for my agent skills scored 0% for as long as I had records. Not low. Not noisy. Exactly zero, every skill, every run. And I believed it. For months I thought my skills were bad, because the number said so and the number never wavered. Then one night I actually read the harness. It had fallen back to the wrong auth token. Every call it made came back 401, and it quietly graded each one a failure. The skills never got a chance to fail on their own. I was not measuring them at all. I was reading a broken thermometer.

Real weakness is jagged

Here is what took me too long to see. When a system is genuinely bad, it scores 40% one week and 60% the next. It passes the easy cases and trips over the hard ones. It has good days. Incompetence has texture, because an incompetent system is still in contact with the world, and the world varies.

A flat number has no texture. A flat number means the measurement stopped touching the thing being measured somewhere upstream, and what you are reading is the instrument's resting state. Doctors know this. A heart monitor drawing a perfectly straight line does not mean the patient is calm.

Only a broken thermometer writes the same number every time.

Key insight: A performance number with no variance is a reading of the instrument, not of the thing being measured.

The same bug in three industries

I run systems in advertising, in healthcare billing, and in agent operations, and the same shape shows up in all of them. In advertising I found a dashboard figure that had been hardcoded for two years. Nobody questioned it, because it looked right, and it looked right because it never moved. In agent operations, an account-rotation bug in one of my pipelines overwrote every real error with the same generic message, "no active accounts," so for a while every distinct failure in that system looked identical. And in the denial-assessment engine I run for a medical-billing operation, an agreement metric came back at 44.7%, alarmingly low, until I noticed the two sides of the comparison were scored in different units. One side used an equivalence map; the other compared raw strings. Twenty-seven of the "disagreements" were the same operational decision written two ways. Rescored in units that meant something, the engine sat at 92.0% on classification and 90.7% on action against the golden set.

Three industries, one pattern. When a number is perfectly flat, perfectly round, or perfectly terrible, the odds you are looking at the system go down, and the odds you are looking at the pipe go way up.

0% skill eval score, every run on record
401 the status code behind every one of those zeros
44.7% an agreement score that was really a units mismatch
90.7% action accuracy once the eval scored meaning

What a healthy number looks like

The week after I fixed that eval, the numbers got interesting. Interesting is the point. Mapped-action accuracy on the golden set went from 36.7% to 76.7% across four runs in one afternoon of prompt work. Classification went from 80% to 93.3%. Errors went from one to zero. The line wobbled, jumped, and climbed, and every point on it told me something I could act on.

That is what contact with reality looks like. A healthy metric breathes. So I have inverted my instincts. Steady numbers used to feel comforting and volatile ones used to feel alarming. Now the number that never moves is the one that keeps me up at night. I judge the autonomous operators I run the same way. When one of them turns in the same figure day after day, I do not ask what it is doing wrong. I ask what stopped being measured.

What broke: I believed a broken harness over my own work for as long as I had records. Zero felt like an answer, so I never read the grader. The fix was one auth token. The cost was every decision I made off that number.

Grade the grader

The rule I run now is simple. Before I trust an eval, I make it prove it can pass. Feed it a case where the right answer is known and pinned, and watch it score a success. A harness that has never emitted a passing grade is not strict. It is dead. And dead harnesses grade perfectly, forever.

The result: The denial engine's eval now gates every ship at 90% against a golden set, and I trust that gate because I have watched it pass, fail, and change its mind as the system changed. It moves. That is how I know it is alive.

Distrust any metric with no variance. Jagged is what real looks like.
A flat zero is a flatline. Check the instrument's pulse before the patient's.
Grade the grader. Run a known-good case through every eval and watch it pass.
Never let a fallback swallow an error. One wrong token cost me months of belief.
Perfectly round and perfectly stable deserve the same suspicion as perfectly bad.

The zero on my dashboard was never a grade. It was an instrument talking to itself. The numbers I trust now are the ones that move, because a number that moves is still touching the world. Most of us grade everything except the grader. Start there.

Originally published at nabbilkhan.com.

Write Your Exceptions Down

Nabbil Khan — Sat, 18 Jul 2026 23:50:17 +0000

I had a rule with no exceptions. RETSBAN, my primary agent, runs local inference only: open models on my own hardware, no cloud, no fallback. If the GPU box is down, the agent is down. Then Mia, the agent that runs my marketing, needed a frontier model to do her job well. And I did something that felt strangely formal for a one-person company. I amended my own policy, in writing, with the date attached.

No one was in the room

Here is what took me a while to see. A human employee remembers the day you made an exception. They were in the room when you said fine, just this once. An agent was never in the room. There is no room. Every session starts cold from the files, and the files are the only memory the company has.

So when the policy says local only, no exceptions, and reality contains an exception, one of two things happens. Either the agent obeys the file and blocks work I actually want done, or it notices the contradiction and starts guessing which side to trust. The guessing is the dangerous case. A rule that has been contradicted once, silently, is not a rule anymore. Every agent that loads it gets to decide, in every session, whether to believe it. You never see the deciding. You just see the drift.

An exception that lives in your head does not bend a rule. It erases it.

The amendment

On 2026-04-23 I opened the policy file, a file literally named local_only_no_anthropic, and narrowed it instead of breaking it. The amendment names who is exempt: Mia, and only Mia. It says why: marketing work that needs capability the local stack does not have. It carries the date, and it points to the full model-topology doc for anyone, human or agent, who wants the whole picture. RETSBAN's constraint did not move an inch. Still local only. Still no fallback. Still down when the GPU box is down.

That is the difference between an amendment and a repeal. An unwritten exception repeals the rule and hides the repeal. A written amendment narrows the rule and makes it stronger, because now the rule has visibly survived contact with a real exception. The next agent to read the file does not have to wonder if the local-only line is stale. It can see the line was tested, on a specific date, for a specific reason, and held.

What broke: Before this I kept preferences in my head and re-explained them session after session. Every correction died with the session that heard it. I was giving the same review, over and over, to a workforce with perfect skill and no memory.

The same move in medical billing

This is not an agent-fleet quirk. Last week, in my medical-billing business, I retired a ship gate on the denial-assessment engine. The old gate required the tuned API to agree with a weaker agent rater. A 150-case run showed the gate measured the wrong thing: on genuine divergences the API matched the golden answers about 78 percent of the time, and the agent rater about 38. The gate could never pass, and passing it would have meant nothing.

I did not quietly stop running it. The change record says the gate was retired, says why, and says when the change was ratified: 2026-07-08. It says what replaced it: score the API against golden answers directly, with the floor at 90 percent. Whoever touches that system next quarter, human or agent, inherits the reasoning and not just the residue.

0 cloud calls RETSBAN has made, before and after the amendment
1 agent named in the exception
150 cases in the run that killed the old ship gate
90% the floor the replacement gate has to clear

The result: Nearly three months after the amendment, both rules hold exactly as written. RETSBAN has made zero cloud calls, Mia has exactly the access the file grants, and no session has argued with either fact.

Your judgment is production config

You version your code. You probably version your infrastructure. The strange part of running an AI agent workforce is that your own judgment joins the list. What you want, what you forbid, what you decided to allow anyway: the agents execute whatever the file says. So the file is the policy, and an edit to the file is a deploy.

Key insight: An exception you keep in your head deletes the rule. Amend the file instead, with the date, the scope, and the reason attached.

So this is what I do now, and what I would tell anyone running agents:

Put the rule where the agent reads, not where you think. A rule outside the files does not exist.
Amend, do not repeal. Narrow the rule and keep it.
Attach the date, the scope, and the reason. A future session cannot ask you follow-up questions.
The moment you catch yourself carrying an exception in your head, the rule is already gone. Writing it down is the undo.

A rule you never amended is not a rule. It is a rumor about a rule, and every session of every agent decides whether to believe it. Write your exceptions down. A rule is only as real as its history.

Originally published at nabbilkhan.com.

Who Owns the Row

Nabbil Khan — Sat, 18 Jul 2026 23:20:15 +0000

One of my systems runs a nightly job with two sets of manners. On 154 rows it overwrites whatever it finds and asks nobody. On 15 rows it cannot change a single field; the best it can do is file a request and wait. Same job, same feed, same table. The only difference is whether a human being ever touched the row.

Two doors

The system is a command center for a workforce development operation. The State of Illinois publishes a directory of workforce centers, and every night my sync pulls that feed and compares it against what I have. Most of those sites, 154 right now, are anonymous directory entries. Nobody on my side has ever opened one. For those, the feed writes straight through. New address, new phone, new hours, done. Arguing with the state about a row nobody looks at would be silly. The feed is the only party that knows anything about that row, so the feed decides.

Fifteen sites are different. Those are partnered centers, rows people on my side have filled with staff notes and visit history. Someone drove out there. Someone wrote down which door to use and who to ask for. When the feed disagrees with one of those rows, the sync does not write. It queues the change in a table called WorkNetPendingChange and waits for a person to approve it or reject it.

And nothing gets deleted, ever. A site that vanishes from the state's export just gets a missingSince date. Feeds hiccup. Exports get truncated. A date is reversible. A delete that cascades through visit history is not.

154 directory sites the feed overwrites freely
15 partnered centers where changes wait for review
0 rows the sync is allowed to delete

The row, not the feed

The usual way to design a sync is to ask whether the feed is the source of truth. That question hides an assumption: that authority belongs to the feed. It does not. It belongs to the row.

Think about what each side actually knows. The state knows what the state published. It knows nothing about the visit last Tuesday, or the note saying the listed phone number rings a fax machine. On a row nobody here has touched, the feed's knowledge is all the knowledge there is, so it should win every conflict. On a row people have worked on, the feed holds the smaller share, and letting it overwrite means destroying the larger one.

Authority is not a property of the feed. It is a property of the row.

Key insight: Give a sync write authority row by row. It overwrites freely where no person has done any work, and it only gets to propose changes where someone has.

What broke: Every sync I have ever regretted failed the same way. An upstream export had a bad night, and the job faithfully copied the bad night over months of human work. The overwrite took milliseconds. Getting back what a coordinator knew about a site took weeks, when it happened at all.

The same rule keeps showing up

Once I had the rule, I started seeing it all over my stack. The same command center runs an inbox agent that triages a shared mailbox. Routine messages it handles by itself. Anything that touches a real relationship goes to a review queue. Two doors again.

My publishing pipeline works the same way. Work-log entries publish to the enterprise timeline on their own, because nobody's judgment rides on a changelog line. Public essays, the ones that carry my name, wait for approval. And in a different industry entirely, the denial-assessment engine I run in medical billing acts directly on the mechanical cases and calls escalate_to_human on the few where judgment actually lives.

Three corners of my work, one rule. Automation gets full authority over work no person has touched. The moment a person touches it, the automation drops from writer to proposer.

The result: The sync keeps 154 rows fresh with zero human effort, and every drift against the 15 partnered centers gets human eyes before it lands. Fresh and safe at the same time, without trading one for the other.

What to build

None of this is exotic. It is a WHERE clause and a pending-changes table. The hard part is deciding to build it, because "the feed is the source of truth" sounds so clean. If you are wiring an external feed into a system people actually work in, here is what I would do:

Give write authority per row, not per feed. Source of truth is a row-level question.
Let human work demote the feed. The first staff note on a row turns the feed from a writer into an advisor.
Never let a sync delete. Record the absence and let a person decide what it means.
Make every change pick a door: direct write or review queue. No third path.
Keep the queue short enough that people actually read it. A review queue nobody reads is just a delete with extra steps.

Machines should own what nobody cares about, so people can own what they do. Go look at which rows in your database have fingerprints on them. Those are the ones the feed should have to ask about.

Originally published at nabbilkhan.com.

Who Owns the Clock

Nabbil Khan — Sat, 18 Jul 2026 22:50:14 +0000

A patient asked why she got two reminder texts a day when everyone else got one. Fair question. She had two open episodes in my Medicare RTM engine, and I had written the reminder logic to run once per open episode. Two episodes, two clocks, twice the nagging. The scheduler fired exactly on time. It always does. A few months later, in a different industry, I caught myself starting to write the same bug again. That was when I finally saw what the bug was.

The scheduler is innocent

When a cadence goes wrong, the scheduler is the first place everyone looks. I looked there too. I read the cron expression. I checked the timezone. I hunted for drift. Everything was fine, because cron is almost always fine. The scheduler is the most audited hundred lines in any system I run. The bug was somewhere I had never thought to look, because I had never noticed I was deciding anything there. I had let the episode own the clock.

The fix was one sentence long: remind once per patient, not once per open episode. A patient does not experience episodes. She experiences her phone buzzing at dinner. The moment I said the fix out loud I could hear how obvious it was, and that is exactly why it survived review. It did not look like a decision. It looked like plumbing.

What broke: A patient with two open episodes got double the reminders because I let each episode own its own clock. The scheduler was innocent. The ownership was wrong.

The same bug in a different industry

Then came workforce development. I built an outreach system where coaches call candidates about medical assistant training. Different industry, different codebase, not one shared line between them. The system has a cadence gate: how long a coach has to wait before touching the same candidate again. My first draft reset that clock on every logged attempt.

Think about what that does. A coach leaves five voicemails in a week. Each one resets the clock, so the system decides the relationship is warm. It stops prompting follow-ups for someone nobody has actually talked to. Five voicemails, and the candidate sinks quietly in the queue, marked fresh by calls she never answered.

This time I caught it before it shipped, because the RTM bug had taught me the question to ask. The fix was again one sentence: the gate resets only on one outcome, reached. Attempts write notes. They do not write time.

A voicemail is not a relationship. It is you, talking to yourself, on the record.

The result: Five logged voicemails now write five notes and move the clock zero seconds. One verb resets the gate: reached. The system stopped mistaking effort for contact.

Nouns and verbs

Healthcare and workforce development. Medicare billing rules on one side, career coaching on the other. Zero shared code, identical bug. Both times I went hunting in the scheduler, and both times the scheduler turned out to be a metronome doing its job. The defect lived in two words I had chosen without noticing I was choosing: which noun owns the clock, and which verb is allowed to reset it.

In the RTM engine the noun was wrong. The episode owned the clock when the patient should have. In the outreach system the verb was wrong. "Tried" would have reset the clock when only "reached" should. That is the whole taxonomy. Every cadence bug I have shipped, in any industry, has been one of those two words.

Why did neither bug look like a bug? Because neither decision looked like a decision. The noun hides in a foreign key. The verb hides in a WHERE clause. Review catches bad logic, and this was not bad logic. It was bad grammar, and the grammar compiled.

Key insight: Every cadence bug lives in two words, not in the scheduler: which noun owns the clock, and which verb is allowed to reset it.

2 open episodes, one patient, twice the reminders
5 voicemails that moved the clock zero seconds
1 verb allowed to reset the outreach gate
0 lines of code shared between the two systems

What I do now

When I build anything with a clock in it, I write the two words down before I write the code. The noun goes at the top of the file. The verb goes next to it. Everything else is implementation.

Name the noun that owns the clock, and make it the noun a human experiences. Patients feel phones buzz. They do not feel episodes.
List every verb allowed to reset the clock, then cut the list. Most clocks deserve exactly one verb.
Let every other verb write a note instead of time. History is cheap. Resets are expensive.
Test the plurals: two episodes, five attempts. The singular case always passes, which is why it proves nothing.

The scheduler was never the interesting part. The interesting part is the sentence you wrote without noticing you were writing it. So when a cadence goes wrong, do not read the cron expression first. Read your nouns and your verbs. And if you run systems in more than one industry and keep meeting the same small bug in different clothes, write it down and tell someone. The builders need to find each other.

Originally published at nabbilkhan.com.

Watching or Working

Nabbil Khan — Sat, 18 Jul 2026 22:20:12 +0000

Until this week, the console that files all my Medicare claims had no way to sign out. Plain forms, no animation, not one hour of visual design. Meanwhile the marketing site for the same product got choreographed scroll animations and hand-built React recreations of those very screens. The copies are prettier than the originals. That sounds like a scandal. I think the money went exactly where it should have, and most software points it the other way.

Where the polish went

clearpathcare.ai is the prettiest thing I have ever shipped. Every scroll position is choreographed with GSAP and Lenis. Every console screen was rebuilt by hand in React so it could slide in at just the right moment. Days of art direction for a page most people will see once.

The console behind it, the tool that enrolls patients, logs the calls, and files the claims, got none of that. Zero hours of visual design. Not because it matters less. It matters more; it is the part that makes the money.

The difference is what the person in front of each screen is doing. A visitor to the site is leaning back. They are judging, and they decide in about thirty seconds whether you are real. Looking good is the whole job. A biller in the console is leaning in. They are there all day, and every animation between them and the claim is friction. The best thing that screen can do is get out of the way.

The koi fish

I know what happens when you get this wrong, because I got it wrong once. MOTOR is a quoting engine I built for construction contractors. At some point I gave the quoting flow a koi companion: a little animated fish that swam along while you built a quote. I thought it was charming.

Contractors killed it inside three quotes. The feedback was not a design critique. It was "pretty annoying" and "just give me the quote."

What broke: MOTOR's quoting flow got an animated koi fish to keep users company. Contractors killed it inside three quotes. The verdict was "pretty annoying" and "just give me the quote." Decoration inside a work tool is a tax you charge on every single use.

Just give me the quote. That sentence taught me more about design than any book. A worker does not experience your polish as a gift. They experience it as one more thing standing between them and being done.

The rule

So here is the rule I use now. Do not ask how important the system is. Ask what the person in front of it is doing. Watching? Spend on looks. Working? Spend on speed.

By that rule, the console being the most important thing I own and the plainest is not a contradiction. It is the rule working.

Key insight: Spend design where people watch and speed where people work. The budget follows what the user is doing, not how much the system matters.

Polish is for people who are watching. Speed is for people who are working.

This does not mean work tools get no design. This week the console finally got real design attention, and every bit of it was invisible: the browser back button works now, breadcrumbs show where you are, a confusing device field got a sane default and a plain hint. Nobody will screenshot any of it. Everyone who works in it will feel it.

And when the console did need to be watched, because prospects wanted to see it run, I did not decorate the console. I bolted a guided demo onto the side: nine chapters, about forty steps, strictly read only. The show got its own stage. The tool stayed plain.

The result: The console's guided demo shipped as nine chapters and roughly forty read-only steps. The show got its own surface, and the screens people actually work in stayed plain and fast.

3 quotes before contractors killed the koi fish
0 hours of visual design in the console that files the claims
9 chapters in the guided demo, a surface built for watching
40 steps of polish kept out of the working screens

The test

I run systems in healthcare, in construction software, and in marketing, and the same one-line question now gates every design hour in all three: is the person on the other side of this screen watching or working?

Ask watching or working before you spend a single design hour.
Spend looks on screens people judge and leave. Spend speed on screens people live in.
Work tools still deserve design, the invisible kind: a back button that works, a sane default, a breadcrumb that tells you where you are.
If a work tool needs to be watched, build the show a separate surface, like a demo, and keep it out of the flow.
When a working user calls something pretty annoying, believe the annoying, not the pretty.

Step back far enough and the pattern is simple: the closer a screen sits to the money, the plainer it gets. If the part of your product that does the real work looks embarrassingly plain, you are probably spending right. And if you run an ugly system that quietly earns, I would like to compare notes, because we are making the same bet. The money is made in plain rooms. The pretty rooms are how people find the door.

Originally published at nabbilkhan.com.

Underpowered Tests Lie

Nabbil Khan — Sat, 18 Jul 2026 21:50:11 +0000

Last month I audited the statistics inside Mia, my ad-testing agent, and found a 20x error that had never broken anything. No exception, no failed assertion, nothing in the logs. The textbook shortcut said each arm of an A/B test needed 1,111 users. The exact formula, run at the 1.9 percent click-through rate ads actually get, said 22,278. Every test Mia had sized was running on a twentieth of the data it needed, and every one of them still announced a winner. I run AI systems in several industries, and the failures that cost the most all have this shape: nothing breaks, and the answer is wrong.

Where the shortcut came from

The shortcut is Lehr's equation: sixteen over the effect size squared. It is in the textbooks because it is easy, and it is easy because it was derived where the math is friendliest, near coin-flip rates, where variance peaks and everything is symmetric. Ads do not live there. Ads live at 1.9 percent, where the effects worth catching are tiny in absolute terms and the samples you need to see them are huge. The shortcut knows none of this. It returns 1,111 and moves on.

The part that took me longest to accept is that the formula is not wrong. It works fine in the world it came from. The bug was mine. I carried it into a different world without re-deriving it.

1.9% the base click-through rate Mia lives at
1,111 per-arm sample the textbook shortcut prescribed
22,278 per-arm sample the exact formula actually requires
14% real false-positive rate of a three-arm test at raw p < 0.05

What an underpowered test says

You would think an underpowered test would fail. It cannot. There is no runtime check for statistical power. The test collects its too-small sample, computes a p-value, and hands down a verdict in the same confident voice it would use with twenty times the data. Sometimes the verdict is even right. That is what makes it dangerous. A crash gets fixed the same afternoon. A confident wrong answer gets acted on.

An underpowered test never says it does not know. It says B wins, and it says it with a straight face.

The audit found a second lie. Mia judged three-arm tests at raw p below 0.05, which sounds like a 5 percent false-positive rate. With three comparisons it is closer to 14 percent. One test in seven was crowning a winner that did not exist. Holm-Bonferroni fixes that in a few lines of code. Writing the lines took minutes. Suspecting I needed them took an audit nobody asked for.

What broke: The 20x sizing error and the 14 percent false-positive rate sat in production together, and the code ran green the whole time. Every signal I could see said the system worked. The only thing wrong was the answers.

The same lie in healthcare

I caught this in ads, but I only recognized it because I had already paid for it in healthcare. Daniel, my denial-assessment agent, is graded against a golden set of 30 cases. When a prompt change moved classification accuracy from 80 to 93.3 percent, I did not believe it until the eval said the same thing four runs in a row. Thirty cases is a small sample. A single run there can lie as smoothly as an underpowered ad test, and it lies the same way: not with an error, with a number.

That is the cross-industry pattern. It has nothing to do with ads or insurance claims. It is about verdicts that sound the same whether or not they earned it. Ad tests, eval harnesses, dashboards: none of them has a voice for "not enough data." If you want that voice, you have to build it yourself.

The result: Mia now sizes every test with the exact unpooled formula at the observed base rate, refuses to declare a winner before that sample is met, and applies Holm-Bonferroni across arms. The fix took a day. Finding it took an audit nobody asked for.

Re-derive at your base rate

Key insight: Every statistical rule of thumb is calibrated for someone else's base rate until you have re-derived it at yours.

That is the rule I run on now, in every industry I touch. Here is what it looks like in practice:

Re-derive the formula at your own base rate before you trust it. The textbook derived it at theirs.
Ask what the system says when it does not know. If the answer sounds the same as when it does, you have a liar, not a tool.
Count your comparisons. Three arms at raw p below 0.05 is a 14 percent false-positive rate, not 5.
Audit the judge, not just the work. The most expensive bugs live in the thing doing the grading.
Treat green as a timestamp, not a verdict. It means the code ran, not that the answer is right.

The tests that fail loudly are the cheap ones. You see them, you fix them, you move on. The expensive ones succeed quietly at the wrong thing, for months, while everyone trusts them a little more each week. If your experiments run at base rates the textbook writers never pictured, this is the week to re-derive your formulas. And if you catch one of these lies in your own stack, write it up. The operators who actually check are rare, and we should find each other. The math was never the hard part. The hard part is doubting a number that looks right.

Originally published at nabbilkhan.com.

Trust the Calculator

Nabbil Khan — Sat, 18 Jul 2026 21:20:09 +0000

The pricing formulas in Motor, the estimating engine I built for a water feature shop, did not come from the manual. I pulled 32 of them out of the JavaScript behind Aquascape's contractor calculator, the tool contractors actually use to bid jobs. The manual was sitting right there, official and free. Ignoring it was the best design decision in the whole system.

A vendor never ships a sloppy calculator

Why trust the calculator over the manual? Because of what happens when each one is wrong. If the manual sizes a pump wrong, a reader shrugs and moves on. If the calculator sizes a pump wrong, a contractor bids a job at that number, wins it, and loses money on the install. Then the phone rings. So calculators get fixed and manuals drift. Give it ten years and the two quietly disagree, and everyone in the trade knows which one to trust without anyone saying so.

A vendor will ship a sloppy PDF. They will never ship a sloppy calculator.

Documentation is what a domain says about itself. The artifacts money flows through are what it actually believes. Once you see that split, you cannot stop seeing it.

The other half was in old invoices

Formulas only get you to cost. What a shop charges on top of cost is a belief about its market, and no vendor document holds that number. So I pulled 132 historical quotes out of the shop's CRM. Real quotes, sent to real customers, most of them paid. I calibrated Motor's markup against those, then checked its output against what the shop had actually charged.

The result: Calibrated against 132 real quotes, Motor's estimates landed within 5 percent of what the shop actually charged, with no pricing rule taken from documentation.

I could have just asked the owner what his markup was. But what an owner says and what his invoices show are rarely the same number, and the invoices are the ones customers paid. When the two disagree, believe the invoices.

The same bug in a different industry

I build and run systems in several industries, and the surprising part is how often the same small idea matters in all of them. This year it showed up in medical billing. I run a denial engine that reads insurance denials and recommends the next move: appeal, fix the coding and resubmit, bill the patient, or write it off. The first version reasoned from guidance prose, appeal-strategy text that reads like documentation. On a golden set of 30 adjudicated cases, it picked the right action 36.7 percent of the time.

What broke: The engine kept recommending appeals for coding errors that should just be fixed and resubmitted, because the guidance said appeal and the model believed it. It scored 36.7 percent on action selection. I had banned documentation from my inputs, then smuggled it back in as a prompt.

The fix was the same move as the calculator. We keep a reference layer that records what experienced billers actually do with each denial code, distilled from real worked claims. I anchored the engine to that instead of the prose. Action accuracy went from 36.7 percent to 76.7 percent in one evening, and classification went from 80 percent to 93.3 percent. The model did not get smarter. The spec got real.

32 formulas pulled from the contractor calculator's JavaScript
132 real quotes used to calibrate markup
5% gap between Motor's estimates and actual shop pricing
76.7% denial-action accuracy after anchoring to worked claims, up from 36.7%

Two records

Every domain keeps two records of itself. One is written for readers: manuals, best-practice guides, onboarding docs. The other is written for money: calculators, invoices, worked claims. The first record is what the domain wants to be true. The second is what it paid to learn. When you encode a domain into software, you are choosing which record to believe, whether you know it or not.

Key insight: Encode the rules a domain risked money on (the shipped calculator, the paid invoice, the worked claim) and treat everything it merely wrote down as a rumor.

Pull your rules from the artifacts people bid with. The calculator beats the manual it shipped next to.
Calibrate against money that actually moved, and measure the gap in percent, not in vibes.
Treat documentation as a hypothesis about the domain, never as the spec.
When your model gets a domain wrong, check what you fed it before you blame the model.

None of this is really about ponds or insurance claims. It is about where truth lives in a domain. A domain will tell you anything in its documentation. What it bids with is what it believes. Build from that. And if you are pulling formulas out of someone's calculator at midnight, in an industry nothing like mine, we should compare notes.

Originally published at nabbilkhan.com.

The Signup Form

Nabbil Khan — Sat, 18 Jul 2026 20:50:08 +0000

On April 3 my Kindle pipeline went green. The EPUBs passed epubcheck, the print PDFs matched KDP's trim spec, and the covers came out with spine widths computed from the page counts. Zero to green in one day. It is the middle of July now, and that pipeline has not published a single book, because the only thing between it and a live listing is a signup form that wants a Social Security number and a bank account. The code took a day. The form has taken a quarter.

The same wall

I told myself this was a publishing problem. Then it happened again in professional services. Speckles is an RFP agent I built. You hand it a request for proposal and it answers from the source decks, with citations back to the exact slides it drew from. Empty repo to verified, cited answers in one day. It is still not live. Going live needs a BotFather token from Telegram and three decks that sit behind a Canva login. Not a line of code. A token and a login.

Then I remembered I had seen this wall before and misread it. In medical billing, the software to work a denial comes together in days. Getting credentialed on a payer portal takes months. For years I filed that under healthcare being healthcare. It was the same wall. I just could not see it, because back then the code was slow too, and the paperwork hid behind it.

Publishing books, answering RFPs, appealing denials. Three businesses with nothing in common, and the same wall in all of them. The wall is me.

The machine was ready in a day. I was the part that took a quarter.

1 day zero to a green Kindle pipeline
1 day zero to a verified, cited RFP agent
3 months the green pipeline waiting on one signup form
3 decks stuck behind a single Canva login

What only a person can produce

Look at what is actually blocking. A tax ID. A bank account. An OAuth login. A messaging token tied to an account tied to a phone number tied to a person. None of it is hard. All of it needs a legal human identity, and a venture only gets the identities its founder brings.

Why can't the machine do this part? Because these systems exist to prove a person is behind the button. The form that wants my SSN is not badly designed. It is doing exactly what it was built to do. It marks the line where the economy stops trusting machines and asks for a human it can tax, sue, or pay. AI moved every other line. It cannot move this one, and it should not.

So the critical path of a new venture has quietly changed shape. It used to be months of engineering with paperwork sprinkled through, and the paperwork got done while the code was still being written. Now the engineering compresses into days, and the paperwork stands alone, exposed, as the thing everything else waits on.

Key insight: Once AI does the building, the critical path of a new venture is no longer engineering. It is the short list of artifacts only a legal human identity can produce.

Build to the boundary

Here is the discipline I landed on. Build everything up to the auth boundary, then stop. Every project now ends the same way: a green machine, parked and verified, plus a short list titled PENDING (user). The list holds only things a machine cannot do. Sign up for the account. Enter the SSN. Approve the OAuth grant. Export the decks.

The point of the list is not organization. The point is honesty. When the human blockers stay buried inside a project, they look like engineering, and I respond by engineering more.

What broke: The Kindle toolchain sat green for three months while I kept finding code to polish. It was done on April 3. I was improving the part of the project that was no longer the bottleneck, because polishing code feels like progress and typing my SSN into a form does not.

The result: Speckles went from an empty repo to an RFP agent with verified, cited answers in one day. Everything a machine could produce, it produced. Its pending list has two entries, and both of them are mine.

What I do now

Build to the auth boundary, then stop. Once the machine is green, more engineering is avoidance.
End every project with a green machine and a written PENDING (user) list. Blockers that stay unwritten disguise themselves as work.
Schedule the paperwork like the critical path it is. Right now a form is worth more than a feature.
Find the wall before you start. If a venture is going to stall at a signup form, better to know on day one than on day ninety.

For most of my working life, the scarce thing was working software. I have a Master's in Language and Theology and I ran a 32-rig mining farm before I wrote production code, so I never assumed software would be the easy part. Now it is. Green machines stack up at the boundary like planes waiting on one runway, and the runway is one person with a Social Security number. The constraint is not intelligence anymore, and it is not code. It is the number of hands that can sign. If your machines are parked at the same wall, compare notes with me. The builders need to find each other.

Originally published at nabbilkhan.com.

The Model Is a Witness

Nabbil Khan — Sat, 18 Jul 2026 20:20:06 +0000

Two words in a subject line decide whether a training business made money. A DocuSign email that starts with "Completed:" means someone signed a voucher and the money is real. One that starts with "Complete with Docusign:" means someone was only asked to sign. I fed both to a language model and asked whether the voucher had come in. It said yes to both. To a model, a request to sign and a signed contract are the same sentence with slightly different punctuation. That near miss gave me the rule every money-touching pipeline I run is now built on: the model can recognize the event, but it does not get to define it.

Two words run a business

I did not find that difference in a spec or a data dictionary. I found it by reading the inbox where the business actually runs, thread by thread, the way you would read a ledger. The list of categories behind the inbox agent is hand-written from real messages: which senders carry money, which subjects are contracts, which attachments are just noise. And the one marker that matters, the one that tells booked revenue apart from a polite request, is a plain string check on the subject prefix. No temperature. No embedding. Just startsWith.

What broke: I almost booked revenue that did not exist. The model read "Complete with Docusign:" and reported the voucher as received. A plain prefix check now sits between that answer and the ledger, and it has never once gotten creative.

The model still has a seat. Just not that one. When a coach writes something human, like "signed and attached, sorry for the delay," no prefix rule will catch it. That is real ambiguity, and that is exactly the work the classifier should do.

The deterministic layer makes the law. The model gives testimony.

Enrich first, then reason

Daniel is my denial-management agent for the medical-billing operation. Completely different industry: healthcare revenue cycle instead of training, CARC codes instead of DocuSign subjects. Same shape. Before the model reads a single denial, a deterministic layer does the defining. The CARC code is looked up against a reference database. Payer rules attach. Bundling edits apply. The timely-filing math runs to the day. Only then does the model reason, inside a frame it cannot redraw.

The evals showed how much the seating matters. When Daniel's recommendations floated free, mapped-action accuracy on the 30-case golden set was 36.7%. Tying each recommendation to the reference database's per-code action (write off, bill the patient, correct and resubmit) moved it to 76.7% in one change, and classification went from 80% to 93.3%. The failures that were left were about definitions, not perception. The model wanted to appeal a bundling denial that should be corrected and resubmitted, and it kept writing off deductible balances that the reference layer knows must go to the patient. Every fix was the same fix. Take a definition away from the model and give it to the data.

36.7% action accuracy when the model defined its own actions
76.7% action accuracy after one deterministic anchor
92.0% classification accuracy at the ship gate
90.7% action accuracy across 150 real denial cases

The result: With the deterministic layer making the calls, Daniel cleared the ship gate at 92.0% classification and 90.7% action accuracy against golden across 150 real denial cases. Same model as before. Different seat.

A wrong yes books money that is not there.

Three industries, one seating chart

The rule was learned in training and proven in healthcare, and by the time I built MOTOR, my quoting engine for a water-feature contracting business, it was policy from day one. MOTOR prices jobs with the manufacturer's official formulas, behind three deterministic layers that keep it from ever quoting too low. The model never writes a number on a quote. It reads the messy site notes, the "sloped yard, maybe 16 by 11, client wants a stream" kind of thing that no formula can parse, and turns them into clean inputs. Then arithmetic does the pricing, because arithmetic does not round in the customer's favor just to be agreeable.

None of this is being down on models. Deterministic code cannot read "signed and attached," and a model cannot be trusted with "Completed:". Each one is unbeatable in its own seat and dangerous in the other's. Three industries, no shared regulators, no shared vocabulary, one shared risk: a wrong yes makes money that is not there, or a price that does not cover the job. So the categories get hand-written from real cases, the markers that matter get decided by code, and the model gets the genuinely ambiguous middle. A real job. A hard one. The only one it should have.

Key insight: In any pipeline where a mistake costs money, hand-write the categories from real cases, decide the markers that matter with plain code, and give the model only the genuinely ambiguous middle.

What three ledgers taught me

Read the real inbox before you write a prompt. The categories live in the mail, not in the model.
If a string decides money, check the string. A prefix comparison costs zero tokens and never hallucinates.
Enrich first, then reason. Codes first. Rules first. Deadline math first. Judgment last.
Give the model only the fuzzy middle. "Signed and attached" is classifier work. "Completed:" is not.
When an eval fails, ask which seat the failure was in. Most of Daniel's wrong answers were definitions the model should never have owned.

Training, healthcare billing, and outdoor construction share no regulators, no customers, and no words. They share one operator and one seating chart. And the chart holds because it was learned as a scar in one industry and carried as law into the next two. If you are building anything where a wrong yes makes money out of thin air, decide your prefixes in code, and let the model testify.

Originally published at nabbilkhan.com.

The Number That Never Existed

Nabbil Khan — Sat, 18 Jul 2026 19:50:05 +0000

A single made-up number, 1.9 percent, almost killed an experiment that never ran. Then I found the same lie hiding in my Medicare billing engine and my denial classifier. The fix was the same in all three, and it was not a better guess.

The number that never existed

1.9 percent. That was the baseline conversion rate my ad-experiment engine reported for a test that had never served a single impression. The metrics table was empty. The code reached for a hardcoded default. And 1.9 percent walked out the door wearing a suit.

Here is why that is not just cosmetic. The baseline feeds the formula that decides how long a test has to run before you are allowed to trust it. Feed that formula a made-up baseline and the sample size it asks for can be off by 20 times in either direction. So you either burn weeks on a test that quietly finished long ago, or you crown a winner on noise. The experiment was dead before the first impression, and the dashboard looked healthy the whole time.

An error gets a ticket the same afternoon. A plausible number gets a quarterly review, maybe, eventually. A made-up number does not look made up. That is the whole problem.

Key insight: When a system has no ground truth, the honest move is to refuse in a way a machine can see: an error, a routed state, or a new category. Never a reasonable-looking default.

One fix, three industries

The fix was not a better default. There is no better default; an empty table has no truth to approximate. The endpoint now returns an HTTP 400: pass a real baseline, or do not start the experiment. The refusal moved out of my good intentions and into the contract.

Then I went looking, and found the same disease in two businesses that share nothing with ad tech except me.

My Medicare remote-monitoring reimbursement engine used to treat payer eligibility as a gate on the claim. But under the way we drew the product, the practice's own biller owns checking eligibility. My engine does not hold that answer. Blocking the claim was a guess. Waving it through was a guess. So eligibility stopped being a gate and became a route. Those claims now land in a BILLER_DETERMINED state, a named place in the schema where the human who actually has the answer picks up the work.

The denial engine had the ugliest version. Deductible, coinsurance, and copay denials (PR-1, PR-2, PR-3) are not really denials. The balance just moves to the patient. My taxonomy had no category for that, so a blanket rule mapped "not a real denial" onto "no action, write it off." The engine was quietly telling billers to write off money that patients actually owed. The fix was not a smarter prompt. It was a new category: bill_patient. On the 30-case golden set, classification accuracy went from 83.3 percent to 90.0 percent the moment the schema stopped forcing reality into the wrong box. In the same push, anchoring the recommendations to a fixed per-code action layer took mapped-action accuracy from 36.7 percent to 76.7 percent.

83.3% classification, before
90.0% classification, after
36.7% mapped action, before
76.7% mapped action, after

The model was never the bottleneck. The words it was allowed to say were.

The scars

Let me be clear about whose fault the 1.9 percent was. Mine. Someone typed that constant years ago to keep a day-one dashboard from crashing, and that someone was me. It survived two years of code review because it was plausible. Nobody audits a number that looks right.

And the trap does not only live in return values. My training harness ran a frontier model as a reference, and it scored 0.241 on the hard 54-task set, the same as five small local models. The plausible read: the models are weak. The honest read, after digging: 34 of the 54 tasks were failed by every model, the frontier one included. The exam was broken, not the students. A plausible default can hide in a conclusion just as easily as in a line of code.

What broke: A hardcoded 1.9 percent, typed to stop a day-one dashboard from crashing, survived two years of code review because it looked right. The bug was not in the model or the math. It was in a default nobody thought to question.

What I believe now

When the system does not know, the refusal belongs in the schema, not in a guess. An error, a routed state, a new category. Never a plausible number.
An error is information. A default is a lie with good posture.
Route, do not gate. If someone else holds the answer, build them a named place to stand and hand the work over out loud.
If reality has a case your list of categories lacks, the list is wrong. Reality will not add the missing one for you.
Audit your plausible numbers harder than your errors. Errors wake you at 3am. Plausible numbers bill you quietly for a year.

None of this needed a bigger model or a cleverer algorithm. It needed me to admit, in the type system, in the state machine, in the HTTP status code, that the system did not know. That admission is the whole difference between software that reports reality and software that decorates it.

Every empty table. Every claim you cannot check. Every case your categories cannot name. Each one is a chance to refuse honestly or to lie smoothly, and the lie compiles just fine.

Originally published at nabbilkhan.com.

Rollback Is Deleting a Link

Nabbil Khan — Sat, 18 Jul 2026 19:20:03 +0000

I counted the sections on one page of a workforce dashboard I run. Twenty, stacked top to bottom. The same candidate was drawn more than ten different ways on that one page. Two different modals could edit the same record, and they disagreed about what the record was. The daily loop, the thing staff actually did every morning, was split across three surfaces that were not even next to each other. Everyone knew the page was bad. And I could not take it down to fix it, because people used it to do their jobs.

How a page gets like that

Nobody designs a page with twenty sections. You design a page with three, and then you add one. Each addition is reasonable. Each one is a ten-minute change that solves a real request from a real person. The twentieth section went in for the same good reason as the fourth. That is the trap. A page like this is not the product of bad decisions. It is the product of seventeen good ones.

What broke: That page grew on my watch. I approved most of those sections myself, one at a time, and every one of them looked harmless going in. The mess was not inherited. I built it.

So the page needed a rewrite. But rewriting a live surface is one of the most dangerous things you can do in software. The graveyard is full of v2s that shipped half finished, quietly broke something the old version did, and got rolled back in a panic. Every operator has seen one. So the interesting question is not how to build the new page. It is how to earn the right to ship it.

One rule

I gave myself one rule: the rewrite does not touch the API. Not one endpoint changed, not one field renamed, not one migration. The new module, Voucher HQ, had to be built entirely out of contracts the old page already used. If the new page wanted data the old API could not give it, that was the new page's problem, and it had to wait.

The entire backend diff for the whole rewrite was one additive flag, cached=1, so the new page could ask for a cheaper read. That is it. The old page changed in exactly two places: a button and a card that link into the new module.

20 stacked sections on the old page
10+ different renderings of one candidate
1 backend change, an additive cached flag
0 migrations

Look at what that buys. If Voucher HQ is wrong, nobody is stranded; the old page still works exactly as before. If Voucher HQ is down, nothing else is. And if I decide the whole thing was a mistake, rollback is deleting a link. Not a migration rollback, not a deploy, not a war room. Deleting a link. That is why I could ship it without asking anyone. There was nothing to ask permission for.

Key insight: The license to rewrite a live system does not come from testing the new version. It comes from building the new version in a layer that cannot break the old one, no matter how wrong it is.

The same shape in healthcare

I trusted the rule because I had already used it in a different industry. I run a denial management agent called Daniel in healthcare revenue cycle. When it came time to wire Daniel into a live portal, same rule. The integration reads what it needs, and the only runtime write in the whole thing goes to its own audit table. It cannot corrupt a claim, because it cannot reach one. Rollback there is flipping an environment variable.

The result: Both rewrites shipped into live systems, in different industries, with zero disruption to the thing they sat next to. Not because the new code was perfect, but because the old code could not tell it existed.

Workforce development and healthcare revenue cycle have nothing in common on the surface. One moves candidates through job training. The other moves claims through insurance companies. But the risk has the same shape in both: a live system people depend on, and a new thing that wants to sit beside it. The answer has the same shape too. Constrain the new thing until its blast radius is zero, and you can ship it whenever you want.

What a rewrite should cost

If your rewrite needs a migration, you are not rewriting. You are gambling.

Tests do not earn you the rewrite. Tests tell you the new thing probably works. They say nothing about what happens to the old thing when the new one turns out to be wrong. For a live system, that is the only question that matters: what breaks if this is garbage? If the answer is nothing, you delete a link. And if the answer is nothing, you can ship on a Tuesday afternoon without asking anyone.

Earn the rewrite with blast radius, not test coverage. Zero blast radius is the license.
Reuse every contract the old system already has. If the old API cannot power the new UI, that is a separate project with its own risks. Do not smuggle it into the rewrite.
Make rollback a deletion or a flag flip, never a procedure.
If the plan includes a migration, stop calling it a rewrite. A migration is a bet, and bets need a different kind of approval.

The old page is still up. It still has its twenty sections. People drift over to the new module a little more each week. One day the traffic will hit zero, and I will retire the old page the same way I could have retired the new one: quietly, by deleting a link. If you are staring at your own twenty-section page, do not wait for permission. Build where you cannot break anything, and you will not need it. That is what a rewrite should cost.

Originally published at nabbilkhan.com.

Make the Wrong Answer Cheap

Nabbil Khan — Sat, 18 Jul 2026 18:50:01 +0000

I run two inbox agents. The one nobody supervises acts on anything above 0.7 confidence. The one with a human review queue behind it refuses to act below 0.8. Every engineer I have explained this to assumes I typed the numbers backwards. I did not. The agent with no safety net earned the lower bar, and how it earned it changed the way I think about autonomy.

The backwards numbers

The first agent reads the inbox for a pharma and biotech events business. When a lead comes in, it classifies the email, upserts the contact by email address, and creates an opportunity in the CRM. Nobody checks its work. No queue, no reviewer. Its floor is 0.7.

The second reads the inbox for a workforce development program. It matches candidates to funding vouchers and moves their status in a tool I built called Voucher HQ. That tool has a real review queue with a real person working it. The agent only auto-applies a decision at 0.8 or higher. Anything below that lands in the queue, and a human decides.

So the supervised agent is timid and the unsupervised one is bold. For a while that bothered me. It looks like a mistake. It is the most deliberate pair of numbers in either system.

What a wrong answer costs

The confusion clears up when you stop asking how confident each agent is and start asking what happens when it is wrong.

Take the events agent. It upserts by email address, so it cannot create a duplicate contact. It skips any contact with an open opportunity, so it cannot stomp on a live deal. Walk the failure all the way through: the model misreads an email, fires at 0.72, and the result is a note on a record that already existed. Someone sees it later and shrugs. The wrong answer costs almost nothing, because the writes were built to absorb it.

Now take the workforce agent. When it is wrong, a real candidate moves into a real column in a pipeline that ends in money. A voucher attaches to the wrong person, a caseworker acts on it, and unwinding that means phone calls and apologies. Same kind of model, same kind of inbox. A completely different price for a false positive.

Key insight: Set an agent's threshold from the cost of its wrong answers, not from how nervous it makes you.

The result: The events agent runs with no human in the loop, and the worst a bad call can do is leave an extra note on a contact that already exists. The autonomy came from making the mistakes small, not from making the model sure.

The dial and the wall

A threshold feels like control. It is one number, you can move it, and moving it up makes you feel safer. But it does not change what the agent does when it is wrong. It only changes how often it acts. The dial measures your nerves, not your risk.

A confidence threshold is a dial you turn when you are nervous. Idempotence is a wall you build so you do not have to be.

The wall is different: upserts instead of inserts, guards that refuse to touch live records, actions that degrade into notes instead of state changes. These change what a mistake is. I did not buy the events agent's autonomy with a number. I bought it by making its worst day cheap.

I keep meeting the same idea in work that has nothing to do with inboxes. My denial engine reads medical claim denials and recommends what to do next. In its early eval runs it picked the right operational action 36.7% of the time. My first instinct was the dial: sterner prompts, more warnings, trust it less. The fix that worked was a wall. I anchored every recommendation to a deterministic per-code action table, so the model stopped inventing actions and started selecting from ones that were already correct. Action accuracy went to 76.7%, and classification accuracy on the same runs was 93.3%.

What broke: The denial engine's early runs picked the right action 36.7% of the time, and I burned real hours turning the dial: tighter prompts, more caveats, more nervous instructions. None of it moved the number much. The action table did.

0.7 confidence floor, no human in the loop
0.8 confidence floor, human review queue behind it
76.7% action accuracy, up from 36.7%
93.3% classification accuracy on the same runs

What I check now

Before any agent of mine goes autonomous, I walk its failure to the end. Not the demo path, the wrong path. What exactly happens, record by record, when it fires on a false positive? If the answer is a note, it can run alone. If the answer is money moving or a person being acted on, it gets a queue, a higher bar, or both.

Price the false positive before you set the threshold. The number means nothing on its own.
Buy autonomy with structure. Idempotent writes and guards on live records beat any amount of confidence.
Spend human review where mistakes touch money or people. A queue is a scarce resource, not a default.
When accuracy is low, fix the structure before you turn the dial. The dial mostly measures how you feel.

I run systems in events, in workforce development, in healthcare, and this pattern holds in all of them. That is how I know it is a pattern and not a preference. The agents I trust most are not the ones that are most sure. They are the ones whose mistakes I made too cheap to fear. If your thresholds keep creeping up, stop and price a wrong answer first. And if you have built walls that beat the dial, I want to hear about them. The builders doing this for real need to find each other.

Originally published at nabbilkhan.com.