DEV Community: Nabbil Khan

Nobody Asked the Model

Nabbil Khan — Mon, 10 Aug 2026 14:06:55 +0000

For months my eval dashboard said my skills scored 0 percent, and I believed it. Zero is a believable number for a young system. So I rewrote prompts. I restructured skills. I tightened trigger phrases, reran the suite, and watched the zero not move. The whole time, no model was ever asked a single question. The harness had quietly swapped credentials on me. Every request died at the front door with a 401, and every 401 was scored as a wrong answer. I spent months improving the answers of a model nobody asked.

One line of fallback

The bug was one line. Line 90 of run_eval.py had a fallback: if no API key was set, use the OAuth token from my Claude Code session as an API key. It looks harmless. It is the kind of line you write at midnight so a script will run on a fresh machine. But an OAuth token is not an API key. The API rejected every request with a 401, the harness caught the failure, and it had nowhere to put it except the answer column. Wrong credential became wrong answer. Wrong answer became 0 percent. And 0 percent became a fact I planned around.

What should have tipped me off was that the number never moved. Real evals are noisy. Skills improve a little, regress a little, flake on one case out of twenty. A metric that sits at exactly zero for months is not describing a model. It is describing something upstream that fails the same way every time. Flat numbers are plumbing numbers.

0% reported skill accuracy, for months
401 what every eval case actually returned
1 line of fallback code behind the whole number
0 questions any model was asked

What broke: I treated 0 percent as a fact about my skills for months. I rewrote prompts and restructured workflows to move a number that was actually about a credential. The fix was deleting one line, and the score I had been chasing had never once been about the work.

Two kinds of fallback

I build and run AI systems in several industries: a medical billing operation with a denial engine, an autonomous ad operator named Mia, a quoting engine for water-feature contractors, and the harness that evals my own agents. The same shapes keep turning up in all of them, and this bug taught me to sort fallbacks into two kinds.

A fallback in a compute path costs you quality. The big model times out, so you route to a smaller one. The cache misses, so you recompute. You get a worse answer or a slower one, but it is still an answer about the right subject. Degrading there is fine. Degrading there is the point.

A fallback in an identity path costs you the truth. Identity is the part of the system that says who is talking: which credential, which user, which account. When that part fails and something helpfully substitutes a different identity, nothing crashes. The run finishes and hands you a confident number about the wrong subject. My 0 percent was not a measurement of my skills. It was a measurement of a token, laundered through an eval harness until it looked like a measurement of my skills.

Key insight: A fallback in a compute path costs you quality. A fallback in an identity path costs you the truth, because the run still finishes and hands you a confident number about the wrong subject.

The same shape in an approvals dashboard

A while later the same shape showed up in my ad operation. Mia is my autonomous advertising operator. Mia drafts the moves, and a human approves the risky ones through a dashboard behind Cloudflare Access, which stamps every request with a JWT that says who you are. The tempting fallback was obvious: if the JWT is missing, fall back to a session cookie so the dashboard keeps working.

I said no, and the eval bug is why. An approval attributed to a person who did not approve is worse than a login screen. A login screen costs you a minute. A false approval is a record that lies, and it lies in the one place you will later go to find out what happened. When identity fails, the only honest state is blocked. So the dashboard fails closed. No JWT, no page.

The result: The Mia approvals dashboard fails closed behind Cloudflare Access: no JWT, no page. Every approval in its history is attached to a person who actually clicked it.

The rule under both systems is the same one.

A 401 is not a wrong answer, and a missing JWT is not an approval.

An infrastructure failure is not a verdict about the work. My denial engine draws the line in the same place: its audit trail records which tools actually ran, never which tools the model claims it ran. It is the same line, drawn in three different industries.

Ask who answered

The fix took five minutes. I deleted the fallback and made the script refuse to run without a real key. Auth errors now kill the run loudly instead of flowing into the score. The first honest eval was strange to watch, because the skills were fine. They had always been fine. Nobody had asked them anything.

What I took from it:

Before you ask what a number means, ask who answered. Sometimes nobody did.
Fall back on compute. Fail closed on identity.
Keep error channels out of the answer column. A 401 is infrastructure, not a verdict.
Distrust flat metrics. Real systems are noisy, and a number that never moves is usually measuring the plumbing.

If you run agents in production, go read your fallbacks. Most of them are fine. You are looking for the ones that swap who is talking, because those will hand you a confident number about the wrong subject, and you will believe it, like I did. The skills were never the problem. Nobody asked the model.

Where in your stack does a failed credential quietly turn into a different answer?

Originally published at nabbilkhan.com.

The Card Said AI

Nabbil Khan — Mon, 27 Jul 2026 14:03:12 +0000

The card said to use AI to build a quarterly compliance report out of a client's event platform. Before I wrote a line I cloned the app. Laravel, about 22,000 files. I spent a day reading it instead of building. There was already a scheduled tracker running. There were attendee exports nobody opened. There were seventeen compliance sheets sitting in the repo that nobody had connected to the ask. The report was most of the way built. It just had no name and no owner, so from the outside it looked like it did not exist.

What the card actually meant

The card said AI. What the client meant was that the quarter closes and somebody sits down for three days with spreadsheets and dreads it. Nobody wants a model. They want the dread to stop. Those are different deliverables, and if you take the first one at face value you will build a beautiful thing that solves nothing.

So I did the boring part. I took the compliance template, all 147 columns of it, and went column by column against what the platform already held. Forty seven mapped straight across, same field, different name. Sixty one were derivable, a join or some date math or a rollup away. Thirty nine were real gaps, things the platform never collected at all.

147 template columns triaged
47 mapped directly
61 derivable from existing data
39 real gaps in the data

The triage took longer than the code. And when it was done there was nothing left for a model to do. The mappings were fixed. The derivations were arithmetic. The gaps were gaps, and no model can invent a field nobody collected. It can only write a confident sentence where the field should be.

The mapping was not ambiguous. It was just unknown, and those two feel identical until somebody does the work.

Why I took the model out

I had already wired one in. Pulling it back out felt like losing. Then I asked the question that decides these things: does this report have to come out the same way twice?

Yes. Somebody pulls a compliance report a year later, somebody who was not in the room, and it has to match the one that was filed. Not roughly. Exactly. A model that is right 97 percent of the time is a model that will quietly contradict last quarter's filing, and you find out in an audit.

So it became a fixed alias table and a hash chained log. Every row traces back to the record it came from. Every run leaves a chain you can check. Boring. Replayable. When the auditor asks where a number came from, the answer is a row, not a probability.

Key insight: If the same input has to give the same answer twice, you do not want a model, you want a table.

What broke: I built the model path first because the card said AI, and I threw that day away. I should have read the codebase before I read the requirements.

The same shape keeps showing up

I keep running into this in industries that have nothing to do with each other. In healthcare the sharpest thing I shipped this month was not a model at all. It was a QR code that opens the patient app with no download and no password, plus one slide that says plainly which decisions we still need from the doctor. In pharma compliance it was a lookup table. In the medical billing work the model earns its keep in exactly one place, reading the messy denial letter, and everything downstream of that is deterministic on purpose.

The rule underneath is simple. Use a model where the input is genuinely unstructured and the output is allowed to vary. Everywhere else it is an expensive way to build a system you can no longer explain.

The result: Shipped the quarterly report with no model in it, fully replayable, and handed over the 39 real gaps as a list the client can go fix.

What I would do next time

Read the codebase before you read the requirements. Half of what you are asked to build is already there under a different name.
Do the triage by hand, once, column by column. It is slow, and it is the only thing that tells you whether the problem is hard or just unmapped.
Ask whether the output has to replay. If it does, make it deterministic.
Take the model out when the triage kills it, even though you built it and it feels like waste.
Hand over the gaps as a list. The 39 things the data does not contain are worth more than the 108 you automated.

Sometimes the whole contribution is proving the AI was not needed. You cannot bill for that easily and it does not demo well. But the client stops dreading the quarter, and the report replays the same way in an audit two years out. That is what they were asking for the whole time. They just used the only word anyone gives them for it.

What is your test for when a workflow is genuinely unstructured enough to deserve a model?

Originally published at nabbilkhan.com.

Tools an Agent Can't Use

Nabbil Khan — Mon, 20 Jul 2026 14:05:19 +0000

Last month I tried to fire Webflow. Not because it was slow, and not because it was expensive. The nonprofit site it runs, muslimscholars.us, was doing fine. I tried to fire it because its Designer is a canvas, and a canvas is a thing you drive with your hands. My workforce does not have hands. I run my companies with AI agents now, and any tool an agent cannot operate quietly turns back into my job.

A canvas with no door

Here is what I mean by cannot operate. Webflow has an API, but the API stops at content. It will update a blog post all day. It cannot move a div. The design itself, the layout, the classes, the breakpoints, lives inside the Designer, and the Designer is a rendered canvas. There is no API into it. So I tried browser automation next, the way you would drive any stubborn web app, and it bounced off. The canvas is a picture of the design, not the design. There was nothing underneath to click.

What broke: I spent days pointing browser automation at that canvas before I admitted defeat. Every selector I could reach was part of the frame, not the picture. The tool was not broken. It was sealed.

So the site is moving to code. Not because code is a better site builder. In most ways it is a worse one. But code is the one interface every agent already speaks. A change becomes a diff, and a diff can be reviewed, tested, and reverted by something that never sleeps.

Teaching Unity to work blind

A few days later I did the opposite. I build a VR app that trains pharmacy technicians. It runs on Quest, which means it is built in Unity, and Unity is a giant GUI editor. By the Webflow test it should have been next out the door. But I kept it, because what my agents actually needed from Unity was never the editor. It was one answer, over and over: does this C# still compile against this project? The editor knows, because the editor holds the real reference assemblies. So I pointed Roslyn at those assemblies directly, on a server.

The result: A headless server with no screen, no GPU, and no Unity license now typechecks the Quest app in half a second. Agents edit the C#, get compiler truth back instantly, and never open the editor at all.

One rule, two directions

Two opposite moves in the same week, in two businesses that share nothing: a nonprofit's website and a pharmacy training app in VR. That is the part that surprised me. The rule was the same both times. The difference is where the value lives. Webflow's value is the canvas; take that away and there is nothing left to wrap. Unity's value sits under the canvas: the compiler, the assemblies, the build pipeline. The GUI is just the lid. When the value is under the lid, you pry the lid off. When the value is the lid, you leave.

A tool only I can drive is not a tool anymore. It is a queue with one worker, and the worker is me.

Key insight: When your workforce is agents, the question that decides whether a tool stays is not what it can do. It is whether something that is not you can operate it.

0.5s Unity typecheck on a headless server
0 APIs into the Webflow Designer canvas
2 opposite fixes, one rule, one week

I used to judge software the way everyone does: features, price, polish. That list still matters. But it now has a gate in front of it, and most tools I look at do not get past the gate.

What I ask now

Ask who, or what, can operate it. If the answer is only you, it is a bottleneck wearing a nice interface.
Find where the value lives. Under the GUI, wrap it. In the GUI, replace it.
Do not settle for "has an API." Check that the API reaches the part the tool is actually for. Webflow has one. It just does not touch the thing Webflow is for.
Treat headless as a requirement, not a feature. It is the difference between a tool and a job.

The strange part is that neither tool changed. Webflow is the same product it was a year ago, and so is Unity. What changed is who shows up for work. Software has always competed on what it can do. It is starting to compete on what can do it.

Originally published at nabbilkhan.com.

Zero Is Not a Score

Nabbil Khan — Sun, 19 Jul 2026 00:20:18 +0000

The evals for my agent skills scored 0% for as long as I had records. Not low. Not noisy. Exactly zero, every skill, every run. And I believed it. For months I thought my skills were bad, because the number said so and the number never wavered. Then one night I actually read the harness. It had fallen back to the wrong auth token. Every call it made came back 401, and it quietly graded each one a failure. The skills never got a chance to fail on their own. I was not measuring them at all. I was reading a broken thermometer.

Real weakness is jagged

Here is what took me too long to see. When a system is genuinely bad, it scores 40% one week and 60% the next. It passes the easy cases and trips over the hard ones. It has good days. Incompetence has texture, because an incompetent system is still in contact with the world, and the world varies.

A flat number has no texture. A flat number means the measurement stopped touching the thing being measured somewhere upstream, and what you are reading is the instrument's resting state. Doctors know this. A heart monitor drawing a perfectly straight line does not mean the patient is calm.

Only a broken thermometer writes the same number every time.

Key insight: A performance number with no variance is a reading of the instrument, not of the thing being measured.

The same bug in three industries

I run systems in advertising, in healthcare billing, and in agent operations, and the same shape shows up in all of them. In advertising I found a dashboard figure that had been hardcoded for two years. Nobody questioned it, because it looked right, and it looked right because it never moved. In agent operations, an account-rotation bug in one of my pipelines overwrote every real error with the same generic message, "no active accounts," so for a while every distinct failure in that system looked identical. And in the denial-assessment engine I run for a medical-billing operation, an agreement metric came back at 44.7%, alarmingly low, until I noticed the two sides of the comparison were scored in different units. One side used an equivalence map; the other compared raw strings. Twenty-seven of the "disagreements" were the same operational decision written two ways. Rescored in units that meant something, the engine sat at 92.0% on classification and 90.7% on action against the golden set.

Three industries, one pattern. When a number is perfectly flat, perfectly round, or perfectly terrible, the odds you are looking at the system go down, and the odds you are looking at the pipe go way up.

0% skill eval score, every run on record
401 the status code behind every one of those zeros
44.7% an agreement score that was really a units mismatch
90.7% action accuracy once the eval scored meaning

What a healthy number looks like

The week after I fixed that eval, the numbers got interesting. Interesting is the point. Mapped-action accuracy on the golden set went from 36.7% to 76.7% across four runs in one afternoon of prompt work. Classification went from 80% to 93.3%. Errors went from one to zero. The line wobbled, jumped, and climbed, and every point on it told me something I could act on.

That is what contact with reality looks like. A healthy metric breathes. So I have inverted my instincts. Steady numbers used to feel comforting and volatile ones used to feel alarming. Now the number that never moves is the one that keeps me up at night. I judge the autonomous operators I run the same way. When one of them turns in the same figure day after day, I do not ask what it is doing wrong. I ask what stopped being measured.

What broke: I believed a broken harness over my own work for as long as I had records. Zero felt like an answer, so I never read the grader. The fix was one auth token. The cost was every decision I made off that number.

Grade the grader

The rule I run now is simple. Before I trust an eval, I make it prove it can pass. Feed it a case where the right answer is known and pinned, and watch it score a success. A harness that has never emitted a passing grade is not strict. It is dead. And dead harnesses grade perfectly, forever.

The result: The denial engine's eval now gates every ship at 90% against a golden set, and I trust that gate because I have watched it pass, fail, and change its mind as the system changed. It moves. That is how I know it is alive.

Distrust any metric with no variance. Jagged is what real looks like.
A flat zero is a flatline. Check the instrument's pulse before the patient's.
Grade the grader. Run a known-good case through every eval and watch it pass.
Never let a fallback swallow an error. One wrong token cost me months of belief.
Perfectly round and perfectly stable deserve the same suspicion as perfectly bad.

The zero on my dashboard was never a grade. It was an instrument talking to itself. The numbers I trust now are the ones that move, because a number that moves is still touching the world. Most of us grade everything except the grader. Start there.

Originally published at nabbilkhan.com.

Write Your Exceptions Down

Nabbil Khan — Sat, 18 Jul 2026 23:50:17 +0000

I had a rule with no exceptions. RETSBAN, my primary agent, runs local inference only: open models on my own hardware, no cloud, no fallback. If the GPU box is down, the agent is down. Then Mia, the agent that runs my marketing, needed a frontier model to do her job well. And I did something that felt strangely formal for a one-person company. I amended my own policy, in writing, with the date attached.

No one was in the room

Here is what took me a while to see. A human employee remembers the day you made an exception. They were in the room when you said fine, just this once. An agent was never in the room. There is no room. Every session starts cold from the files, and the files are the only memory the company has.

So when the policy says local only, no exceptions, and reality contains an exception, one of two things happens. Either the agent obeys the file and blocks work I actually want done, or it notices the contradiction and starts guessing which side to trust. The guessing is the dangerous case. A rule that has been contradicted once, silently, is not a rule anymore. Every agent that loads it gets to decide, in every session, whether to believe it. You never see the deciding. You just see the drift.

An exception that lives in your head does not bend a rule. It erases it.

The amendment

On 2026-04-23 I opened the policy file, a file literally named local_only_no_anthropic, and narrowed it instead of breaking it. The amendment names who is exempt: Mia, and only Mia. It says why: marketing work that needs capability the local stack does not have. It carries the date, and it points to the full model-topology doc for anyone, human or agent, who wants the whole picture. RETSBAN's constraint did not move an inch. Still local only. Still no fallback. Still down when the GPU box is down.

That is the difference between an amendment and a repeal. An unwritten exception repeals the rule and hides the repeal. A written amendment narrows the rule and makes it stronger, because now the rule has visibly survived contact with a real exception. The next agent to read the file does not have to wonder if the local-only line is stale. It can see the line was tested, on a specific date, for a specific reason, and held.

What broke: Before this I kept preferences in my head and re-explained them session after session. Every correction died with the session that heard it. I was giving the same review, over and over, to a workforce with perfect skill and no memory.

The same move in medical billing

This is not an agent-fleet quirk. Last week, in my medical-billing business, I retired a ship gate on the denial-assessment engine. The old gate required the tuned API to agree with a weaker agent rater. A 150-case run showed the gate measured the wrong thing: on genuine divergences the API matched the golden answers about 78 percent of the time, and the agent rater about 38. The gate could never pass, and passing it would have meant nothing.

I did not quietly stop running it. The change record says the gate was retired, says why, and says when the change was ratified: 2026-07-08. It says what replaced it: score the API against golden answers directly, with the floor at 90 percent. Whoever touches that system next quarter, human or agent, inherits the reasoning and not just the residue.

0 cloud calls RETSBAN has made, before and after the amendment
1 agent named in the exception
150 cases in the run that killed the old ship gate
90% the floor the replacement gate has to clear

The result: Nearly three months after the amendment, both rules hold exactly as written. RETSBAN has made zero cloud calls, Mia has exactly the access the file grants, and no session has argued with either fact.

Your judgment is production config

You version your code. You probably version your infrastructure. The strange part of running an AI agent workforce is that your own judgment joins the list. What you want, what you forbid, what you decided to allow anyway: the agents execute whatever the file says. So the file is the policy, and an edit to the file is a deploy.

Key insight: An exception you keep in your head deletes the rule. Amend the file instead, with the date, the scope, and the reason attached.

So this is what I do now, and what I would tell anyone running agents:

Put the rule where the agent reads, not where you think. A rule outside the files does not exist.
Amend, do not repeal. Narrow the rule and keep it.
Attach the date, the scope, and the reason. A future session cannot ask you follow-up questions.
The moment you catch yourself carrying an exception in your head, the rule is already gone. Writing it down is the undo.

A rule you never amended is not a rule. It is a rumor about a rule, and every session of every agent decides whether to believe it. Write your exceptions down. A rule is only as real as its history.

Originally published at nabbilkhan.com.

Who Owns the Row

Nabbil Khan — Sat, 18 Jul 2026 23:20:15 +0000

One of my systems runs a nightly job with two sets of manners. On 154 rows it overwrites whatever it finds and asks nobody. On 15 rows it cannot change a single field; the best it can do is file a request and wait. Same job, same feed, same table. The only difference is whether a human being ever touched the row.

Two doors

The system is a command center for a workforce development operation. The State of Illinois publishes a directory of workforce centers, and every night my sync pulls that feed and compares it against what I have. Most of those sites, 154 right now, are anonymous directory entries. Nobody on my side has ever opened one. For those, the feed writes straight through. New address, new phone, new hours, done. Arguing with the state about a row nobody looks at would be silly. The feed is the only party that knows anything about that row, so the feed decides.

Fifteen sites are different. Those are partnered centers, rows people on my side have filled with staff notes and visit history. Someone drove out there. Someone wrote down which door to use and who to ask for. When the feed disagrees with one of those rows, the sync does not write. It queues the change in a table called WorkNetPendingChange and waits for a person to approve it or reject it.

And nothing gets deleted, ever. A site that vanishes from the state's export just gets a missingSince date. Feeds hiccup. Exports get truncated. A date is reversible. A delete that cascades through visit history is not.

154 directory sites the feed overwrites freely
15 partnered centers where changes wait for review
0 rows the sync is allowed to delete

The row, not the feed

The usual way to design a sync is to ask whether the feed is the source of truth. That question hides an assumption: that authority belongs to the feed. It does not. It belongs to the row.

Think about what each side actually knows. The state knows what the state published. It knows nothing about the visit last Tuesday, or the note saying the listed phone number rings a fax machine. On a row nobody here has touched, the feed's knowledge is all the knowledge there is, so it should win every conflict. On a row people have worked on, the feed holds the smaller share, and letting it overwrite means destroying the larger one.

Authority is not a property of the feed. It is a property of the row.

Key insight: Give a sync write authority row by row. It overwrites freely where no person has done any work, and it only gets to propose changes where someone has.

What broke: Every sync I have ever regretted failed the same way. An upstream export had a bad night, and the job faithfully copied the bad night over months of human work. The overwrite took milliseconds. Getting back what a coordinator knew about a site took weeks, when it happened at all.

The same rule keeps showing up

Once I had the rule, I started seeing it all over my stack. The same command center runs an inbox agent that triages a shared mailbox. Routine messages it handles by itself. Anything that touches a real relationship goes to a review queue. Two doors again.

My publishing pipeline works the same way. Work-log entries publish to the enterprise timeline on their own, because nobody's judgment rides on a changelog line. Public essays, the ones that carry my name, wait for approval. And in a different industry entirely, the denial-assessment engine I run in medical billing acts directly on the mechanical cases and calls escalate_to_human on the few where judgment actually lives.

Three corners of my work, one rule. Automation gets full authority over work no person has touched. The moment a person touches it, the automation drops from writer to proposer.

The result: The sync keeps 154 rows fresh with zero human effort, and every drift against the 15 partnered centers gets human eyes before it lands. Fresh and safe at the same time, without trading one for the other.

What to build

None of this is exotic. It is a WHERE clause and a pending-changes table. The hard part is deciding to build it, because "the feed is the source of truth" sounds so clean. If you are wiring an external feed into a system people actually work in, here is what I would do:

Give write authority per row, not per feed. Source of truth is a row-level question.
Let human work demote the feed. The first staff note on a row turns the feed from a writer into an advisor.
Never let a sync delete. Record the absence and let a person decide what it means.
Make every change pick a door: direct write or review queue. No third path.
Keep the queue short enough that people actually read it. A review queue nobody reads is just a delete with extra steps.

Machines should own what nobody cares about, so people can own what they do. Go look at which rows in your database have fingerprints on them. Those are the ones the feed should have to ask about.

Originally published at nabbilkhan.com.

Who Owns the Clock

Nabbil Khan — Sat, 18 Jul 2026 22:50:14 +0000

A patient asked why she got two reminder texts a day when everyone else got one. Fair question. She had two open episodes in my Medicare RTM engine, and I had written the reminder logic to run once per open episode. Two episodes, two clocks, twice the nagging. The scheduler fired exactly on time. It always does. A few months later, in a different industry, I caught myself starting to write the same bug again. That was when I finally saw what the bug was.

The scheduler is innocent

When a cadence goes wrong, the scheduler is the first place everyone looks. I looked there too. I read the cron expression. I checked the timezone. I hunted for drift. Everything was fine, because cron is almost always fine. The scheduler is the most audited hundred lines in any system I run. The bug was somewhere I had never thought to look, because I had never noticed I was deciding anything there. I had let the episode own the clock.

The fix was one sentence long: remind once per patient, not once per open episode. A patient does not experience episodes. She experiences her phone buzzing at dinner. The moment I said the fix out loud I could hear how obvious it was, and that is exactly why it survived review. It did not look like a decision. It looked like plumbing.

What broke: A patient with two open episodes got double the reminders because I let each episode own its own clock. The scheduler was innocent. The ownership was wrong.

The same bug in a different industry

Then came workforce development. I built an outreach system where coaches call candidates about medical assistant training. Different industry, different codebase, not one shared line between them. The system has a cadence gate: how long a coach has to wait before touching the same candidate again. My first draft reset that clock on every logged attempt.

Think about what that does. A coach leaves five voicemails in a week. Each one resets the clock, so the system decides the relationship is warm. It stops prompting follow-ups for someone nobody has actually talked to. Five voicemails, and the candidate sinks quietly in the queue, marked fresh by calls she never answered.

This time I caught it before it shipped, because the RTM bug had taught me the question to ask. The fix was again one sentence: the gate resets only on one outcome, reached. Attempts write notes. They do not write time.

A voicemail is not a relationship. It is you, talking to yourself, on the record.

The result: Five logged voicemails now write five notes and move the clock zero seconds. One verb resets the gate: reached. The system stopped mistaking effort for contact.

Nouns and verbs

Healthcare and workforce development. Medicare billing rules on one side, career coaching on the other. Zero shared code, identical bug. Both times I went hunting in the scheduler, and both times the scheduler turned out to be a metronome doing its job. The defect lived in two words I had chosen without noticing I was choosing: which noun owns the clock, and which verb is allowed to reset it.

In the RTM engine the noun was wrong. The episode owned the clock when the patient should have. In the outreach system the verb was wrong. "Tried" would have reset the clock when only "reached" should. That is the whole taxonomy. Every cadence bug I have shipped, in any industry, has been one of those two words.

Why did neither bug look like a bug? Because neither decision looked like a decision. The noun hides in a foreign key. The verb hides in a WHERE clause. Review catches bad logic, and this was not bad logic. It was bad grammar, and the grammar compiled.

Key insight: Every cadence bug lives in two words, not in the scheduler: which noun owns the clock, and which verb is allowed to reset it.

2 open episodes, one patient, twice the reminders
5 voicemails that moved the clock zero seconds
1 verb allowed to reset the outreach gate
0 lines of code shared between the two systems

What I do now

When I build anything with a clock in it, I write the two words down before I write the code. The noun goes at the top of the file. The verb goes next to it. Everything else is implementation.

Name the noun that owns the clock, and make it the noun a human experiences. Patients feel phones buzz. They do not feel episodes.
List every verb allowed to reset the clock, then cut the list. Most clocks deserve exactly one verb.
Let every other verb write a note instead of time. History is cheap. Resets are expensive.
Test the plurals: two episodes, five attempts. The singular case always passes, which is why it proves nothing.

The scheduler was never the interesting part. The interesting part is the sentence you wrote without noticing you were writing it. So when a cadence goes wrong, do not read the cron expression first. Read your nouns and your verbs. And if you run systems in more than one industry and keep meeting the same small bug in different clothes, write it down and tell someone. The builders need to find each other.

Originally published at nabbilkhan.com.

Watching or Working

Nabbil Khan — Sat, 18 Jul 2026 22:20:12 +0000

Until this week, the console that files all my Medicare claims had no way to sign out. Plain forms, no animation, not one hour of visual design. Meanwhile the marketing site for the same product got choreographed scroll animations and hand-built React recreations of those very screens. The copies are prettier than the originals. That sounds like a scandal. I think the money went exactly where it should have, and most software points it the other way.

Where the polish went

clearpathcare.ai is the prettiest thing I have ever shipped. Every scroll position is choreographed with GSAP and Lenis. Every console screen was rebuilt by hand in React so it could slide in at just the right moment. Days of art direction for a page most people will see once.

The console behind it, the tool that enrolls patients, logs the calls, and files the claims, got none of that. Zero hours of visual design. Not because it matters less. It matters more; it is the part that makes the money.

The difference is what the person in front of each screen is doing. A visitor to the site is leaning back. They are judging, and they decide in about thirty seconds whether you are real. Looking good is the whole job. A biller in the console is leaning in. They are there all day, and every animation between them and the claim is friction. The best thing that screen can do is get out of the way.

The koi fish

I know what happens when you get this wrong, because I got it wrong once. MOTOR is a quoting engine I built for construction contractors. At some point I gave the quoting flow a koi companion: a little animated fish that swam along while you built a quote. I thought it was charming.

Contractors killed it inside three quotes. The feedback was not a design critique. It was "pretty annoying" and "just give me the quote."

What broke: MOTOR's quoting flow got an animated koi fish to keep users company. Contractors killed it inside three quotes. The verdict was "pretty annoying" and "just give me the quote." Decoration inside a work tool is a tax you charge on every single use.

Just give me the quote. That sentence taught me more about design than any book. A worker does not experience your polish as a gift. They experience it as one more thing standing between them and being done.

The rule

So here is the rule I use now. Do not ask how important the system is. Ask what the person in front of it is doing. Watching? Spend on looks. Working? Spend on speed.

By that rule, the console being the most important thing I own and the plainest is not a contradiction. It is the rule working.

Key insight: Spend design where people watch and speed where people work. The budget follows what the user is doing, not how much the system matters.

Polish is for people who are watching. Speed is for people who are working.

This does not mean work tools get no design. This week the console finally got real design attention, and every bit of it was invisible: the browser back button works now, breadcrumbs show where you are, a confusing device field got a sane default and a plain hint. Nobody will screenshot any of it. Everyone who works in it will feel it.

And when the console did need to be watched, because prospects wanted to see it run, I did not decorate the console. I bolted a guided demo onto the side: nine chapters, about forty steps, strictly read only. The show got its own stage. The tool stayed plain.

The result: The console's guided demo shipped as nine chapters and roughly forty read-only steps. The show got its own surface, and the screens people actually work in stayed plain and fast.

3 quotes before contractors killed the koi fish
0 hours of visual design in the console that files the claims
9 chapters in the guided demo, a surface built for watching
40 steps of polish kept out of the working screens

The test

I run systems in healthcare, in construction software, and in marketing, and the same one-line question now gates every design hour in all three: is the person on the other side of this screen watching or working?

Ask watching or working before you spend a single design hour.
Spend looks on screens people judge and leave. Spend speed on screens people live in.
Work tools still deserve design, the invisible kind: a back button that works, a sane default, a breadcrumb that tells you where you are.
If a work tool needs to be watched, build the show a separate surface, like a demo, and keep it out of the flow.
When a working user calls something pretty annoying, believe the annoying, not the pretty.

Step back far enough and the pattern is simple: the closer a screen sits to the money, the plainer it gets. If the part of your product that does the real work looks embarrassingly plain, you are probably spending right. And if you run an ugly system that quietly earns, I would like to compare notes, because we are making the same bet. The money is made in plain rooms. The pretty rooms are how people find the door.

Originally published at nabbilkhan.com.

Underpowered Tests Lie

Nabbil Khan — Sat, 18 Jul 2026 21:50:11 +0000

Last month I audited the statistics inside Mia, my ad-testing agent, and found a 20x error that had never broken anything. No exception, no failed assertion, nothing in the logs. The textbook shortcut said each arm of an A/B test needed 1,111 users. The exact formula, run at the 1.9 percent click-through rate ads actually get, said 22,278. Every test Mia had sized was running on a twentieth of the data it needed, and every one of them still announced a winner. I run AI systems in several industries, and the failures that cost the most all have this shape: nothing breaks, and the answer is wrong.

Where the shortcut came from

The shortcut is Lehr's equation: sixteen over the effect size squared. It is in the textbooks because it is easy, and it is easy because it was derived where the math is friendliest, near coin-flip rates, where variance peaks and everything is symmetric. Ads do not live there. Ads live at 1.9 percent, where the effects worth catching are tiny in absolute terms and the samples you need to see them are huge. The shortcut knows none of this. It returns 1,111 and moves on.

The part that took me longest to accept is that the formula is not wrong. It works fine in the world it came from. The bug was mine. I carried it into a different world without re-deriving it.

1.9% the base click-through rate Mia lives at
1,111 per-arm sample the textbook shortcut prescribed
22,278 per-arm sample the exact formula actually requires
14% real false-positive rate of a three-arm test at raw p < 0.05

What an underpowered test says

You would think an underpowered test would fail. It cannot. There is no runtime check for statistical power. The test collects its too-small sample, computes a p-value, and hands down a verdict in the same confident voice it would use with twenty times the data. Sometimes the verdict is even right. That is what makes it dangerous. A crash gets fixed the same afternoon. A confident wrong answer gets acted on.

An underpowered test never says it does not know. It says B wins, and it says it with a straight face.

The audit found a second lie. Mia judged three-arm tests at raw p below 0.05, which sounds like a 5 percent false-positive rate. With three comparisons it is closer to 14 percent. One test in seven was crowning a winner that did not exist. Holm-Bonferroni fixes that in a few lines of code. Writing the lines took minutes. Suspecting I needed them took an audit nobody asked for.

What broke: The 20x sizing error and the 14 percent false-positive rate sat in production together, and the code ran green the whole time. Every signal I could see said the system worked. The only thing wrong was the answers.

The same lie in healthcare

I caught this in ads, but I only recognized it because I had already paid for it in healthcare. Daniel, my denial-assessment agent, is graded against a golden set of 30 cases. When a prompt change moved classification accuracy from 80 to 93.3 percent, I did not believe it until the eval said the same thing four runs in a row. Thirty cases is a small sample. A single run there can lie as smoothly as an underpowered ad test, and it lies the same way: not with an error, with a number.

That is the cross-industry pattern. It has nothing to do with ads or insurance claims. It is about verdicts that sound the same whether or not they earned it. Ad tests, eval harnesses, dashboards: none of them has a voice for "not enough data." If you want that voice, you have to build it yourself.

The result: Mia now sizes every test with the exact unpooled formula at the observed base rate, refuses to declare a winner before that sample is met, and applies Holm-Bonferroni across arms. The fix took a day. Finding it took an audit nobody asked for.

Re-derive at your base rate

Key insight: Every statistical rule of thumb is calibrated for someone else's base rate until you have re-derived it at yours.

That is the rule I run on now, in every industry I touch. Here is what it looks like in practice:

Re-derive the formula at your own base rate before you trust it. The textbook derived it at theirs.
Ask what the system says when it does not know. If the answer sounds the same as when it does, you have a liar, not a tool.
Count your comparisons. Three arms at raw p below 0.05 is a 14 percent false-positive rate, not 5.
Audit the judge, not just the work. The most expensive bugs live in the thing doing the grading.
Treat green as a timestamp, not a verdict. It means the code ran, not that the answer is right.

The tests that fail loudly are the cheap ones. You see them, you fix them, you move on. The expensive ones succeed quietly at the wrong thing, for months, while everyone trusts them a little more each week. If your experiments run at base rates the textbook writers never pictured, this is the week to re-derive your formulas. And if you catch one of these lies in your own stack, write it up. The operators who actually check are rare, and we should find each other. The math was never the hard part. The hard part is doubting a number that looks right.

Originally published at nabbilkhan.com.

Trust the Calculator

Nabbil Khan — Sat, 18 Jul 2026 21:20:09 +0000

The pricing formulas in Motor, the estimating engine I built for a water feature shop, did not come from the manual. I pulled 32 of them out of the JavaScript behind Aquascape's contractor calculator, the tool contractors actually use to bid jobs. The manual was sitting right there, official and free. Ignoring it was the best design decision in the whole system.

A vendor never ships a sloppy calculator

Why trust the calculator over the manual? Because of what happens when each one is wrong. If the manual sizes a pump wrong, a reader shrugs and moves on. If the calculator sizes a pump wrong, a contractor bids a job at that number, wins it, and loses money on the install. Then the phone rings. So calculators get fixed and manuals drift. Give it ten years and the two quietly disagree, and everyone in the trade knows which one to trust without anyone saying so.

A vendor will ship a sloppy PDF. They will never ship a sloppy calculator.

Documentation is what a domain says about itself. The artifacts money flows through are what it actually believes. Once you see that split, you cannot stop seeing it.

The other half was in old invoices

Formulas only get you to cost. What a shop charges on top of cost is a belief about its market, and no vendor document holds that number. So I pulled 132 historical quotes out of the shop's CRM. Real quotes, sent to real customers, most of them paid. I calibrated Motor's markup against those, then checked its output against what the shop had actually charged.

The result: Calibrated against 132 real quotes, Motor's estimates landed within 5 percent of what the shop actually charged, with no pricing rule taken from documentation.

I could have just asked the owner what his markup was. But what an owner says and what his invoices show are rarely the same number, and the invoices are the ones customers paid. When the two disagree, believe the invoices.

The same bug in a different industry

I build and run systems in several industries, and the surprising part is how often the same small idea matters in all of them. This year it showed up in medical billing. I run a denial engine that reads insurance denials and recommends the next move: appeal, fix the coding and resubmit, bill the patient, or write it off. The first version reasoned from guidance prose, appeal-strategy text that reads like documentation. On a golden set of 30 adjudicated cases, it picked the right action 36.7 percent of the time.

What broke: The engine kept recommending appeals for coding errors that should just be fixed and resubmitted, because the guidance said appeal and the model believed it. It scored 36.7 percent on action selection. I had banned documentation from my inputs, then smuggled it back in as a prompt.

The fix was the same move as the calculator. We keep a reference layer that records what experienced billers actually do with each denial code, distilled from real worked claims. I anchored the engine to that instead of the prose. Action accuracy went from 36.7 percent to 76.7 percent in one evening, and classification went from 80 percent to 93.3 percent. The model did not get smarter. The spec got real.

32 formulas pulled from the contractor calculator's JavaScript
132 real quotes used to calibrate markup
5% gap between Motor's estimates and actual shop pricing
76.7% denial-action accuracy after anchoring to worked claims, up from 36.7%

Two records

Every domain keeps two records of itself. One is written for readers: manuals, best-practice guides, onboarding docs. The other is written for money: calculators, invoices, worked claims. The first record is what the domain wants to be true. The second is what it paid to learn. When you encode a domain into software, you are choosing which record to believe, whether you know it or not.

Key insight: Encode the rules a domain risked money on (the shipped calculator, the paid invoice, the worked claim) and treat everything it merely wrote down as a rumor.

Pull your rules from the artifacts people bid with. The calculator beats the manual it shipped next to.
Calibrate against money that actually moved, and measure the gap in percent, not in vibes.
Treat documentation as a hypothesis about the domain, never as the spec.
When your model gets a domain wrong, check what you fed it before you blame the model.

None of this is really about ponds or insurance claims. It is about where truth lives in a domain. A domain will tell you anything in its documentation. What it bids with is what it believes. Build from that. And if you are pulling formulas out of someone's calculator at midnight, in an industry nothing like mine, we should compare notes.

Originally published at nabbilkhan.com.

The Signup Form

Nabbil Khan — Sat, 18 Jul 2026 20:50:08 +0000

On April 3 my Kindle pipeline went green. The EPUBs passed epubcheck, the print PDFs matched KDP's trim spec, and the covers came out with spine widths computed from the page counts. Zero to green in one day. It is the middle of July now, and that pipeline has not published a single book, because the only thing between it and a live listing is a signup form that wants a Social Security number and a bank account. The code took a day. The form has taken a quarter.

The same wall

I told myself this was a publishing problem. Then it happened again in professional services. Speckles is an RFP agent I built. You hand it a request for proposal and it answers from the source decks, with citations back to the exact slides it drew from. Empty repo to verified, cited answers in one day. It is still not live. Going live needs a BotFather token from Telegram and three decks that sit behind a Canva login. Not a line of code. A token and a login.

Then I remembered I had seen this wall before and misread it. In medical billing, the software to work a denial comes together in days. Getting credentialed on a payer portal takes months. For years I filed that under healthcare being healthcare. It was the same wall. I just could not see it, because back then the code was slow too, and the paperwork hid behind it.

Publishing books, answering RFPs, appealing denials. Three businesses with nothing in common, and the same wall in all of them. The wall is me.

The machine was ready in a day. I was the part that took a quarter.

1 day zero to a green Kindle pipeline
1 day zero to a verified, cited RFP agent
3 months the green pipeline waiting on one signup form
3 decks stuck behind a single Canva login

What only a person can produce

Look at what is actually blocking. A tax ID. A bank account. An OAuth login. A messaging token tied to an account tied to a phone number tied to a person. None of it is hard. All of it needs a legal human identity, and a venture only gets the identities its founder brings.

Why can't the machine do this part? Because these systems exist to prove a person is behind the button. The form that wants my SSN is not badly designed. It is doing exactly what it was built to do. It marks the line where the economy stops trusting machines and asks for a human it can tax, sue, or pay. AI moved every other line. It cannot move this one, and it should not.

So the critical path of a new venture has quietly changed shape. It used to be months of engineering with paperwork sprinkled through, and the paperwork got done while the code was still being written. Now the engineering compresses into days, and the paperwork stands alone, exposed, as the thing everything else waits on.

Key insight: Once AI does the building, the critical path of a new venture is no longer engineering. It is the short list of artifacts only a legal human identity can produce.

Build to the boundary

Here is the discipline I landed on. Build everything up to the auth boundary, then stop. Every project now ends the same way: a green machine, parked and verified, plus a short list titled PENDING (user). The list holds only things a machine cannot do. Sign up for the account. Enter the SSN. Approve the OAuth grant. Export the decks.

The point of the list is not organization. The point is honesty. When the human blockers stay buried inside a project, they look like engineering, and I respond by engineering more.

What broke: The Kindle toolchain sat green for three months while I kept finding code to polish. It was done on April 3. I was improving the part of the project that was no longer the bottleneck, because polishing code feels like progress and typing my SSN into a form does not.

The result: Speckles went from an empty repo to an RFP agent with verified, cited answers in one day. Everything a machine could produce, it produced. Its pending list has two entries, and both of them are mine.

What I do now

Build to the auth boundary, then stop. Once the machine is green, more engineering is avoidance.
End every project with a green machine and a written PENDING (user) list. Blockers that stay unwritten disguise themselves as work.
Schedule the paperwork like the critical path it is. Right now a form is worth more than a feature.
Find the wall before you start. If a venture is going to stall at a signup form, better to know on day one than on day ninety.

For most of my working life, the scarce thing was working software. I have a Master's in Language and Theology and I ran a 32-rig mining farm before I wrote production code, so I never assumed software would be the easy part. Now it is. Green machines stack up at the boundary like planes waiting on one runway, and the runway is one person with a Social Security number. The constraint is not intelligence anymore, and it is not code. It is the number of hands that can sign. If your machines are parked at the same wall, compare notes with me. The builders need to find each other.

Originally published at nabbilkhan.com.

The Model Is a Witness

Nabbil Khan — Sat, 18 Jul 2026 20:20:06 +0000

Two words in a subject line decide whether a training business made money. A DocuSign email that starts with "Completed:" means someone signed a voucher and the money is real. One that starts with "Complete with Docusign:" means someone was only asked to sign. I fed both to a language model and asked whether the voucher had come in. It said yes to both. To a model, a request to sign and a signed contract are the same sentence with slightly different punctuation. That near miss gave me the rule every money-touching pipeline I run is now built on: the model can recognize the event, but it does not get to define it.

Two words run a business

I did not find that difference in a spec or a data dictionary. I found it by reading the inbox where the business actually runs, thread by thread, the way you would read a ledger. The list of categories behind the inbox agent is hand-written from real messages: which senders carry money, which subjects are contracts, which attachments are just noise. And the one marker that matters, the one that tells booked revenue apart from a polite request, is a plain string check on the subject prefix. No temperature. No embedding. Just startsWith.

What broke: I almost booked revenue that did not exist. The model read "Complete with Docusign:" and reported the voucher as received. A plain prefix check now sits between that answer and the ledger, and it has never once gotten creative.

The model still has a seat. Just not that one. When a coach writes something human, like "signed and attached, sorry for the delay," no prefix rule will catch it. That is real ambiguity, and that is exactly the work the classifier should do.

The deterministic layer makes the law. The model gives testimony.

Enrich first, then reason

Daniel is my denial-management agent for the medical-billing operation. Completely different industry: healthcare revenue cycle instead of training, CARC codes instead of DocuSign subjects. Same shape. Before the model reads a single denial, a deterministic layer does the defining. The CARC code is looked up against a reference database. Payer rules attach. Bundling edits apply. The timely-filing math runs to the day. Only then does the model reason, inside a frame it cannot redraw.

The evals showed how much the seating matters. When Daniel's recommendations floated free, mapped-action accuracy on the 30-case golden set was 36.7%. Tying each recommendation to the reference database's per-code action (write off, bill the patient, correct and resubmit) moved it to 76.7% in one change, and classification went from 80% to 93.3%. The failures that were left were about definitions, not perception. The model wanted to appeal a bundling denial that should be corrected and resubmitted, and it kept writing off deductible balances that the reference layer knows must go to the patient. Every fix was the same fix. Take a definition away from the model and give it to the data.

36.7% action accuracy when the model defined its own actions
76.7% action accuracy after one deterministic anchor
92.0% classification accuracy at the ship gate
90.7% action accuracy across 150 real denial cases

The result: With the deterministic layer making the calls, Daniel cleared the ship gate at 92.0% classification and 90.7% action accuracy against golden across 150 real denial cases. Same model as before. Different seat.

A wrong yes books money that is not there.

Three industries, one seating chart

The rule was learned in training and proven in healthcare, and by the time I built MOTOR, my quoting engine for a water-feature contracting business, it was policy from day one. MOTOR prices jobs with the manufacturer's official formulas, behind three deterministic layers that keep it from ever quoting too low. The model never writes a number on a quote. It reads the messy site notes, the "sloped yard, maybe 16 by 11, client wants a stream" kind of thing that no formula can parse, and turns them into clean inputs. Then arithmetic does the pricing, because arithmetic does not round in the customer's favor just to be agreeable.

None of this is being down on models. Deterministic code cannot read "signed and attached," and a model cannot be trusted with "Completed:". Each one is unbeatable in its own seat and dangerous in the other's. Three industries, no shared regulators, no shared vocabulary, one shared risk: a wrong yes makes money that is not there, or a price that does not cover the job. So the categories get hand-written from real cases, the markers that matter get decided by code, and the model gets the genuinely ambiguous middle. A real job. A hard one. The only one it should have.

Key insight: In any pipeline where a mistake costs money, hand-write the categories from real cases, decide the markers that matter with plain code, and give the model only the genuinely ambiguous middle.

What three ledgers taught me

Read the real inbox before you write a prompt. The categories live in the mail, not in the model.
If a string decides money, check the string. A prefix comparison costs zero tokens and never hallucinates.
Enrich first, then reason. Codes first. Rules first. Deadline math first. Judgment last.
Give the model only the fuzzy middle. "Signed and attached" is classifier work. "Completed:" is not.
When an eval fails, ask which seat the failure was in. Most of Daniel's wrong answers were definitions the model should never have owned.

Training, healthcare billing, and outdoor construction share no regulators, no customers, and no words. They share one operator and one seating chart. And the chart holds because it was learned as a scar in one industry and carried as law into the next two. If you are building anything where a wrong yes makes money out of thin air, decide your prefixes in code, and let the model testify.

Originally published at nabbilkhan.com.