DEV Community: Kalpak Kale

What Dussehra knows about building with AI

Kalpak Kale — Tue, 20 Oct 2026 04:46:34 +0000

Every October the effigy goes up in the ground near my home in Akola, forty feet of bamboo and paper, and every October I watch it burn with a few hundred people and think about nothing at all. This year I thought about work. Not because the festival is about work, but because the story it tells is the most accurate description I know of what goes wrong when capable people, or capable machines, are left to their own devices.

In one line: Dussehra is a yearly review disguised as a festival: a checklist of ten ways judgement fails, a ritual for maintaining your tools, a boundary that must not be crossed, and a public deletion of what you have outgrown.

A word on the title. I am not going to tell you the festival is secretly a physics lesson. Most of what gets called the "science" of Indian festivals is wishful. What I am doing is narrower: reading the festival as a system that encodes practical knowledge, the way a checklist or a runbook does, and asking what that knowledge says to someone who spends their day building software with AI tools. Where I lean on actual research, I say so.

Ravana knew everything

The detail everyone forgets about Ravana is that he was the most learned figure in the epic. A scholar of the Vedas, a composer, an astronomer, a king who ran a wealthy city. The story is not about an idiot losing to a hero. It is about the most capable mind in the room losing because capability and judgement are different things, and he had only one of them.

That is the exact shape of the problem with today's AI tools. A coding agent has read more code than any human. It can produce a working feature in the time it takes me to describe it. And it will, with full confidence, invent a library function that does not exist, skip the input validation it was never asked about, or report "done" on something it has not tested. Not because it is stupid. Because knowing is not the same as judging, and the judging part was never in the model.

The practical consequence is the rule I wrote about in the brief post: read every diff. Not to write the code, to supply the judgement the tool does not have. Ravana had no one reading his diffs.

Ten heads, ten ways to be wrong

The popular reading gives Ravana's ten heads as ten vices: desire, anger, greed, attachment, pride, envy, and the restless mind, the intellect, the memory and the ego turned inward. Different tellings vary the list. The point is not the exact ten. The point is that failure has recognisable shapes, and naming them is most of the defence.

Here are the ten I see most in AI-assisted work, which is my own list, not scripture:

Speed over understanding. Accepting output because it arrived fast.
Scope creep. Ten changes of mind cost ten rebuilds. The agent makes indecision cheap and therefore common.
Confidence without evidence. The answer that sounds right and cites nothing.
Skipping the boring defaults. Encryption, signed links, secrets outside the repository. Always "later".
Trusting the demo path. It works on the one route somebody tried.
Letting the tool choose the architecture. It will pick what it has seen most, not what you need.
Not filing the bug you fixed in the same breath. The record disappears.
Mistaking fluency for correctness. Prose quality says nothing about truth.
No test set. If you cannot name how it fails, you do not understand the corpus yet.
Ego. Believing you no longer need the review because the last ten went fine.

Pin that list above the monitor and you have most of the value of the festival without the bamboo.

Ayudha Puja: honour the tool by maintaining it

On the day before, or on the day itself depending on where you are, families clean their tools and put them away for the day. Carpenters their chisels, drivers their vehicles, students their books. In Akola the auto drivers garland their rickshaws. In our office we have started doing the same with the thing we actually depend on: the evaluation set.

A ritual of maintenance sounds quaint until you notice how rarely maintenance happens without one. The 21-case test that decides whether our chatbot ships only gets run because running it after every change became a rule, not a judgement call. There is good evidence that rituals work this way. A 2018 review in Personality and Social Psychology Review by Hobson, Schroeder, Risen, Xygalatas and Inzlicht found that rituals regulate emotion, performance and social connection largely by making an action fixed and repeatable rather than optional. Make the tool check a festival and it happens. Leave it to willpower and it does not.

So, this week: the proxy that serves AI tools to our developers gets its spend limits reviewed, the evaluation sheet gets its cases re-read, the backups get tested by restoring one. Garlands optional.

The line that must not be crossed

Before Ravana arrives, Lakshman draws a line around the hut and tells Sita not to step over it. The whole disaster starts when she does, for a good reason, to feed a visitor. The guardrail was not wrong. The exception felt reasonable.

Every guardrail I have put around an AI tool has been crossed for a reasonable exception. "Just this once, let it push to production." "Just this once, skip the review, it is a one-line change." The rule for us now is that a guardrail can be changed, in writing, with a reason, but never stepped over in the moment. The agent does not get to decide that the visitor is harmless, and neither do I at 11 at night.

Burn the effigy

The part of the festival people travel for is the deletion. The thing you have been building for weeks goes up in flames, in public, on purpose. Nobody keeps the effigy for next year.

Code does not get this treatment. We keep the prototype "in case", the old prompt "for reference", the branch "we might need". This year the review includes a list of things to burn: the chatbot's first Flask prototype, which has been dead since the FastAPI service shipped; three Claude Code configurations nobody has used since the proxy; and two internal tools that a single spreadsheet replaced. Deleting is a skill, and the festival gives it a date.

Begin

Vijayadashami is also the day children are traditionally started on their letters, and the day new ventures are supposed to begin. The festival that burns things also starts things. The two go together: you cannot start the next thing while the effigy of the last one is still standing in the yard.

So the final item on the list is the one thing I have been deferring. For me this year it is the evaluation template, the 21-case sheet in a form other teams can use, which I have talked about for a month and not written. It starts today.

What this comes to

Capability without judgement loses. Name the ways you fail. Maintain the tools on a schedule, not a whim. Draw the line and do not step over it for a reasonable exception. Burn what you have outgrown. Then begin.

That is the festival, and it turns out to be a decent engineering review. Happy Dussehra.

Sources

Hobson, N. M., Schroeder, J., Risen, J. L., Xygalatas, D. and Inzlicht, M. (2018). The psychology of rituals: an integrative review and process-based framework. Personality and Social Psychology Review, 22(3), 260 to 284.
The Ramayana, Yuddha Kanda, for Ravana's learning and his defeat; the Lakshman Rekha episode appears in later tellings such as the Ramcharitmanas rather than in Valmiki.
The 21-case test that decides whether a RAG chatbot ships and A brief goes in, software comes out, for the evaluation and review practices referred to above.

Originally published at kalpakkale.com, where I write Dispatches, field notes on putting AI to work.

A brief goes in, software comes out. Shipping client products without a dev team assigned.

Kalpak Kale — Tue, 20 Oct 2026 04:45:24 +0000

I am not a developer by title. I run operations for an IT services firm. Yet in the last year I have taken several client briefs to deployed, paying, production software without a development team assigned to them. Not prototypes: systems with payments, encryption, QA cycles and clients using them daily.

In one line: An operations manager with an agentic coding tool can take a client brief to a paying production system, but only with a fixed process that puts payments and security first.

The tool is an agentic coding assistant. The tool is not the point. The process is the point, and it is repeatable.

The shape of the process

Every build runs through the same eight stages. "Agent" is the coding assistant; "me" is the human in the loop. The gates are the points where nothing moves until a person has looked.

Brief to decisions. The agent reads the statement of work and lists every question a careful engineer would ask. I get the answers from the client: what happens when payment fails, who sees which fields, what "done" means for the first release. Gate: no code until the page of decisions is signed off.
Slice plan. The product is cut into vertical slices, each one a thing a user can do end to end. One form saved. Then payment. Then the admin view.
Build one slice. The agent builds it against the decisions page, in its own run.
Read the diff. I read every change before it is committed. Not to write it, to understand it. Gate: human review. A diff I cannot explain does not merge.
Test in a real browser, commit, record. The slice is exercised by hand on the running app, committed, and the tracker gets an entry with the measured time.
Security pass before the first real data. The five non-negotiables below are checked, and the deploy script's leak check runs. Gate: security testing. Personal data does not enter the system until this passes.
QA cycle. Test, file, fix, deploy, verify, close. Every bug is filed, even the ones fixed in the same breath, because the record is what the client is paying for.
Deploy and verify, then hand over. After each production deploy: the live URL is smoke-tested, the payment reconciliation job is watched through one real cycle, and the rollback path is tried once. Gate: deployment verification. Then the runbook, admin panel, environment notes and rollback steps go to the client, so their next developer can pick it up from the repository alone.

Stages 3 to 5 repeat per slice. Stage 6 happens once before real data and again whenever a slice touches personal data or money. Stage 8 happens on every release, not only the first.

What a travel platform taught me about payment-first design

The first product I built this way was a travel entry platform: travellers fill in the right government form for their destination and pay for the service. The obvious design records the application, then takes payment. That design produces orphaned applications, angry refunds, and support tickets.

The design that worked is payment-first and idempotent:

Nothing is recorded as an application until the payment provider confirms the money cleared.
Confirmation can arrive three ways: the provider's webhook, the customer returning to the success page, and a cron job that reconciles anything the first two missed. All three paths create the same record once, keyed on the payment reference, so it does not matter which arrives first or twice.
Every state change is logged with a source. When a customer says "I paid and nothing happened", the answer is in the log within a minute.

This pattern is not specific to travel. Any product that takes money before doing work should be built this way, and an agent will not do it unless you ask.

Security defaults I no longer negotiate

Because the platform handled passport-level personal data, a few things became fixed rules for every build since:

Personal data encrypted at rest with a modern authenticated cipher, keys outside the database.
Documents behind short-lived signed links, never a public URL that works forever.
Uploads checked by content, reading the file's magic bytes, not trusting the extension the browser sent.
Secrets in environment files, never in the repository, and a check in the deploy script that refuses to ship if one leaks.
Admin actions behind a real login with rate limiting, not a shared link.

None of these are hard. All of them are skipped by default in a demo, and demos have a way of becoming production.

Forty prototypes, and a colleague who now runs the process

The same process, with the security rules relaxed and mock data instead of real, is how our sales team gets prototypes. In six months we built more than forty for prospects across government, finance, healthcare and consumer businesses: secure messaging for a state agency, a fraud-detection demo, scheme portals, clinical dashboards. A prospect who can click through a working screen behaves very differently from one reading a proposal. Several of those prototypes went into investor pitches and converted into paid projects.

Once the process was stable I wrote it down as a standard operating procedure and trained a colleague to run it. That is the real test of a process: it works when you are not the one running it.

Results so far

Production systems shipped this way: several, each with payments, encryption, a QA record and clients using it daily.
Travel entry platform: six destination workflows live in production, payment-first and idempotent.
Prototypes for sales: more than 40 in six months, across government, finance, healthcare and consumer businesses. Several went into investor pitches and converted into paid projects.
Process handed over: written up as a standard operating procedure and run by a trained colleague without me in the loop.
Security defaults applied on every build since: encryption at rest with keys outside the database, short-lived signed links for documents, uploads checked by content, secrets kept out of the repository with a deploy-time check, admin actions behind a rate-limited login.

What the numbers do not show is the shape of the work: most of the time goes into the decisions page and reading diffs, not into typing code. That is the part the agent cannot do for you.

Where it breaks

When the brief keeps moving. Agents amplify indecision. Ten changes of mind cost ten rebuilds, fast but not free.
When nobody reads the code. I read every diff. Not to write it, to understand it. The day I stop is the day something ships that nobody can explain.
When "it works on the demo path" is accepted as done. The QA cycle exists to make that unacceptable.

If you run operations, product, or a small business, and you have a brief that has been waiting for a free developer for three months: the wait may be over. Just do not skip the page of decisions, the vertical slices, or the payment-first rule.

Originally published at kalpakkale.com, where I write Dispatches, field notes on putting AI to work.

Giving 19 developers AI access through one proxy, and what actually changed

Kalpak Kale — Tue, 20 Oct 2026 04:40:45 +0000

When a company decides its developers should use AI tools, the first instinct is to buy seats. That gets you nineteen people with nineteen different setups, nineteen billing surprises, and no idea whether any of it helped. We did it differently, and most of what follows would work for any team between ten and a hundred people.

In one line: Rolling AI coding tools out to 19 developers worked because of one gateway, a spending policy, an evaluation habit and a set of working rules, not because of the licences.

One gateway, not nineteen API keys

Every model call from the team goes through a self-hosted proxy. Developers get a key to the proxy, not to any vendor. The proxy holds the vendor keys, routes each request to the model configured for that team, logs usage per person and per project, and enforces spending caps.

This one decision solved four problems at once:

Cost visibility. A weekly report shows who used what, on which project, at what cost. Nobody argues about the AI bill any more, because it is itemised.
Model changes without disruption. When a better or cheaper model appears, it is one config change at the proxy. Nobody edits their editor settings.
Off-boarding. Revoke one proxy key and the person's access is gone everywhere.
A place to add rules later. Rate limits, allowed models per project, redaction of secrets in prompts: all of it lives in one place.

We used an open-source proxy. It runs on a small VPS, took an afternoon to set up, and has needed almost no attention since.

Standardise the workflow, not the editor

People are attached to their editors, so we did not fight that battle. We standardised on an agentic command-line tool for the heavy work, and let people keep whatever editor they like for the rest. The workflow that stuck:

Read before writing. The agent reads the relevant part of the codebase and states its plan in a few lines. The developer approves or corrects.
One logical change per run. Not "build the feature", but "add the migration", then "add the endpoint", then "add the test".
Verify against something real. Type checks and tests, then the actual page in a real browser. An agent that only reports "tests pass" is not finished.
Commit with the time it took. Every completed change is logged in the tracker with the commit hash and the measured, AI-assisted time. Clients know the work is AI-assisted, and they see honest numbers.

The fourth rule turned out to be the important one. Measured time makes the productivity claim concrete, and it stops the temptation to bill imaginary hours.

Evaluate tools relentlessly, and expect to say no

Over about six months I ran 38 structured evaluations of AI tools, plugins, models and services. Each one followed the same short template:

What job would this do for us, in one sentence?
One hour of real use on a real task from a live project.
What it did well. What it did badly. What it would cost per month at our size.
Recommendation: adopt, trial for a month, or decline. Written down, with a date.

Most ended in "decline". A few became standard. The value of the exercise was not the wins; it was that every "no" came with a reason, so the same tool was not re-evaluated by a different person three weeks later. The written verdict is the product.

Invest in people, not just seats

Twelve engineers enrolled in a vendor's certification programme for building on its platform. Not because certificates matter, but because a structured course gives a shared vocabulary. When everyone understands tool use, context windows and evaluation the same way, the arguments get shorter and better.

What changed, honestly

Prototypes for sales conversations went from weeks to days. More than forty were built in six months, and several converted into paid work.
Small client projects that used to wait for a free developer now get built by whoever holds the brief, with the agent doing the typing.
The senior developers spend more time reviewing and less time writing boilerplate. They like this.
Junior developers need more supervision, not less. An agent will confidently build the wrong thing at speed. Read-before-write and one-change-per-run exist for this reason.
The bill is real, itemised, and lower than the licensing route would have been, because heavy users and light users are no longer paying the same flat fee.

If you are about to do this

Set up the proxy first, before anyone gets a key. Write the four working rules on one page. Pick one evaluation template and use it every time. Measure time honestly. Then buy the seats, if you still need them. In our case, we mostly did not.

Originally published at kalpakkale.com, where I write Dispatches, field notes on putting AI to work.

The 21-case test that decides whether a RAG chatbot ships

Kalpak Kale — Tue, 20 Oct 2026 04:39:46 +0000

Every retrieval-augmented chatbot demo looks the same: paste in a folder of PDFs, ask a question, get a plausible answer, applause. Then it meets real users, who ask real questions, and the plausible answers start being wrong in ways nobody planned for.

In one line: A regulated financial client's document chatbot shipped only after a weighted 21-case evaluation, nine rounds of structured QA and a daily accuracy report the client ran themselves.

I spent most of this year taking one such chatbot from a demo to a client beta for a wealth-management firm in a regulated market. The model mattered less than I expected. The evaluation mattered more than anything else. This is the evaluation.

Start with the failure modes, not the happy path

Before writing a single test case I listed the ways the bot could be wrong. Three families covered almost everything:

Intent failures. The bot answers a different question from the one asked, usually because the question was short, colloquial, or used the user's internal jargon.
Retrieval failures. The right passage exists in the corpus but the bot never sees it, so it answers from the wrong document, or from nothing.
Semantic drift. The bot retrieves the right passage and still says something the passage does not say: an extra claim, a softened caveat, a number rounded the wrong way.

Each family got seven cases, so 21 in total. Not more. A test set you can run by hand in an afternoon gets run. A test set of 300 gets automated badly or ignored.

The 21 cases at a glance

Family	What it checks	A typical case	Weight
Intent	The bot answers the question that was asked, not a nearby one	A four-word question in the client's internal jargon	1 to 2
Retrieval	The right passage reaches the model	A figure that lives only inside a chart on a slide	1 to 2
Semantic drift	The answer says only what the passage says	A number the passage gives to two decimals, which the bot must not round	2
Scope, inside each family	The bot declines or redirects what it must not answer	"Which of these two funds should my mother buy"	3

Scope cases are spread across the three families rather than kept as a fourth, because a scope failure usually looks like an intent or retrieval failure until you read the answer. Every weight-3 case is a scope case.

Weight the cases by what a wrong answer costs

Each case carries a weight of 1, 2 or 3, set by what a wrong answer would cost the client, not by how hard the case is.

Weight 1: an embarrassing answer. Wrong product name, clumsy citation, a date off by a day.
Weight 2: an answer that would mislead a customer. A rounded number, a softened caveat, a feature the document does not promise.
Weight 3: an answer that would put the client outside the regulator's boundaries. Advice, a recommendation, an opinion on a product's suitability.

A round's score is the weighted pass rate: the sum of the weights of the cases that passed, divided by the sum of all weights. A case passes only if a reviewer reads the answer and agrees it is correct, cited and inside scope; a near miss is a fail.

Two rules sit on top of the score:

Any weight-3 failure blocks the release, whatever the overall score. A 95 percent round with one weight-3 failure does not ship.
A case that passed last round and fails this round is a regression, and a regression blocks the release until it is explained. Prompt "improvements" caused most of ours.

The weight-3 cases were all about scope: questions the bot must decline or redirect, phrased in ways that make declining hard. "Which of these two funds should my mother buy" is a weight-3 case. The correct answer is a polite refusal plus a pointer to a human, and it has to be the correct answer every time.

Nine rounds, and what each one found

We ran the 21 cases as a formal round after every meaningful change, nine rounds in total before beta. A sample of what each round caught:

Retrieval was fine on long questions and poor on short ones. Fix: expand short queries with the user's previous message before embedding.
Slide decks in the corpus were nearly unreadable. Standard OCR on exported PowerPoint pages captured roughly 75 to 80 percent of the text and lost everything inside charts. Fix: render each slide to an image and extract with a vision model, which took accuracy to 95 to 98 percent and finally read the numbers in the charts.
The bot answered questions about a product that had been discontinued, from a document nobody had removed. Fix: a document registry with an "in force" flag, and retrieval filtered by it.
Three weight-3 scope cases failed after a prompt "improvement" made the bot friendlier. Rolled back. Friendliness is not a requirement; staying inside the lines is.
Duplicate questions in the same session got different answers. Fix: cache by normalised question within a session.

Rounds six to nine were smaller: citation formatting, a date-parsing bug, and a case where the correct answer was "the document does not say", which the bot found very hard to produce.

What "ready for beta" meant, written down

The release criteria lived on one page, and every round ended by checking it:

All seven weight-3 cases pass.
Weighted score at or above the pass line we had agreed with the client, for two consecutive rounds.
No regressions against the previous round.
Every document in the corpus is in the registry with an "in force" flag, and retrieval filters on it.
The bot can say "the document does not say" and did so in at least one case this round.
The client's operations team has the daily scoring sheet and has used it on real questions for a week.

The last criterion is the one people skip. The beta did not open until the client could measure it without us.

Hand the scoring to the client

The most useful thing we did was let the client's operations team score the bot daily, in their own spreadsheet, against their own real questions. Every incorrect answer got one of three labels:

Bot issue: the bot had the right material and still got it wrong.
Document issue: the material was missing, outdated, or contradictory.
User understanding: the question was ambiguous or outside scope, and a human would have asked a follow-up.

A typical day in the beta: 79 queries, 87 percent correct, 10 incorrect, of which 7 were bot issues, 2 were user-understanding cases, and 1 was a duplicate. The trend over the first weeks moved between 82 and 91 percent. Two things came out of this that no internal test could have given us.

First, the split told us where to spend effort. When document issues dominate, the fix is content governance, not engineering. When user-understanding dominates, the fix is a clarifying-question behaviour, not a better model.

Second, the client stopped asking "is it accurate?" and started asking "what is today's number and why did it move?". That is the conversation you want.

The stack, briefly

The prototype was a React front end talking to a Flask script with a FAISS index. For beta it became a containerised FastAPI service with Celery workers, Redis, a Milvus vector store and PostgreSQL, behind Traefik, with images built in CI and pulled from a registry. None of that improved accuracy. It made the evaluation repeatable, which is what improved accuracy.

What I would tell anyone starting one of these

Lessons, in the order we learned them:

Failure modes first, prompt second. If you cannot name seven ways each family fails, you do not understand the corpus yet.
Twenty-one cases you run by hand beat three hundred you automate badly. The set has to fit in an afternoon or it stops being run.
Weight by the cost of being wrong, and let one heavy failure block a release.
Read the corpus the way the bot does. Slides that looked fine to us were 75 to 80 percent readable to standard OCR; rendering each slide as an image and reading it with a vision model took that to 95 to 98 percent and recovered the numbers in the charts.
Stale documents are a retrieval failure in disguise. A registry with an "in force" flag fixed a whole class of wrong answers that no prompt could.
Friendliness is not a requirement. One prompt change that made the bot warmer broke three scope cases. We rolled it back.
Hand the scoring to the client with a three-way label: bot issue, document issue, user understanding. The split tells you where to spend the next week.
The stack is plumbing. Moving from a Flask script and a FAISS index to FastAPI, Celery, Redis, Milvus and PostgreSQL behind Traefik did not change accuracy. It made the rounds repeatable, and repeatable rounds changed accuracy.

The bot is still in beta. The number still moves. But every move now has a reason attached to it, and that is the difference between a demo and a product.

Originally published at kalpakkale.com, where I write Dispatches, field notes on putting AI to work.