DEV Community: sagar jain

The Boring Layer Around Every LLM Call: Timeouts and Retries

sagar jain — Fri, 28 Aug 2026 09:00:56 +0000

Every LLM call in production needs a timeout tuned to the expected output, a retry policy with a hard cap and jitter, an idempotency key on any side effect that follows it, and a circuit breaker for the day the provider is degraded. None of that is AI work. All of it decides whether your feature stays up on a bad afternoon, and it's the layer most AI codebases I inherit are missing.

Why do LLM calls need different defaults from a normal HTTP call?

Because the latency distribution has a long tail and every retry costs real money. A typical internal API answers in 50 milliseconds with a p99 near 300. An LLM call answers in about 2 seconds with a p99 of 25 or more, and a long generation can legitimately run past a minute.

Copy your usual 10-second HTTP timeout onto that and you'll cancel healthy requests, retry them, pay twice, and hit your rate limit sooner. Our defaults, which we then tune per call site, start from the shape of the call:

Call shape	Total timeout	Idle between tokens	Retries
Short structured call	30s	15s	2
Long generation	90 to 120s	15s	1
Batch job item	30s	15s	2, under a cost ceiling

Connect timeout is 5 seconds everywhere. The idle timeout is the useful one, because we stream what we can and a stalled stream is a much earlier and cheaper signal than a total timeout.

How many retries, and on what?

Two, at most. Retry on 429s, 5xx responses, connection errors, and timeouts. Use exponential backoff with jitter and honour Retry-After when the provider sends it. Never retry a 400 or a schema-validation failure with the identical request; change something (feed the error back, shorten the input) or give up.

The mistake that taught us the jitter part: a batch job that retried five times with fixed one-second gaps. During a provider incident, a queue of about four thousand items all failed together and all retried together, in lockstep, five times each. We turned a provider slowdown into a self-inflicted rate-limit ban that lasted longer than the original incident. Now every retry policy carries a cost budget too, a per-hour ceiling on retry spend, and the job stops rather than burning through it.

Idempotency: the part people forget

The LLM call itself is usually safe to repeat. What follows it often isn't: send the email, create the ticket, post the comment, charge the card. If your timeout fires after the provider actually finished, and you retry, the model answers twice and the side effect runs twice.

The pattern is borrowed wholesale from payment processing:

Write the intent before you call the model, as a row with a unique key and status pending.
Pass that key through as the idempotency key on the side effect itself.
On any timeout, check whether the work already completed before you retry.
Mark the row done only once the side effect confirms it ran.

Circuit breakers and what "degraded" looks like

A circuit breaker trips after a threshold of failures inside a window, and while it's open the system stops sending hopeful requests. We start with five failures in sixty seconds per model endpoint. Degraded means you fall back on purpose instead of queueing traffic that has nowhere useful to go.

Fall back to a second provider behind the same interface, a smaller model, a cached answer from an earlier run, or an honest degraded mode ("we'll email you the summary in a few minutes"). Then close the breaker gradually with a trickle of test traffic.

I keep coming back to this because the industry conversation is about which agents are real and which are slop, and I think a large share of what separates agents that last from agents that are slop is this unglamorous layer. At Shanti Infosoft it lives in one shared client wrapper that every AI project imports, so nobody has to remember it and nobody can skip it. It's also the first file we write when a software development team hands us an AI feature that works on their laptop. If your provider's p99 has ever ruined an afternoon, a short call is a cheap way to compare notes.

What does your feature do at 2am when the provider's p99 triples for an hour?

Sagar Jain is the technical co-founder of Shanti Infosoft, where 80+ engineers write the boring layer around the interesting part.

The Boring Layer Around Every LLM Call: Timeouts and Retries

sagar jain — Fri, 28 Aug 2026 08:01:56 +0000

Why do LLM calls need different defaults from a normal HTTP call?

Call shape	Total timeout	Idle between tokens	Retries
Short structured call	30s	15s	2
Long generation	90 to 120s	15s	1
Batch job item	30s	15s	2, under a cost ceiling

Connect timeout is 5 seconds everywhere. The idle timeout is the useful one, because we stream what we can and a stalled stream is a much earlier and cheaper signal than a total timeout.

How many retries, and on what?

Idempotency: the part people forget

The pattern is borrowed wholesale from payment processing:

Write the intent before you call the model, as a row with a unique key and status pending.
Pass that key through as the idempotency key on the side effect itself.
On any timeout, check whether the work already completed before you retry.
Mark the row done only once the side effect confirms it ran.

Circuit breakers and what "degraded" looks like

What does your feature do at 2am when the provider's p99 triples for an hour?

Sagar Jain is the technical co-founder of Shanti Infosoft, where 80+ engineers write the boring layer around the interesting part.

Logging for LLM Apps: What to Capture and What to Redact

sagar jain — Wed, 26 Aug 2026 08:00:07 +0000

Log every LLM call as one structured event: prompt version, model id, sampling settings, token counts, time to first token and total latency, tool calls made, validation outcome, retries, and an estimated cost. Redact user content by default and keep a sampled, access-controlled raw store for debugging. Without that, the one question that matters after an incident, what the model actually saw, has no answer.

What goes in the event?

One event per LLM call, written when the call completes, holding roughly ten fields about the call and nothing about its content. It has to answer two questions later: how the call behaved, and what it cost, without storing what the user typed. Here's the field list we start from, boring on purpose.

A trace id that ties together every call made for one user action.
The prompt template version and a hash of the rendered prompt.
Model id, temperature, max tokens, and any provider-specific flags.
Input and output token counts, taken from the provider response, never estimated.
Time to first token and total duration.
Every tool call: name, arguments hash, duration, success or failure.
Validation result against the output schema, plus the retry count.
The route taken, if you use model routing, and the feature-flag state.
A cost estimate computed from the token counts.
A hashed user or tenant id.

Notice what isn't in there: the prompt, the response, the user's message. That omission is the whole design.

Why redact by default?

Because prompts are where the sensitive data ends up. Contract text, medical notes, salaries, customer messages: whatever your product touches flows into the prompt in plain form. If your log pipeline stores prompts raw, your logging system has quietly become your most sensitive database, usually with the weakest access controls.

The moment that made this personal for us: an engineer debugging a bad response pasted a full log line into a Slack channel to ask for help. The line contained the entire prompt, and the prompt contained a customer's message with their phone number and account details. Nobody meant harm. The system had made it the path of least resistance.

So we redact at ingestion, and we split the storage in two.

What you store	Default structured event	Sampled raw store
Token counts, latency, cost	100 percent of calls	Not needed
Prompt version and hash	Yes	Yes
Full prompt and response text	Never	1 to 5 percent, plus all failures
Retention	Normal log retention	7 to 30 days, TTL enforced
Who can read it	Anyone who reads logs	Row-level access, reviewed

A regex pass catches the obvious patterns (emails, phone numbers, card-like sequences, national id formats), a small classifier catches names and free-text PII, and the redacted event stays the default view.

How do you keep logs useful as they grow?

Sample raw content at one to five percent for healthy traffic, and keep one hundred percent of failures: validation errors, timeouts, tool errors, user thumbs-down. The failures are what you'll be reading. Emit spans through OpenTelemetry so an LLM call sits inside the same trace as the database query before it.

Then build four dashboards and resist building more: latency (p50 and p95) by prompt version, validation-failure rate by model, cost per successful task, and escalation or fallback rate. Every incident I've worked on an AI feature was visible in one of those four before a human noticed.

What breaks when you don't do this?

Model upgrades break first. A provider ships a new version, your prompt behaves slightly differently, and with no logged baseline there is nothing to compare against and no way to reproduce the old behaviour. The team blames the model, the feature loses trust, and nobody can prove what changed.

Teams whose AI projects survive tend to have this instrumentation in place before launch, which is one of the habits shared by AI projects that don't get cancelled. At Shanti Infosoft this logging schema ships in the first sprint of every AI build, before the prompt is even good, because you can't improve what you didn't record. It's the same schema we drop into an AI development engagement that arrives with a working feature and no telemetry, which is most of them.

If a customer told you their answer yesterday at 3pm was wrong, how long would it take you to see exactly what the model saw?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 company, and has spent more incident calls than he'd like reading LLM logs that weren't there.

Designing the Human Approval Step So It Isn't Rubber-Stamped

sagar jain — Mon, 24 Aug 2026 08:00:33 +0000

A human-in-the-loop step only works if the reviewer has a real decision to make and can see the evidence needed to make it. It stops working the moment the queue can be cleared by approving everything. Most approval steps I review fail on both counts, and the audit log then faithfully records humans "approving" things nobody read. If you can't prove your reviewer would catch a bad item, the step is decoration.

Why does approval fatigue set in so fast?

Volume and sameness. An agent drafts two hundred items a day and ninety-five percent of them are fine, so the reviewer learns within a week that approving is the fast path. Four seconds per item. Batching before month end. The step still exists on the architecture diagram, and it still fails.

We learned this on ourselves. When we tried running parts of our own operations through agent "departments", one internal agent prepared vendor follow-up emails for a person to approve. After a week the approval rate was 100 percent. So we seeded five deliberately wrong drafts (a wrong amount, a wrong recipient, a duplicate of one already sent, and one referencing a contract that didn't exist). All five went out approved. The reviewer was diligent by nature; the interface had trained them that nothing needed reading.

Seeded failures are now the first thing I ask any team to run. If the catch rate on planted errors is low, the human is near the loop rather than in it.

Queue design	What the reviewer faces	What we saw happen
Everything routed to a person	200 items daily, most of them fine	Four seconds each, 100 percent approved
Low-risk auto-approved and sampled	Roughly 20 hard items with diffs	Items get read properly
Seeded failures mixed in	Same queue, planted errors inside	Catch rate becomes a real number

What should the approver actually see?

The approver should see three things: the diff of what changed, the raw evidence the agent used, and one line explaining why this item needed a human at all. Anything else on the screen is cost. A reviewer who can check a claim in twenty seconds will check it.

The diff instead of the document. If the agent changed three fields on a record, show those fields with old and new values.
The evidence: the retrieved records, the tool results, the policy snippet it matched, the customer's original message. The evidence itself, never a summary of it.
A risk grade, so the reviewer spends attention where it matters.

Then stop showing them the easy stuff. Auto-approve the low-risk slice with random sampling for QA, and route only the top-risk slice to a person. A reviewer looking at twenty hard items reads them. A reviewer looking at two hundred mostly-fine items reads none.

How do you make approval a real decision?

Approval becomes a real decision when approving costs the reviewer something small: a reason code, a second button, a cap on volume, a randomised order. Each mechanic breaks the reflex that turns a queue into a rhythm. The number that tells you whether it worked is the disagreement rate.

Require a reason on approval for high-risk items. A dropdown with four options is enough; the act of choosing breaks the reflex.
Separate "looks fine" from "verified." Two buttons, with different downstream permissions.
Cap approvals per hour, and randomise order so the reviewer can't pattern-match on batches.
Measure the disagreement rate. If your reviewer rejects or edits zero percent of items over a month, the step is either unnecessary or broken. Either way, act on it.
Keep running seeded failures, quietly, and put the catch rate on the dashboard next to the agent's own accuracy.

When should you take the human out?

Take the human out when sampling shows the low-risk slice has been clean for a sustained period, and only for reversible actions. Keep a person on anything irreversible: payments, deletions, outbound messages to customers, anything with legal weight. The reviewer then moves up to the harder slice.

That split (automate the reversible, gate the irreversible) is how we design agent workflows at Shanti Infosoft, and it's the first thing we redraw during an AI consulting engagement on a stalled agent rollout. It keeps the approval queue short enough that people read it, which is the only property that matters. If you want to pressure-test your own queue design, a call and a whiteboard usually gets there.

If you planted five wrong items in your approval queue tomorrow, how many would get caught?

Sagar Jain is the technical co-founder of Shanti Infosoft, which has built software for 700+ companies and now designs the approval layers that sit around their agents.

Structured Output Is the Most Underrated AI Reliability Fix

sagar jain — Sat, 22 Aug 2026 08:00:15 +0000

If your AI feature does anything besides show text to a human, make the model return a schema-validated structure and treat a validation failure as a normal, counted error path. This one change removes a whole family of production incidents: parsing failures, invented field names, the helpful preamble ("Sure! Here's the JSON:"), and the silent default that kicks in when a regex finds nothing.

Why does free text fail quietly?

Because the failure doesn't look like a failure. The pipeline gets a string, the regex finds nothing or finds the wrong thing, a variable ends up null, and the null flows into a default branch that was written for a different reason. No exception and no alert, just a log line that looks like every other log line.

The demo never shows this. In a demo the operator reads the output, so a stray sentence is charming. In production nobody reads the output; a machine does, and that is the exact reason a working demo is the problem: the same output that impressed a room breaks a parser.

A concrete one from our side. An internal routing step classified incoming requests into one of eight buckets. About three percent of responses came back with a trailing remark after the label ("Category: billing. Note that this could also be..."), our parser took the whole line as the label, matched nothing, and the request landed in the catch-all bucket. Catch-all was a legitimate destination, so nothing complained. It ran that way for weeks.

What the model does	Free text plus a regex	Schema-validated output
Adds a polite preamble	Parse misses, field goes null	Rejected, retried with the error
Invents a ninth category	Lands in the catch-all bucket	Enum violation, counted
Drops a required field	A default fires downstream	Validation error you can alert on
Behaves differently after an upgrade	Nobody notices for weeks	Failure rate moves the same day

What does "structured" mean in practice?

Structured output means the model returns data conforming to a schema you defined in code, and your application validates it before using it. In practice that is a Pydantic, Zod or JSON Schema definition, the provider's structured-output mode where one exists, and a validation step that runs on every single call.

The details that matter:

Use enums for closed sets. Eight categories means an enum with eight values plus one explicit unable_to_determine. Never let the model invent a ninth.
Keep schemas shallow. Deep nesting invites partial objects.
If you want reasoning, give it a field, and put that field before the answer fields. Order affects quality because the model generates top to bottom.
Mark as optional only what is genuinely optional. "Optional because the model sometimes forgets it" is a bug you're hiding from yourself.

The validation step is where the reliability comes from. The model can still be wrong, but now it's wrong inside a shape you can inspect and count.

How should validation failures behave?

A validation failure should behave like any other error in your system: retried once with the error fed back, then failed over to a deterministic default or a human queue, counted in a metric, and alerted on when the rate crosses a threshold. What it must never do is coerce silently.

The sequence we implement every time:

Retry once, feeding the validation message back to the model ("field priority must be one of low, medium, high").
On a second failure, fall back to a deterministic default or route the item to a human queue.
Increment a counter tagged with the model id and the prompt version.
Alert when the failure rate crosses something like one percent.

That last threshold is your earliest signal that a model update changed behaviour, which is worth more than the retry itself.

What about features where text is the product?

Wrap them anyway. Return {answer, citations[], refusal_reason} instead of a bare string, and the output becomes testable: citations exist when they should, refusals happen for the right reasons, the length stays inside a bound, and the same input returns the same shape twice. Every field is addressable, which makes logging and evals easier later.

We do this on every AI build at Shanti Infosoft, including the ones where the client insists the output is "just text," because six months in someone always wants to route on it or filter it. It's a standing rule on the AI development work we take on for exactly that reason. A schema on day one costs an hour. Retrofitting one after the regexes have spread through the codebase costs a sprint.

Which of your AI outputs is still being parsed with a regex, and what happens on the day the model adds one polite sentence?

Sagar Jain is the technical co-founder of Shanti Infosoft, where 80+ engineers build AI features that other people's systems have to consume without breaking.

Model Routing in Production: Cheap First, Escalate on Doubt

sagar jain — Thu, 20 Aug 2026 08:00:28 +0000

Route most requests to the cheapest model that passes your evals, and send a request to the expensive model only when a cheap, checkable signal says the first answer is doubtful. That one design decision usually cuts the blended inference bill by more than half without a measurable quality drop. Routing "by vibes" (long prompt goes to the big model, short prompt goes to the small one) gives you the worst of both.

Why does everything end up on the flagship model?

Because the demo was built on it and nobody went back. During the pilot the team was optimizing for "does it work," the flagship worked, and the model id got hard-coded. Six months later the bill is a line item the CFO asks about, and switching feels risky because there's no eval set to prove the cheaper model is fine.

The math is worth writing down. Say the small model costs a tenth of the flagship per token. Blended cost is then whatever share the cheap tier can actually carry:

Share the cheap tier handles	Blended cost vs flagship-only	Escalation rate
0 percent	1.00	none
70 percent	0.37	30 percent
85 percent	0.235	15 percent

Illustrative, at a tenth the price per token and before caching. Those are the numbers that make an automation case hold up once you look at the real bill for running it instead of the build quote.

What signal decides escalation?

An escalation signal is any cheap, checkable fact about the first answer that predicts it is wrong: a failed schema validation, a disagreeing checker, a weak retrieval score, or a task class known to be hard. The one signal I don't trust is the model rating its own confidence.

It's poorly calibrated and it drifts between versions. Signals that have held up for us:

Structured-output validation failing (schema errors, an invalid enum, a missing field).
A cheap checker disagreeing: a rules pass, or a second small-model call asked a narrow yes/no question about the first answer.
Retrieval quality below a threshold, when the task depends on retrieved context.
An explicit task class from a lightweight classifier: "refund policy question" stays cheap, "multi-step account change" escalates immediately.
The user retrying or giving a thumbs-down, which triggers a re-run on the bigger model.

Each of those is checkable in code and testable on its own, and each one gets logged with the request so you can see later why a call escalated.

Where does routing go wrong?

Routing goes wrong when the rule measures something other than difficulty. Input length and user tier both correlate with cost, and neither says how hard a request is. Aggregate quality metrics then look fine while one specific slice quietly degrades, and nobody spots it until somebody reads the complaints.

Our own version of that mistake, on an internal support tool: we routed by input length. Long tickets went to the flagship, short ones went cheap. Then we read the complaints and saw they clustered on short tickets, because short tickets were the ambiguous ones ("still not working"), and the cheap model was confidently guessing. Routing by task class plus validation outcome fixed it in a day.

Order matters as much as the rule. The path a request takes now:

Check the cache, exact match first, then a normalized version of the input.
Classify the task with a lightweight classifier and pick a tier.
Call the cheap model and validate the output against its schema.
Escalate to the flagship only when one of the signals above fires, and log which one.
Record the route, the outcome and the cost on the request.

There is no cheaper model than the one you don't call, which is why the cache sits at step one.

How do you keep the router honest?

Four habits keep a router from rotting: an eval set per route, the route decision logged on every request, an escalation rate somebody watches, and a re-run of the whole comparison whenever a provider ships a new model. Skip them and the cheap tier drifts without anyone noticing.

The escalation rate is the one people skip. If 60 percent of requests escalate, your cheap tier is theatre and you're paying for two calls per request. Re-run the comparison on every provider release, because this year's cheap tier often beats last year's flagship and your thresholds were tuned for the old pair.

We ship a lot of AI features at Shanti Infosoft for mid-sized companies where the monthly inference bill matters as much as the accuracy number, and routing is the first lever we reach for. It's also the first thing we look at when auditing an AI integration somebody else built. Rarely glamorous, but it tends to decide whether a feature survives the budget review. If you want a second opinion on your own tier split, a short call is usually enough to find the obvious win.

What percentage of your requests truly need the most expensive model you're paying for?

Sagar Jain, technical co-founder at Shanti Infosoft, a CMMI Level 5 firm, spends most of his week on the cost side of AI features rather than the demo side.

Treat Prompts Like Code: Versioning and Rollbacks in Production

sagar jain — Tue, 18 Aug 2026 09:00:18 +0000

A prompt that runs in production is code. It belongs in version control, it gets a version id that shows up in your logs, it passes a regression suite before it ships, and it can be rolled back in a minute without a deploy. I still walk into teams where the prompt lives in a database column that someone edits from an admin panel on a Friday evening, and then everyone spends the following week wondering why the output "feels different."

Why does a prompt edit break things weeks later?

The short answer is that prompt edits look harmless and their effects are statistical. A wording change does not break a build or throw an exception. It shifts the distribution of outputs by a few percent, and that shift only becomes visible weeks later, in a support complaint nobody traces back to the edit.

On one project we changed a single instruction from "summarize the ticket" to "briefly summarize the ticket." Reasonable edit. Shorter output, lower cost. What we didn't notice was that the JSON field holding action items started coming back empty in roughly one out of every twenty responses, because the model read "briefly" as permission to drop the list. Downstream, an empty list was valid, so nothing errored. We found out from a support lead ten days later, and it took a further afternoon to connect the complaint to the edit.

That afternoon is the real cost, and it's decided entirely by where the prompt lives.

After a bad week	Prompt in an admin panel	Prompt in the repo
What changed?	Nobody can say	A diff
Who changed it, and when?	Maybe an `updated_at` column	Commit author and timestamp
Was it reviewed?	No	A PR approval
Did it pass anything first?	No	The suite that ran in CI
Roll it back	Retype it from memory	Check out the previous version

What does "prompt as code" look like in practice?

Prompt-as-code means the prompt text, the model id, and the sampling settings live in the repo as files, ship through a pull request, and carry a version id that appears in every log line. Rolling back is a checkout or a config flip, never a database edit.

Our setup is deliberately unexciting.

Prompts are files in the repo, one folder per use case, with a template and explicit variables (prompts/ticket_triage/system.md, prompts/ticket_triage/user.md).
The model id and the sampling settings sit next to the prompt in a small config file, so a "prompt version" means the whole calling contract, and a model swap is a visible diff.
Every LLM call logs a content hash of the rendered template plus the config version. If a log line says triage@v14, I can check out that exact text.
Prompt changes ship through the same pipeline as code: branch, PR, CI, deploy.

None of this needs a special tool. A prompt-management SaaS can be pleasant for non-engineers, but if it doesn't give you diffs, review, a version id in the logs, and a rollback, it's a fancier admin panel.

How do you regression-test a prompt?

Build a small suite of thirty to fifty real inputs with known-good outputs, and assert on structure rather than exact wording: the JSON parses, required fields exist, enums are valid, length stays inside bounds, forbidden phrases are absent. Run it in CI whenever a file under prompts/ changes.

For the fuzzy parts (is this summary faithful to the ticket) a judge model with a rubric works, as long as the judge's own prompt is versioned too. Fifty cases on a mid-tier model costs cents. Block the merge on failure. That is the entire mechanism that would have caught our "briefly" incident before it reached a customer.

Who gets to edit prompts?

Anyone with the domain knowledge, through the same gate as code. Product managers and domain experts should be able to change prompts, and they should do it through a pull request or a staged environment that runs the regression suite. The reviewer and the green suite stay mandatory either way.

This is where I see engineering leadership matter most right now: as more of the codebase gets written by tools, the job shifts toward governing what ships rather than typing it, and prompts are the most under-governed part of the AI systems I get shown. Versioning them is the first thing we put in place on generative AI builds we inherit mid-flight, and at Shanti Infosoft we treat a prompt PR exactly like a code PR: a reviewer, a green suite, an owner, and a rollback path, no matter who wrote the words.

Where do your production prompts live right now, and could you tell me what changed in them last month?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 team of 80+ engineers who put AI systems into production and then have to maintain them.

Treat Prompts Like Code: Versioning and Rollbacks in Production

sagar jain — Tue, 18 Aug 2026 08:01:18 +0000

Why does a prompt edit break things weeks later?

That afternoon is the real cost, and it's decided entirely by where the prompt lives.

After a bad week	Prompt in an admin panel	Prompt in the repo
What changed?	Nobody can say	A diff
Who changed it, and when?	Maybe an `updated_at` column	Commit author and timestamp
Was it reviewed?	No	A PR approval
Did it pass anything first?	No	The suite that ran in CI
Roll it back	Retype it from memory	Check out the previous version

What does "prompt as code" look like in practice?

Our setup is deliberately unexciting.

Prompts are files in the repo, one folder per use case, with a template and explicit variables (prompts/ticket_triage/system.md, prompts/ticket_triage/user.md).
The model id and the sampling settings sit next to the prompt in a small config file, so a "prompt version" means the whole calling contract, and a model swap is a visible diff.
Every LLM call logs a content hash of the rendered template plus the config version. If a log line says triage@v14, I can check out that exact text.
Prompt changes ship through the same pipeline as code: branch, PR, CI, deploy.

How do you regression-test a prompt?

Who gets to edit prompts?

Where do your production prompts live right now, and could you tell me what changed in them last month?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 team of 80+ engineers who put AI systems into production and then have to maintain them.

What We Deliberately Don't Automate

sagar jain — Tue, 11 Aug 2026 08:00:21 +0000

I build automation for a living, so people are sometimes surprised by how much my team does by hand on purpose. We deploy with a human pressing the button. We review certain data changes manually. We answer some support tickets one at a time.

None of this is because we can't automate it. We ran the math, and manual won.

Automation has a fixed cost you pay forever

An automated process isn't free once it's written. It's a thing you now own. It breaks when the API it calls changes. It does the wrong thing confidently when an assumption shifts under it. It needs monitoring, and the monitoring needs maintaining too.

For a task you run a hundred times a day, that ongoing cost is obviously worth it. For a task you do twice a month, you can easily spend more engineering time maintaining the automation than you'd ever spend just doing the task. The break-even point is real, and a lot of automation sits on the wrong side of it.

I got this wrong early on. I automated our monthly report generation, a job that took maybe twenty minutes by hand. The script worked for four months, then a data source changed its format and it quietly produced a wrong number that went into a client deck. Fixing it, and rebuilding trust in that report, cost far more than years of doing it by hand would have.

The line I watch for: judgment

The clearest signal that something shouldn't be fully automated is when it needs judgment that changes case by case.

Deploying to production is a good example. The mechanical steps are easy to script, and we do script them. But deciding whether to deploy right now, with these changes, given whatever else is going on, is judgment. So the pipeline is automated and the trigger is a person. That person isn't doing toil. They're doing the one part that's actually hard.

There's a softer version of this in support. Canned responses and routing, fine, automate all of it. But the ticket where a customer is upset and the situation is ambiguous is exactly where a templated reply does the most damage. We send that to a human on purpose, because the cost of getting it wrong dwarfs the minutes saved.

At Shanti Infosoft we draw the line at exactly that spot. Automate the mechanical steps, keep a human on the decision. The failures I've seen from over-automation almost always trace back to scripting away a judgment call that quietly needed a brain.

Manual first, automate what hurts

Our default for a new process is to do it by hand a few times before writing a single line of automation. This feels slow, and it is, but it buys two things.

It shows you what the process actually is, including the messy parts you'd never predict from a whiteboard. And it tells you whether the task is even stable enough to automate, or whether it's still changing every time you touch it. Automating a moving target just bakes today's version of the mess into code.

If a manual task starts to hurt, if it's frequent and repetitive and the steps have stopped changing, that's when automation earns its place. Pain plus stability, not novelty, is the trigger. We automate the things that have proven they deserve it.

We put actual numbers on that trade-off in the automation bill nobody quotes you on.

What's something your team automated that you'd honestly be better off doing by hand?

Build, Buy, or Call an API: How We Actually Decide

sagar jain — Sun, 09 Aug 2026 08:00:20 +0000

Clients ask me why we don't just build our own model. It's a fair question, and most of the time the honest answer is that building our own would be the slowest and priciest route to a result slightly worse than an API hands us on day one.

Engineers like building, and 'we made our own' reads well in a pitch. So I put every AI capability through the same three gates before we commit: whether it's our actual edge, how fast it's changing, and what it costs at our real volume.

Call the API when the capability is a commodity

Text generation, transcription, translation, OCR, embeddings, general chat. These are commodities now. A hosted API from OpenAI, Anthropic, or Google gives you a better result on day one than a small team builds in a quarter, and someone else pays to keep it current.

We reach for an API when the capability isn't our differentiator and the vendor improves faster than we could. The trade you accept is a per-call price, a dependency on their uptime, and data leaving your walls. For most features that trade is worth it. Both risks are manageable: put the provider behind one interface so you can switch, and read the data-handling terms before anything sensitive goes over the wire.

Buy when it's already someone's whole product

Some things are a company, not a feature. A mature vector database. A full observability stack for LLM apps. A content-moderation service. When a vendor's entire business is the thing you need, and it's a solved, unexciting problem, buying beats shipping a worse version and maintaining it forever.

The question here is whether owning it would be a distraction. If maintaining the thing pulls the team off the product the client is actually paying for, we buy.

Build when it's the reason they hired you

Building earns its cost in a narrow band. When the capability is your real differentiator, when your proprietary data changes the outcome, or when the per-call math at your volume finally beats a subscription, owning it makes sense.

Even then, building rarely means training a model from scratch. It usually means orchestration, our own data pipeline, retrieval tuned to the domain, and evaluation nobody else has. The base model still comes from an API. The edge lives in the plumbing around it.

Design the exit before you commit

Whatever we choose, we design the switch first. The provider sits behind an interface. Prompts and data stay ours, in a format we control. Costs get an alert, because the math that made an API cheap at 10,000 calls a day can flip at a million.

A build-or-buy call isn't permanent. Volume grows, prices fall, a vendor rewrites its terms, and last year's right answer becomes this year's migration. Choosing well once matters less than being able to change your mind cheaply.

On AI work at Shanti Infosoft, most capabilities end up as an API call wrapped in a thin, switchable layer, and only the genuine differentiator gets built in-house. If you want to see how we draw that line on real projects, it's at https://shantiinfosoft.com.

We drew this exact line in how the build-vs-buy decision moved.

Which capability did you build that you'd call an API for if you were starting over today?

The Second Version Is Always Smaller

sagar jain — Thu, 06 Aug 2026 08:00:39 +0000

Every time I've rebuilt a system I understood well, the second version came out smaller than the first. Less code, fewer files, fewer special cases. This surprised me the first few times. Now I expect it, and when a rewrite comes out bigger, I treat it as a warning sign.

The clearest case I remember was a billing module. The first version ran around 2,000 lines, full of branches for edge cases we thought we'd hit. The rebuild, done two years later by someone who'd supported the original in production, came in near 700 lines and handled more real scenarios. Nothing clever happened. We just finally knew which branches were fiction.

The first version is mostly guessing

The first time you build something, you don't fully know what it needs to do. You know what you were told, and you know what you can imagine, but you don't know what's real yet.

So you hedge. You add a config flag because someone might want to turn this off. You add an abstraction because there might be a second case someday. You handle an input that can't actually happen, because you aren't certain it can't. Every one of these is a reasonable bet made under uncertainty.

A year later, most of those bets have resolved. The config flag was never flipped. The second case never came. The impossible input stayed impossible. All that code is still there, still maintained, still confusing the next person, and all of it is compensation for uncertainty that no longer exists.

Understanding is what shrinks it

The second version is smaller because you're building with answers instead of guesses. You know which cases are real. You know which flags mattered. You know the actual shape of the data because you watched it move through production for a year.

This is why I'm careful about who should rewrite something. The person who understands the current system deeply will make it smaller. The person who just finds it ugly will make it bigger, because they'll re-add every hedge from scratch under the same uncertainty the first author had.

There's a trap worth naming. Ugliness and complexity look the same from the outside. A senior engineer learns to ask whether an ugly piece of code is bad, or ugly because the problem is genuinely messy and the code is being honest about it. Those two need opposite treatments.

Before anyone rebuilds a component at Shanti Infosoft, the first question is whether we understand why the current one is shaped the way it is. If we can't explain the weird parts, we aren't ready to replace them. The weird parts are usually load-bearing.

Smaller is the evidence, not the goal

I want to be careful here. Small code isn't automatically good code. You can compress something until nobody can read it, and that's worse than the verbose version. Shrinking isn't the target.

Smaller is the symptom of understanding. When the second version comes out dramatically simpler, it usually means we finally understood the problem. When it comes out bigger and more clever, it usually means we didn't, and we hid that from ourselves behind abstraction.

So I don't chase line count. I chase understanding, and I read the line count as a signal for whether we got there.

That's really a governance point, and we made the fuller argument in why writing code was never the bottleneck.

When you last rebuilt something, did it come out smaller, and if it didn't, are you sure you understood the original?

The AI Feature Is Cheap to Build and Expensive to Run

sagar jain — Tue, 04 Aug 2026 09:00:18 +0000

The quote everyone remembers is the build cost. The number that decides whether an AI feature survives is the monthly one, and it tends to show up in month two, right when the trial credits run dry and real traffic arrives.

I budget AI features the way I'd budget a delivery van. Buying it happens once. Fuel, insurance, and the driver run forever. Here's where the fuel actually hides.

Where the money goes

Tokens, including the ones you forget. Everyone counts the user's question. Fewer people count the system prompt, the retrieved context, the few-shot examples, and the model's own output, all billed on every call. A feature carrying a 3,000-token context that looked tiny in testing can run 10x the estimate once every request drags that prompt along.

Retries and retrieval. A retry on failure doubles the cost of that call. A RAG feature also pays to embed every document, store the vectors, and run a similarity search per query. The model bill is one line on a longer receipt.

The machinery around the model. Vector database hosting. Logging and observability, which for AI features is not optional. Egress. The cache you'll add later to stop paying twice for the same answer.

Humans in the loop. If a person reviews flagged outputs, that review time is a running cost of the feature and belongs in the budget, even though no vendor ever invoices you for it.

How we actually budget it

We estimate a cost per action before a line of the feature exists. Average tokens in, average tokens out, times the model's price, times expected volume. It's back-of-envelope, and it usually lands close, because the inputs are knowable.

Then we pick the cheapest model that passes evaluation, not the highest one on the leaderboard. A smaller model that's good enough on your real task can cut the bill 5 to 10x. We send the easy 80% of requests to the cheap model and escalate only the hard ones. Caching repeat queries shaves off another slice.

The last step is a hard spend cap wired in before launch. Per user, per day, per feature. A runaway loop or a scraper pounding your endpoint should trip a limit and page a human, not keep billing until the card declines.

Give the client the real number

When we scope AI work, the client gets both numbers. Build once, and run monthly at your expected volume, with the assumptions written down beside them. A client who signed off on a $600-a-month running cost stays calm when the bill reads $600. A client shown only the build price feels ambushed, and they're right to.

That conversation is unglamorous, and it's the one that stops a project from souring six weeks in. The team at Shanti Infosoft treats the running-cost estimate as part of the quote, not a thing we discover together later, and you can see how we scope it at https://shantiinfosoft.com.

We break the running-cost math down further in the automation trade-off nobody quotes you on.

What did an AI feature actually cost you to run each month, and how far off was the first estimate?