DEV Community: sagar jain

Do You Need a Model Abstraction Layer? Usually a Thin One

sagar jain — Tue, 15 Sep 2026 08:00:21 +0000

Yes, build one, and keep it thin. One internal function that takes a typed request, handles retries, timeouts, logging, and routing, and returns a typed response, with the provider SDK hidden behind an adapter. Don't build a general framework, and don't adopt a heavy one blindly. The leaky parts (tool-call formats, streaming shapes, structured-output modes, error codes) will leak through anything, and a thick abstraction means you spend more time fighting it than you would have spent switching providers.

What belongs in the thin layer

Roughly three hundred lines you own outright: a typed request and response, two or three provider adapters, model aliases in config, the reliability layer, logging and cost accounting in one place, and flags for model and prompt version. Every item on that list is work you'd do with or without an abstraction.

A typed request (messages, tools, output schema, an alias for the model tier) and a typed response (content, tool calls, token usage, finish reason).
Two or three provider adapters, each translating that request into the vendor's SDK call and back.
Model aliases in config: fast, smart, long-context, mapping to actual model ids per environment. Application code never mentions a vendor's model name.
Timeouts, bounded retries with jitter, a circuit breaker, and an idle-stream watchdog.
Feature flags for model and prompt version, so rollback is a config change.

What to leave out

Prompt DSLs. Chain and graph orchestrators baked into the core. "Memory" modules with opinions about your database. Vendor-specific "agents" that hide their loop from you. Prompts stay as versioned files, and orchestration stays plain code on a queue you already run, with state in the database you already have.

When a calling convention changes, and it changes a few times a year, a thin layer means editing one adapter. A framework means waiting for a release and then reading its changelog with your fingers crossed.

	Thin layer you own	Heavy framework
Size	~300 lines, all yours	Thousands of lines, none yours
Breaking upstream change	Edit one adapter	Wait for a release, then adapt
Debugging a bad response	Your own stack, top to bottom	Nine frames of library code first
Cost to start	About two days	An afternoon
Cost to leave	Nothing to leave	A week, in our case

I keep seeing teams reach for the framework because it feels like buying instead of building. In practice this is one of the places where the build-vs-buy line has moved toward building the small thing, precisely because the small thing is small.

When a heavier framework earns its place

Four cases earn it: prototyping, where speed of trying ideas beats everything; teams that genuinely need many providers and many modalities on day one; teams without the capacity to maintain even three hundred lines; and regulated environments where a vendor's audited framework saves a compliance conversation. All four are legitimate, and all four still need guardrails.

Pin the version. Wrap it behind your own interface so application code never imports it directly. Write one integration test per callback you depend on. Keep an ejection plan.

Our own lesson: we adopted a popular orchestration framework early on one product. A minor version bump changed the semantics of a streaming callback, and partial responses started being treated as complete. It took most of a day to trace, because the stack was nine frames deep in library code before it reached anything we wrote. Ejecting took a week. Writing our own layer at the start would have taken two days, and we'd have understood every frame of every stack trace since.

The switching test

The whole point of the layer fits into one test you can run in an afternoon, without touching application code. If it passes, your abstraction is doing its job. If switching means a branch, a refactor, a week of QA, and a nervous release, the abstraction has quietly become the thing you're locked into.

Change the model behind the smart alias in config, in one file.
Rerun the eval set against old and new.
Compare pass rate, latency, and cost per call side by side.
Decide, then roll forward or back with the same config change.

At Shanti Infosoft that thin client is shared across projects, and it has quietly outlived four generations of "must-have" frameworks, which tells you most of what I think about the trade. It's the default we start from on custom software development work where a model sits somewhere in the stack.

If your provider doubled prices tomorrow, how many files would you have to touch to move?

Sagar Jain is the technical co-founder of Shanti Infosoft; his team of 80+ engineers works to a CMMI Level 5 standard and typically has a first working build in front of a client in about two weeks.

Golden Datasets Rot: Keeping Your Eval Set Honest Over Time

sagar jain — Sun, 13 Sep 2026 08:00:42 +0000

An eval set is a snapshot of what your product needed the day you built it, and products, users, models, and even the golden answers themselves drift after that. So treat the set as a living asset: version it, feed it with real production failures every week, retire cases that no longer represent real traffic, and re-check the labels on a schedule. A 100 percent pass rate on a set nobody has touched in six months is a warning sign, whatever the dashboard says.

How does an eval set rot?

Quietly, and from several directions at once. Product policy changes while the golden answers still encode the old rule. A new user segment arrives asking unfamiliar questions. Eval cases leak into the prompt as few-shot examples. Some labels were wrong on day one. And the set gets easy, because every failure was fixed and no hard cases replaced them.

A refund window moving from 14 days to 30 is enough to turn a dozen gold answers into confidently wrong ones. The dashboard rarely tells you which mode you're in:

Rot mode	What the dashboard shows	What is actually happening
Policy change	Pass rate holds steady	Gold answers encode a rule the product dropped
New user segment	Pass rate holds steady	The set no longer resembles real traffic
Few-shot leakage	Pass rate climbs	The prompt is graded on cases it memorized
Labels wrong on day one	Pass rate holds steady	You're optimizing toward somebody's mistake
Set gone too easy	Pass rate near 100%	Every hard case was fixed, none replaced

We hit the leakage one ourselves. Pass rate sat at 98 percent for two months while user complaints climbed. Someone had lifted six of the "hard" eval cases into the system prompt as examples of good output. The eval was grading the prompt on its own homework.

Feeding it from production

Every week, pull four buckets out of the logs and label twenty to thirty cases from them. Every case carries a source tag and a date. Ambiguous ones get two labellers, and disagreements get discussed rather than averaged. That's the whole maintenance habit, and it takes about an hour once somebody owns it.

Schema-validation failures. Anything the parser rejected outright.
Thumbs-downs. The user has already told you; believe them.
Escalations, either to the bigger model or to a human.
A plain random sample, so the set doesn't become a museum of failures.

Keep a separate "hard set" of the cases that failed at least once in production. It stays small and it runs on every prompt change, which is how it catches the regressions the big set is too diluted to notice.

Retiring and re-labelling

Once a quarter, re-label a random 10 percent of the set blind, without showing the stored gold answer. If agreement with the stored labels drops below roughly 90 percent, audit the whole slice that sample came from. Retire cases tied to removed features, and archive them instead of deleting.

Features come back, and so do their bugs.

Bake in a rule that a case cannot enter the set without a named labeller. "Generated by the model and looked fine" is how you end up with an eval set that agrees with the model by construction.

The numbers I watch

Four of them. Pass rate sliced by the date a case was added. A per-case flakiness score, from running each case three times, so you know which failures are noise. The labeller agreement rate from the quarterly re-label. And the age distribution of the whole set, which tells you at a glance whether anyone is still feeding it.

The first one earns its keep fastest. If cases from the last month pass at 80 percent while cases from a year ago pass at 99, the product has drifted away from the old set and the old set is flattering you.

This is also my favourite question to put to vendors selling "agents." Real ones can show you their eval set and how it changed over the last quarter, and it's one of the fastest ways to tell a real agent from agent-washing. At Shanti Infosoft the eval set is a deliverable with a version number, handed over alongside the code, because a client who inherits a model without its eval set inherits something they can't safely change. That handover closes every machine learning engagement we take on.

When was the last time a case in your eval set was re-labelled by a human who didn't see the original answer?

Sagar Jain leads engineering as technical co-founder of Shanti Infosoft, a CMMI Level 5 team of 80+ engineers.

Hiding LLM Latency Without Lying to the User

sagar jain — Fri, 11 Sep 2026 08:00:05 +0000

You cannot make a six-second model call feel instant, but you can make it feel short and honest: get the first token on screen inside a second, do the predictable work before the user asks for it, arrange the answer so the useful part arrives first, and show progress that corresponds to something real. Fake spinners and a "thinking..." label with no end in sight teach users that the feature is slow and shifty. Both are fixable, and most of the fixes are product decisions rather than infrastructure work.

Where does the time actually go?

Two numbers, measured separately. Time to first token is dominated by queueing at the provider and by prompt processing, which scales with input length. Total time is generation, output tokens multiplied by per-token speed, plus every tool call, and tool calls run one after another unless you deliberately make them run together.

A system prompt of six thousand tokens plus a big retrieved context adds a noticeable delay before anything appears, and every one of those tokens costs money too.

The most common surprise when we instrument a slow feature: the model was fine, and the time went to three retrieval calls done sequentially, or to a bloated prompt nobody had trimmed since the pilot.

What works

Seven fixes, roughly in the order I'd apply them. Most are product and code decisions rather than infrastructure spend, which is why they're worth trying before anyone proposes a faster model or a dedicated endpoint. The first two usually account for the bulk of the improvement a user can feel.

Stream everything visible. A first token inside a second changes how the whole wait feels, even when the total is unchanged.
Parallelize tool calls that don't depend on each other. Three sequential 800-millisecond lookups become one 800-millisecond wait.
Prefetch on intent. If a user has typed a question and paused, start retrieval before they hit enter. Wrong guesses cost pennies; right guesses save seconds.
Split the work by model. A small, fast model produces the first visible piece (a title or a one-line summary) while the larger model produces the detail.
Order the output. Ask for the direct answer first and the reasoning after. The user reads the useful part while the rest streams.
Trim the prompt. Long system prompts are the cheapest latency fix nobody makes, because trimming feels risky. That's what the eval set is for.
Cap output length for tasks that don't need an essay.

What honest progress looks like

Honest progress shows real steps in words a user recognizes, and each label changes only when that step completes. If a task will genuinely take thirty seconds, say thirty seconds and offer to notify the user instead. Always give them a way to cancel. A progress indicator that corresponds to nothing is worse than no indicator at all.

"Searching your last 30 tickets," then "drafting reply." Those are labels a support agent can check against reality.

We learned the dishonest version the hard way. Early on we added a progress bar that animated to about 90 percent and then sat there until the response arrived. Actual latency didn't change. User feedback got noticeably worse, because a bar stuck at 90 percent feels broken in a way that a plain "working on it, about 20 seconds" doesn't. We ripped it out within two weeks.

Loading state	What the user infers	Whether it survived
Bar animating to 90%, then stalling	"This thing is broken"	Removed after two weeks
Bare spinner with no end in sight	"This is always slow"	Replaced with named steps
Named steps that advance on completion	"It's working through my request"	Current default
"About 20 seconds, want a ping?"	"Fine, I'll do something else"	Used for anything over ~15s

Why this matters commercially

Buyers judge AI features by feel, in the first minute of a demo and the first week of use. Slow and vague loses to fast and honest even when the slow one is more accurate, because latency is the one quality signal a buyer can read without doing the evaluation work themselves.

Anyone who has watched how buyers evaluate AI agents knows that latency reads as competence. At Shanti Infosoft we now put a time-to-first-token budget in the acceptance criteria for any user-facing AI feature, alongside the accuracy target, because one without the other doesn't ship. It goes into the scope on every generative AI build we take on.

What does your loading state tell the user right now, and is any of it true?

Sagar Jain is technical co-founder of Shanti Infosoft, a CMMI Level 5 company that has built software for 700+ businesses.

Least Privilege for AI Agents: Scope the Tools and the Tokens

sagar jain — Wed, 09 Sep 2026 09:00:10 +0000

An agent should get the narrowest tools and the shortest-lived tokens that let it do the one job in front of it. The same goes for data: only the slice that job needs. Most agent security incidents I've looked at came down to an agent holding an admin key it never needed, so an ordinary mistake turned into a data leak. Prompt injection makes the headlines, but sloppy scoping causes most of the damage I've seen.

Why do agents end up with god-mode credentials?

Convenience, and the pilot. The developer building the demo used their own token or an existing service account, and it had access to everything because that's what made the demo work on Friday. The demo became the product, and "we'll tighten permissions later" turned into a ticket that lost to features every sprint.

This is the same dynamic that produces the security bill that comes due after vibe coding: speed borrowed against a bill nobody scheduled.

Scope the tools

Scoping tools means one tool per action, each with the smallest surface that action needs. No generic "run SQL." Reads and writes split into separate tools with separate permissions. Arguments validated in your own code before execution. Minimal fields returned. The model chooses the tool; your code decides whether the parameters are acceptable.

Expose get_order(order_id) and list_orders(customer_id, limit) rather than a query endpoint.
Split reads from writes, so a read-only agent literally cannot call a write.
Allow-list tables and hosts, and enforce strict id formats, before anything runs.
If the agent needs an order status, don't hand it the whole customer record.
Give write tools a dry-run mode, and use it in evals and shadow phases.

Scope the tokens

Each agent gets its own service identity, and each task gets a token minted for that task. A support agent reading one customer's orders gets a token bound to that tenant, scoped to orders:read, with a ten-minute lifetime. If the agent is tricked, the blast radius is one customer's read-only order history for ten minutes.

	Shared admin service account	Task-scoped token
Scope	Everything the account can reach	One tenant, `orders:read`
Lifetime	Until somebody remembers to rotate it	Ten minutes
Blast radius if tricked	Every tenant, read and write	One customer's order history
Audit trail	"the AI service" did it	Named agent id, per call

Secrets never go into the prompt or the context window. The model shouldn't be able to see a key, so it can't be talked into repeating one. Every tool call is logged with the agent id, so the audit trail reads like a person's, because it should.

Scope the data

Retrieval has to respect the same access rules as the human asking. If your RAG index was built by a crawler running as an admin, the agent will happily surface documents the user could never open, and it will do it with a citation attached, which makes the leak look authoritative.

We caught exactly this in an internal pilot. A knowledge-base bot indexed a shared drive using a service account with broad read access, which included folders belonging to HR. During testing a colleague asked it, out of curiosity, what a particular role was paid, and it answered with a figure and a filename. It never left the pilot group, and it changed how we build every index since: document-level ACLs stored alongside the embeddings, filtered at query time by the caller's identity, with a re-sync whenever source permissions change. Row-level security in the vector store is mandatory if the source had it.

The checklist we run before an agent touches production

The review takes about an hour and covers tools, tokens, and the data behind retrieval. Nothing on it is exotic. It's a list of questions with named answers, and the value comes from writing those answers down somewhere a person outside the build team can read them.

For each tool: its narrowest scope, whether it reads or writes, what validates the arguments, and what a dry run looks like.
For each token: who mints it, what it's bound to, how long it lives, and where the call is logged.
For the data: whose permissions retrieval inherits, and how we know those permissions are current.

It's the best hour in the whole build. At Shanti Infosoft this review is a hard gate; the agent doesn't get a production identity until someone outside the build team has signed it off. It runs on every agent we deliver through our AI development work, including the ones a client will operate themselves afterwards.

If your agent were tricked into doing the worst thing its current permissions allow, what exactly would happen?

Sagar Jain is the technical co-founder of Shanti Infosoft, where agent systems ship under the same CMMI Level 5 process as everything else the team delivers.

Least Privilege for AI Agents: Scope the Tools and the Tokens

sagar jain — Wed, 09 Sep 2026 08:01:09 +0000

Why do agents end up with god-mode credentials?

This is the same dynamic that produces the security bill that comes due after vibe coding: speed borrowed against a bill nobody scheduled.

Scope the tools

Expose get_order(order_id) and list_orders(customer_id, limit) rather than a query endpoint.
Split reads from writes, so a read-only agent literally cannot call a write.
Allow-list tables and hosts, and enforce strict id formats, before anything runs.
If the agent needs an order status, don't hand it the whole customer record.
Give write tools a dry-run mode, and use it in evals and shadow phases.

Scope the tokens

	Shared admin service account	Task-scoped token
Scope	Everything the account can reach	One tenant, `orders:read`
Lifetime	Until somebody remembers to rotate it	Ten minutes
Blast radius if tricked	Every tenant, read and write	One customer's order history
Audit trail	"the AI service" did it	Named agent id, per call

Scope the data

The checklist we run before an agent touches production

For each tool: its narrowest scope, whether it reads or writes, what validates the arguments, and what a dry run looks like.
For each token: who mints it, what it's bound to, how long it lives, and where the call is logged.
For the data: whose permissions retrieval inherits, and how we know those permissions are current.

If your agent were tricked into doing the worst thing its current permissions allow, what exactly would happen?

Sagar Jain is the technical co-founder of Shanti Infosoft, where agent systems ship under the same CMMI Level 5 process as everything else the team delivers.

Putting a Number on Tech Debt So It Competes for Roadmap Time

sagar jain — Mon, 07 Sep 2026 08:00:32 +0000

Tech debt loses roadmap fights because it arrives as a feeling and features arrive as numbers. So we put a number on it: the hours per month the debt costs us right now, and the hours the fix would take. Then the debt item goes on the same list as the features and gets ranked by the same rule. Some debt loses that fight honestly, and that's fine. What isn't fine is debt losing because nobody bothered to price it.

Why "we have a lot of tech debt" never wins

Because it's unfalsifiable, and it sounds like engineers wanting to rewrite things. Leadership hears a preference. The competing feature request arrives with a customer name attached and a revenue guess beside it. Given one claim with numbers and one without, the numbered claim wins every planning round, and it should.

The pressure gets worse when the board is pushing for AI. I've written before about boards mandating AI before there's a use-case, and the engineering version of that story is a team asked to bolt an AI feature onto a system whose deploy takes forty minutes and whose test suite is red half the time. The debt is now the thing blocking the mandate, and still nobody has priced it.

How we measure the interest

We treat debt as a loan and measure its interest in engineer-hours per month across four buckets: recurring toil, incident time, friction, and blocked work. Each bucket is a count you can source from a calendar, a postmortem, or a CI dashboard, so the total is a number somebody else can check.

Recurring toil. Manual steps × frequency. A twenty-minute manual database migration step done three times a week is four hours a month, every month, forever.
Incident time. Hours spent on incidents where this debt was a cause or made recovery slower. Be honest, and use the last quarter's postmortems as the source.
Friction. Build and deploy time multiplied by how often it happens, plus time-to-first-merged-PR for a new joiner. A flaky forty-minute CI run at thirty runs a day is twenty hours of engineers waiting, per day, even if half of them context-switch productively.
Blocked work. Features that can't ship until this is fixed. List them by name; don't estimate their value, just make the dependency visible.

A worked example from an internal system, the deploy pipeline:

Bucket	Interest	Where the number came from
Recurring toil	~8 hrs/month	Manual migration step, three times a week
Incident time	~6 hrs/month	Last quarter's postmortems
Friction	~40 hrs/month	Slow CI run × runs per day
Blocked work	2 features	Named on the card, not valued
Total	~54 hrs/month	Fix estimated at 80 hours

Payback in about six weeks. Written that way it beat a mid-sized feature in the next planning round, and nobody had to argue about craftsmanship.

What goes on the roadmap card

Write the debt item exactly like a feature card: the outcome in plain language, the cost to fix, the monthly interest, the risk of ignoring it, and an expiry date. The expiry is the part people skip. Some debt is fine forever, because the system it lives in is being retired or barely changes.

For the deploy pipeline the outcome line read "deploys in five minutes, migrations automated," which a product manager can hold in their head without knowing what a migration is. Say on the card when a system is on its way out, and stop paying attention to its debt.

What changed once we did this

Fewer requests to rewrite things, because a rewrite priced honestly rarely wins. The debt that did win was the unglamorous kind: pipelines, flaky tests, a manual step everyone had normalized, a config file only one person understood. Engineers stopped bringing "everything is bad" to planning and started bringing invoices, which product people can argue with.

Working across many client codebases at Shanti Infosoft, the pattern is the same everywhere: the debt that hurts most is almost never the debt engineers complain about most. Pricing it is how you find out which is which, and it's the first exercise we run on a software consulting engagement before anyone proposes a roadmap. If you want a second opinion on your own numbers, book half an hour and bring last quarter's postmortems.

What is the most expensive piece of tech debt on your system right now, in hours per month, and could you defend that number to your CFO?

Sagar Jain co-founded Shanti Infosoft on the technical side. The firm runs a CMMI Level 5 delivery process across a team of 80+ engineers.

Ship AI Features Behind a Flag: Shadow Mode First, Then Canary

sagar jain — Sat, 05 Sep 2026 08:00:20 +0000

The safest way to ship an AI feature is to run it in shadow mode first (compute the output on real traffic, log it, show the user nothing), compare it against what the human or the old system did, then canary it to a small slice with a kill switch you have actually tested. Big-bang launches of a probabilistic feature mean your customers find the failure modes for you, in public.

What does shadow mode tell you that staging can't?

Shadow mode tells you the real input distribution, real latency under real concurrency, the true daily cost at production volume, and the agreement rate against whatever the feature replaces. Staging gives you none of those, because staging traffic is a sample you wrote yourself and production traffic is a sample nobody wrote.

The agreement rate is where the work sits. If the model and the humans agree 91 percent of the time, the job is sorting the other 9 percent into "model wrong," "human wrong," "both defensible," and "input was garbage," which you can only do with production examples.

Run it for one to two weeks, long enough to include a weekday peak and whatever monthly cycle your users have. On one project the shadow phase showed that a third of the daily cost came from a single automated client hammering the endpoint with near-duplicate requests. Nobody would have found that in staging, and a cache rule fixed it before launch.

How do you canary a probabilistic feature?

Canary a probabilistic feature by slicing traffic on a stable key, never per request, then stepping the slice up in stages with guardrail metrics that roll back automatically. A user who sees the AI answer on Monday and the old flow on Tuesday will report both as bugs, so consistency per user matters more than the percentage.

Slice by tenant or by a stable user hash, then walk it up:

One percent, held two or three days, watching guardrails more than product metrics.
Five percent, long enough to cover a full weekday cycle.
Twenty-five percent, where cost and latency problems stop being theoretical.
One hundred percent, with the flag still in the code and still drilled.

Guardrail metrics with automatic rollback thresholds are what make the canary safe instead of merely small:

Schema-validation failure rate above a set ceiling.
p95 latency above what the product accepts.
Thumbs-down or complaint rate versus the control slice.
Cost per day above the budget line.

If any of them trips, the flag flips back without a human deciding at 3am. That's the entire point of automating the threshold.

The kill switch you have tested

A tested kill switch is four flags: feature on/off, model version, prompt version, and force fallback. Rollback becomes a config flip that lands in seconds, with no deploy. Untested, it's a switch whose fallback path hasn't run since the AI path was added, which is a different thing entirely.

Flag	What it rolls back	What usually trips it
Feature on/off	The whole AI path	A guardrail breach or a live incident
Model version	A bad model upgrade	Accuracy or cost regression after a swap
Prompt version	A bad prompt change	Schema failures spiking after a release
Force fallback	Traffic to the old deterministic flow	Provider outage or latency blowout

Test the switch in production, on purpose, on a schedule. We do a monthly flag drill. The reason we do it monthly is a switch we once didn't drill: flipping it took the whole endpoint down, because the fallback code path had never been exercised since the AI path was added, and a dependency it needed had been removed in the meantime. The kill switch worked perfectly. It killed everything.

Why does this matter more for AI than for an ordinary feature?

Because trust in a probabilistic feature collapses on visible failures and rarely comes back. A deterministic bug gets fixed and forgiven. An AI feature that confidently gave twenty customers wrong answers on launch day gets switched off by the business, and that decision tends to outlast whatever the next version would have scored.

The project then joins the long list of AI agent projects expected to be dead by 2027. Shadow mode and a slow canary are how you find those twenty answers before customers do.

At Shanti Infosoft no AI feature goes to 100 percent without a shadow phase and a drilled kill switch. The same discipline carries into AI integration work inside someone else's existing product, where the blast radius isn't ours to clean up. Clients occasionally push back on the extra week. The ones who have shipped AI before never do.

Have you flipped your kill switch in production since the day you built it?

Sagar Jain is technical co-founder at Shanti Infosoft, a CMMI Level 5 shop that has shipped software for 700+ companies.

An Afternoon Data-Readiness Audit Before Any AI Build

sagar jain — Thu, 03 Sep 2026 08:00:25 +0000

Before scoping any AI feature, spend one afternoon answering six questions about the data it will run on: where it lives, who owns it, how fresh it is, whether ground truth exists, what's missing, and who is allowed to see it. Most of the "the model isn't good enough" failures I've seen were data-readiness failures discovered in month three instead of hour three. The audit is cheap. Skipping it is how projects die.

Why do AI builds stall on data?

AI builds stall on data because the pilot ran on a clean export and production runs on the real system. Real systems carry conflicting definitions, status columns that changed meaning, scanned PDFs, and access rules nobody mapped. None of that shows up in a demo, and all of it shows up around week five.

The real thing is fourteen tables with three different definitions of "customer," a status column that has meant four things over the years, PDFs that are scans, and a set of fields nobody outside finance is cleared to read.

Regulated industries feel this hardest, which is a large part of why AI projects die faster in fintech and healthtech: the model was fine, and the data-access question was never asked out loud.

The six questions, and what a bad answer sounds like

The six questions are location and access path, ownership, freshness needed versus actual, ground truth, gaps and dirt, and permissions. Each one has a green answer and a red one, and the red answers are specific enough that you can hear them coming in a scoping call.

Location and access path. Good: a read replica or an API with a named owner. Bad: "someone runs an export monthly and emails it."
Ownership. A person, with a name, who can answer questions about the schema. "The vendor" or "the old team" means nobody.
Freshness, needed versus actual. A daily batch is fine for a monthly report and useless for a support agent answering about an order placed an hour ago.
Ground truth. You should be able to show me two hundred examples of the right answer, produced by someone whose judgment you trust. If the answer is "we'll know it when we see it," you can't build an eval set, and you can't build the feature.
Gaps and dirt. Null rates, free-text fields that should be enums, IDs that don't join, dates stored as text. Ten minutes of GROUP BY and COUNT(*) reveals most of it.
Permissions. PII, contract terms, per-tenant isolation, retention rules. If the AI feature will read something a junior employee couldn't, that's a design constraint from day one.

A concrete example

One project wanted a support-ticket classifier trained and evaluated on the existing "category" field. A two-hour audit found that 40 percent of tickets were tagged "Other," that twelve categories had been renamed twice, and that busy agents had been picking the first item in the dropdown. The ground truth was fiction.

That afternoon saved us from three sprints of tuning a model against a label nobody believed. The first sprint became relabelling five hundred tickets with two reviewers each, and the classifier that followed had something honest to be measured against. The same trade shows up on every finding:

What the audit turns up	Cost if you find it in hour three	Cost if you find it in month three
No usable ground truth	One sprint of relabelling	Three sprints tuning against a label nobody believes
Monthly emailed export, no API	A scoping decision	A rebuild of the ingest path
PII the feature can't legally read	A design constraint	A launch held by legal review
Schema owner left the company	An hour finding a replacement	Weeks reverse-engineering columns

What to do with the answers

Score each of the six questions green or red, amber only if you must. Every red item becomes sprint zero, before a model is touched, and the answers go into the scope document so the client and the team hold the same picture. Re-run all six at every phase.

Data readiness decays. An owner leaves, a schema changes, a new source appears, a retention rule tightens.

At Shanti Infosoft this audit is part of how we scope every AI engagement, and it's the single most common reason a quoted timeline changes before a contract is signed. It's also the first thing we run when a team brings us in for AI consulting on a stalled build, because the fix is usually upstream of the model. I'd rather have that conversation in the first week than the fifteenth. If you want a second pair of eyes on your six answers, grab a slot and bring your schema.

If I asked you tomorrow morning for two hundred examples of the right answer, how long would it take you to find them?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 firm whose 80+ engineers ship AI systems into production.

The Take-Home Test We Replaced With a Debugging Session

sagar jain — Tue, 01 Sep 2026 09:00:12 +0000

We stopped sending take-home coding tests. Coding tools turned them into a measure of who has the better subscription. In their place we run a sixty-minute live debugging session in a small, realistic codebase, with any AI tool the candidate wants, and we watch how they read errors and how they decide what to trust. It predicts how someone performs in month three far better than a polished take-home ever did.

Why the take-home stopped working

The submissions got perfect and the conversations got worse. Clean code, full test coverage, a tidy README, sensible commit messages, and then a follow-up call where the candidate couldn't explain why a function existed. Meanwhile strong candidates were declining to spend six unpaid hours on a test, so we were filtering for free time rather than skill.

There's a bigger reason too. The way engineers work has shifted; coding agents are the fastest-growing thing on GitHub, and the day job now involves supervising generated code as much as writing it. A take-home tests the writing. We needed something that tests the supervising.

What we wanted to know	Take-home test	Live debugging session
Can they produce clean code	Yes, or their tool can	Not measured
Can they explain their own work	Often not, on the follow-up call	Watched live, out loud
Do they verify a wrong suggestion	Invisible to us	The timezone bug tests it directly
Cost to the candidate	Around six unpaid hours	Sixty minutes, scheduled

What the session looks like

A small service, roughly a thousand lines: an HTTP API, a database, a background job on a queue, and a tiny admin page. It runs locally in a container in under a minute, and we seed four bugs of different kinds into it before the call starts.

A timezone bug that only shows up for users west of UTC.
A race between the job and the API on the same row.
An off-by-one in cursor pagination that drops the last record of every page.
An error message that lies (it says "not found" when the real problem is a permission check).

The candidate gets the repo, a terminal, a browser, and a running database, and can use whatever assistant they normally use. We ask them to think out loud, and we say up front that fixing all four is not the goal. Most people fix two. Some fix one, and it's still a strong session.

What the rubric scores

The rubric has five rows, each scored one to three by two interviewers independently, and none of them counts bugs fixed. It scores how the candidate reads an error, whether they reproduce before fixing, whether they verify what the assistant suggests, how they pick between a quick patch and the right fix, and when they stop to ask.

Reads the error and the surrounding code before prompting or searching.
Reproduces the bug before trying to fix it.
Verifies what the assistant suggests instead of pasting it. We specifically watch what happens when the tool proposes a plausible wrong fix for the timezone bug, which it usually does.
Explains the trade-off of the fix and picks deliberately.
Knows when to stop and ask a question instead of thrashing.

Two interviewers scoring independently, then comparing, cut most of the "I just liked them" bias we used to have.

What changed in who we hired

We hired fewer framework-trivia specialists and more people who can operate a system they didn't write. Two surprises came out of it: some senior candidates struggled because they hadn't personally debugged anything in years, and some juniors who learned to code by prompting froze when the assistant's fix didn't work.

Both were useful things to know before an offer, and neither would have surfaced in a take-home. Hiring steadily for a team of 80-plus engineers at Shanti Infosoft means we run this session a lot, and it keeps earning its slot. It costs us an hour of two engineers' time per candidate. It saves us the far larger cost of finding out in month two. We now run the same format when staffing an IT consulting engagement where the client interviews our people too, and it survives that scrutiny better than a portfolio does.

When did you last watch a candidate debug something live, with the tools they'd actually use on the job?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 firm, and sits in on more of these sessions than any other kind of interview.

The Take-Home Test We Replaced With a Debugging Session

sagar jain — Tue, 01 Sep 2026 08:01:11 +0000

Why the take-home stopped working

What we wanted to know	Take-home test	Live debugging session
Can they produce clean code	Yes, or their tool can	Not measured
Can they explain their own work	Often not, on the follow-up call	Watched live, out loud
Do they verify a wrong suggestion	Invisible to us	The timezone bug tests it directly
Cost to the candidate	Around six unpaid hours	Sixty minutes, scheduled

What the session looks like

A timezone bug that only shows up for users west of UTC.
A race between the job and the API on the same row.
An off-by-one in cursor pagination that drops the last record of every page.
An error message that lies (it says "not found" when the real problem is a permission check).

What the rubric scores

Reads the error and the surrounding code before prompting or searching.
Reproduces the bug before trying to fix it.
Verifies what the assistant suggests instead of pasting it. We specifically watch what happens when the tool proposes a plausible wrong fix for the timezone bug, which it usually does.
Explains the trade-off of the fix and picks deliberately.
Knows when to stop and ask a question instead of thrashing.

Two interviewers scoring independently, then comparing, cut most of the "I just liked them" bias we used to have.

What changed in who we hired

When did you last watch a candidate debug something live, with the tools they'd actually use on the job?

Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 firm, and sits in on more of these sessions than any other kind of interview.

Reviewing AI-Written Pull Requests Without Burning Out Reviewers

sagar jain — Sun, 30 Aug 2026 08:00:35 +0000

When AI writes most of the diff, code review becomes the bottleneck, and the fix is a review process built for volume rather than reviewers who read faster. Smaller PRs enforced by tooling, a written intent block from the author, automated checks that clear the mechanical issues before a human looks, and a daily review budget per engineer. Reading faster doesn't scale. Reading less, and better, does.

Why does review load explode?

Because generation capacity went up and review capacity did not. An engineer with a coding agent produces three or four times the diff volume they used to, while the number of hours a reviewer has in a day stayed exactly the same. The queue absorbs the difference until somebody starts skimming.

Skimming an AI-written diff is worse than skimming a human one, because the code looks confident and idiomatic even when it's wrong. Fluent bugs. Those are the ones that reach production, and it's exactly why more AI code needs a heavier QA layer, starting with review itself.

What we require in every AI-assisted PR

Every AI-assisted pull request carries four things before a reviewer opens it: an intent block, a verification checklist the author signs, a size under the cap, and tests in the same change. The author does more work up front so the reviewer does less, which is the whole trade.

An intent block at the top of the description: what this changes and why, then what it deliberately leaves out. Three short paragraphs at most.
A verification checklist the author ticks honestly: ran it locally, added or updated tests, read every line of the diff, and can explain any file touched outside the ticket's scope. "The agent added a helper in a file I never opened" is a common finding and a red flag.
A size cap. Ours is around 400 changed lines. Bigger than that, split it or stack it. The tooling rejects the PR, so it never becomes a negotiation.
Tests in the same PR, and a note on what the tests don't cover.

None of this is anti-AI. Use whatever tool you like to write the code. The PR is still your claim that it's correct.

Which checks run before a human reads a line?

Everything mechanical runs first: lint, type checks, the test suite, coverage delta, dependency diff, secret scanning, and a generated summary telling the reviewer where the risk sits ("touches auth middleware and two migrations"). By the time a person opens the PR, what's left is judgment: does this design hold up, and what happens when it fails.

Layer	Who runs it	Blocks the merge?
Lint, types, tests, coverage delta	CI	Yes
400-line size cap	The tooling, automatically	Yes
AI first-pass review	A bot, before review is requested	No
Does this design hold up	A named human via CODEOWNERS	Yes

That AI first-pass reviewer has one firm rule: its comments are suggestions the author resolves before requesting review, never blockers and never a substitute for the human. It's good at catching the missing null check and terrible at knowing whether the change should exist at all.

How do you protect reviewer time?

Give review an explicit budget. Two hours a day per engineer, and after that the queue waits until tomorrow. Route risky paths through CODEOWNERS to the people who know them, let anyone review the rest, and track two numbers: time to first review, and the PR size distribution.

If sizes creep up, the cap isn't being enforced. Our own failed experiment: for a while we tried "everyone reviews everything" to spread knowledge. Time to first review went past three days and people rubber-stamped to clear the queue. Knowledge didn't spread anyway, because skimming teaches nothing. Small PRs plus rotating ownership worked better on both counts.

At Shanti Infosoft this is now the standard across our engineering teams, and it's the first process we rewrite when a software consulting engagement starts with "our velocity went up and our defect rate went up with it." The interesting side effect is that PR quality from human-only authors went up too. The intent block and the size cap were always good practice; the volume forced us to enforce them.

How long does a PR sit in your queue before someone reads it properly, and do you know that number or are you guessing?

Sagar Jain is the technical co-founder of Shanti Infosoft, which has delivered software for 700+ companies and reviews a great deal of machine-written code every week.

The Boring Layer Around Every LLM Call: Timeouts and Retries

sagar jain — Fri, 28 Aug 2026 09:00:56 +0000

Every LLM call in production needs a timeout tuned to the expected output, a retry policy with a hard cap and jitter, an idempotency key on any side effect that follows it, and a circuit breaker for the day the provider is degraded. None of that is AI work. All of it decides whether your feature stays up on a bad afternoon, and it's the layer most AI codebases I inherit are missing.

Why do LLM calls need different defaults from a normal HTTP call?

Because the latency distribution has a long tail and every retry costs real money. A typical internal API answers in 50 milliseconds with a p99 near 300. An LLM call answers in about 2 seconds with a p99 of 25 or more, and a long generation can legitimately run past a minute.

Copy your usual 10-second HTTP timeout onto that and you'll cancel healthy requests, retry them, pay twice, and hit your rate limit sooner. Our defaults, which we then tune per call site, start from the shape of the call:

Call shape	Total timeout	Idle between tokens	Retries
Short structured call	30s	15s	2
Long generation	90 to 120s	15s	1
Batch job item	30s	15s	2, under a cost ceiling

Connect timeout is 5 seconds everywhere. The idle timeout is the useful one, because we stream what we can and a stalled stream is a much earlier and cheaper signal than a total timeout.

How many retries, and on what?

Two, at most. Retry on 429s, 5xx responses, connection errors, and timeouts. Use exponential backoff with jitter and honour Retry-After when the provider sends it. Never retry a 400 or a schema-validation failure with the identical request; change something (feed the error back, shorten the input) or give up.

The mistake that taught us the jitter part: a batch job that retried five times with fixed one-second gaps. During a provider incident, a queue of about four thousand items all failed together and all retried together, in lockstep, five times each. We turned a provider slowdown into a self-inflicted rate-limit ban that lasted longer than the original incident. Now every retry policy carries a cost budget too, a per-hour ceiling on retry spend, and the job stops rather than burning through it.

Idempotency: the part people forget

The LLM call itself is usually safe to repeat. What follows it often isn't: send the email, create the ticket, post the comment, charge the card. If your timeout fires after the provider actually finished, and you retry, the model answers twice and the side effect runs twice.

The pattern is borrowed wholesale from payment processing:

Write the intent before you call the model, as a row with a unique key and status pending.
Pass that key through as the idempotency key on the side effect itself.
On any timeout, check whether the work already completed before you retry.
Mark the row done only once the side effect confirms it ran.

Circuit breakers and what "degraded" looks like

A circuit breaker trips after a threshold of failures inside a window, and while it's open the system stops sending hopeful requests. We start with five failures in sixty seconds per model endpoint. Degraded means you fall back on purpose instead of queueing traffic that has nowhere useful to go.

Fall back to a second provider behind the same interface, a smaller model, a cached answer from an earlier run, or an honest degraded mode ("we'll email you the summary in a few minutes"). Then close the breaker gradually with a trickle of test traffic.

I keep coming back to this because the industry conversation is about which agents are real and which are slop, and I think a large share of what separates agents that last from agents that are slop is this unglamorous layer. At Shanti Infosoft it lives in one shared client wrapper that every AI project imports, so nobody has to remember it and nobody can skip it. It's also the first file we write when a software development team hands us an AI feature that works on their laptop. If your provider's p99 has ever ruined an afternoon, a short call is a cheap way to compare notes.

What does your feature do at 2am when the provider's p99 triples for an hour?

Sagar Jain is the technical co-founder of Shanti Infosoft, where 80+ engineers write the boring layer around the interesting part.