The Real-Time AI Latency Budget, Allocated

Parikshit Talesara — Tue, 13 Oct 2026 12:18:25 +0000

Most latency targets for AI features arrive as one number on a ticket: "under 200 ms." Nobody owns 200 ms. Someone owns the vector search, someone owns the model server, someone owns the network in between, and each of them assumes they get most of it.

Below is the budget I'd start from for a retrieval-backed AI endpoint. A query comes in, relevant items are retrieved, a model ranks or scores them, and the result goes back while the user waits. Semantic search, recommendations and RAG-style lookups all have roughly this shape. Copy it, adjust it to your stack, and argue about it in the design review, not the incident review.

The budget

Bucket	Line item	p99 budget	What usually blows it
Network	Client to edge, round trip	25 ms	Fresh TLS handshakes, distant regions
Network	Internal hops (gateway, app, model)	10 ms	Sidecars and services added over time
Network	Response payload	5 ms	Returning full documents instead of IDs
Retrieval	Vector search (ANN)	20 ms	Index outgrows memory, filters applied after search
Retrieval	Metadata and feature fetch	25 ms	One call per item instead of one batched call
Retrieval	Permission and business filters	15 ms	Filtering in application code after retrieval
Inference	Query embedding	10 ms	Cold model, silent CPU fallback
Inference	Queue and batching wait	10 ms	Batch window tuned for throughput
Inference	Reranker over top candidates	45 ms	Too many candidates, input length creep
Inference	Post-processing	5 ms	Formatting in slow loops
Reserve	Headroom	30 ms	Spent on day one if nobody guards it
Total		200 ms

How to read it

Every number is a p99, not an average. Budgeting each hop at p99 is deliberately conservative, because every hop rarely hits its tail on the same request. The one place it isn't conservative enough is anything that fans out, where the request waits on the slowest of many calls. That's why the metadata line gets more than the vector search itself.

Inference is the biggest bucket, and the reranker is the biggest single line. A cross-encoder's cost grows with the number of candidates and their length, so cutting the rerank set from 50 to 20 usually buys back more time than any amount of model tuning.

Network looks generous at 40 ms. With users spread across continents it isn't, so serve from more regions or rebalance the table before launch.

The headroom is 15% of the total. It covers deploys; garbage collection pauses, noisy neighbors and the hop someone adds next quarter. If you can't find 15% spare on paper, you won't find it in production.

When the numbers don't fit

Cut in this order.

Do less work. Re-rank fewer candidates, return IDs instead of documents, and push filters into the index query so you don't retrieve things you'll throw away.

Move work off the request path. Cache embeddings for frequent queries, keep metadata in a store the request can hit once, and refresh features when events arrive rather than when users ask.

Then buy speed. Faster hardware or a smaller, quantized model. It works, but it's the most expensive fix, and the one teams reach for first.

Once the table balances, put a p99 alert on each line, not just on the total. When the total breaks, you want to know which line broke it.

First, check that 200 ms is the right number

The table assumes 200 ms is the right target, and sometimes it isn't. A feature that can answer within a second, or show up after the page loads, deserves a cheaper design. If that decision isn't settled yet, this walkthrough of real-time AI architecture, from latency tiers to the serving layer, is worth reading before you defend a single line above.

The Engineering Challenges Nobody Talks About When Scaling from 10 to 50 Developers

Parikshit Talesara — Thu, 16 Jul 2026 11:32:11 +0000

Growing an engineering team sounds like a problem every company wants to have.

More customers mean more features. More features require more developers. Hiring accelerates, new squads are formed, and suddenly the engineering organization has doubled in size.

On paper, this looks like progress.

Many companies discover that the same engineering practices that worked for a ten-person team begin to break down as the organization grows.

The codebase becomes more complex. Communication takes longer. Pull requests remain open for days. Releases require coordination across multiple teams, and developers spend more time discussing work than delivering it.

Scaling software development isn't simply about increasing headcounts. It's about building an engineering organization that continues to move efficiently as complexity grows.

The Communication Problem Arrives First

One of the earliest signs of growing pain isn't technical debt—it's communication debt.

With ten developers, everyone usually understands the product's roadmap, architecture, and release process.

At fifty developers, that's no longer possible.

Different teams have different services. Product managers prioritize competing initiatives. Infrastructure changes affect multiple applications, and engineers often don't realize another team is solving a similar problem.

Without structured communication, duplication becomes inevitable.

Successful engineering organizations reduce this risk by creating clear ownership boundaries rather than expecting everyone to know everything.

Architecture Must Scale Alongside the Team

As organizations grow, architecture becomes less about choosing the "best" technology and more about enabling independent delivery.

Large engineering teams benefit from:

Clearly defined service boundaries

Stable APIs

Well-documented contracts

Backward compatibility

Automated testing between services

The objective isn't simply building microservices.

It's allowing one team to improve a service without accidentally breaking another team's work.

Architecture should reduce dependencies—not create new ones.

Documentation Stops Being Optional

Many startups treat documentation as something they'll write later.

That works for small teams where everyone remembers why technical decisions were made.

It doesn't work when thirty new engineers join within a year.

Architecture Decision Records (ADRs), onboarding guides, API documentation, deployment runbooks, and coding standards become essential engineering assets.

Good documentation doesn't slow developers down.

It reduces repeated conversations, accelerates onboarding, and improves consistency across multiple teams.

Ownership Matters More Than Individual Productivity

Engineering leaders often measure productivity at an individual level.

But as organizations grow, team ownership becomes significantly more important.

Every service should answer simple questions:

Who owns it?

Who reviews architectural changes?

Who responds to incidents?

Who maintains documentation?

Who approves of deployments?

Ambiguous ownership creates bottlenecks so that no amount of hiring can be solved.

Clear ownership allows engineering teams to make decisions confidently without waiting for approval from half the organization.

Automation Becomes Your Biggest Multiplier

Hiring twenty additional engineers without improving engineering workflows simply creates larger queues.

High-performing organizations invest heavily in automation.

Continuous Integration catches regressions before reviewing.

Continuous Delivery reduces release of friction.

Infrastructure as Code keeps environments consistent.

Automated security scanning identifies vulnerabilities early.

Instead of adding manual processes to manage growth, successful teams automate repetitive work, so engineers can focus on solving customer problems.

The Cost of Losing Product Context

One challenge that rarely appears on hiring dashboards is context loss.

Every time experienced engineers leave—or project teams rotate out—valuable knowledge disappears with them.

New developers eventually understand the code, but they don't immediately understand the decisions behind it.

Why was this service separated?

Why does this workflow exist?

Which customers depend on this legacy endpoint?

That knowledge only develops through long-term involvement with the product.

Organizations that retain engineering continuity usually make better technical decisions over time because fewer assumptions need to be rediscovered.

Scaling Capacity Without Starting Over

Growing companies don't always have the luxury of waiting six months to recruit every engineer internally.

Hiring remains important, but product roadmaps rarely pause while recruitment catches up.

Many engineering organizations address this by extending existing teams with embedded engineers who work inside the same sprint cadence, development standards, and delivery processes instead of operating as a separate project vendor.

This dedicated engineering team model works particularly well because new engineers contribute within existing workflows rather than creating parallel development streams. The emphasis remains on shared ownership, long-term product knowledge, and continuous delivery—not simply completing a fixed scope of work.

The Right Growth Model Depends on the Product

There's no universal blueprint for scaling software teams.

Some organizations benefit from aggressive in-house hiring.

Others rely on specialized platform teams.

Many combine internal leadership with embedded engineering partners to expand delivery capacity while preserving technical continuity.

What's important is evaluating these options beyond hourly rates or recruitment costs. Factors such as onboarding time, engineering productivity, knowledge retention, and long-term delivery efficiency often have a much greater impact on product success.

Before deciding how to scale, it's worth understanding the cost comparison between dedicated engineering teams and traditional hiring models, particularly products expected to evolve over several years rather than a single release cycle.

Final Thoughts

Engineering organizations don't become more productive simply because they become larger.

Sustainable growth comes from improving the systems around software development—clear ownership, scalable architecture, effective documentation, automation, and strong collaboration.

When those foundations exist, adding engineers increases delivery capacity instead of operational complexity.

Whether teams are built entirely in-house or expanded through embedded engineering partnerships, the organizations that scale successfully all share one characteristic: they invest as much in how their teams work together as they do in the technology they build.

DEV Community: Parikshit Talesara