DEV Community: Pawan Kumar

The Request Is the Wrong Unit of Scale for LLMs on Kubernetes

Pawan Kumar — Thu, 21 May 2026 03:32:01 +0000

Series links

Part 1: Everything You Know About Scaling Web Apps Breaks When You Serve an LLM

Your dashboard says traffic is flat. Requests per second barely moved. CPU looks fine. Memory looks normal. The HPA is calm. Then latency starts drifting. Time to first token gets worse. GPU memory pressure rises. Queues grow. Users complain that the model is "thinking forever."

Part 1 introduced why LLM serving breaks the normal web-scaling model. Part 2 zooms into one reason: the HTTP request is only the envelope. The real work is token processing.

For a normal web app, a request is often a useful approximation of work. One request hits an API, does some bounded work, maybe talks to a database, returns JSON, and ends. LLMs do not behave like that. One request may contain a 20-token question and produce a 50-token answer. Another may contain a long system prompt, full chat history, retrieved documents, tool output, metadata, and a user asking for a 4,000-token report.

Both are one HTTP request.

They are not the same workload.

Kubernetes may see one request. Your ingress may see one request. Your API gateway may see one request. But the GPU sees tokens: prefill work, decode work, KV cache growth, memory pressure, queueing, and time spent generating output one token at a time.

Tokens are the work.

Why request count worked for web apps

Most platform teams grew up around request-based thinking. We look at requests per second, p95 latency, p99 latency, error rate, CPU usage, memory usage, queue depth, pod count, and replica count. That model works reasonably well for many web services because requests are often similar enough for capacity planning.

Not always, of course. A login request is not the same as an export request. A cached read is not the same as a database-heavy query. Every experienced SRE has seen one "simple" endpoint melt something important. But request count still gives a useful first approximation in many normal systems.

With LLM serving, that approximation breaks faster. A request does not tell you how long the prompt is, how many retrieved documents were added, how much chat history was included, how many output tokens the model generated, how much KV cache was needed, or how long the request occupied the GPU.

This is why a Kubernetes deployment can look stable at the HTTP layer while the model server is under real pressure. The API did not necessarily get more traffic.

The traffic got heavier.

Input tokens and output tokens are different problems

When people first hear "tokens are the unit of work," they often treat all tokens as one bucket. That is a good starting point, but it is not enough.

For serving, input tokens and output tokens stress the system differently.

At a high level, LLM inference has two phases:

Prefill
Decode

The prefill phase processes the input prompt. This includes the system prompt, developer instructions, chat history, retrieved documents, tool results, the user's message, and whatever formatting your application adds before the request reaches the model. The decode phase generates the response one token at a time. The model predicts a token, appends it to the sequence, uses that updated sequence to predict the next token, and keeps going until it hits a stop condition or a token limit.

A practical way to remember it:

Input tokens decide how heavy it is to start answering.

Output tokens decide how long the model stays busy.

Long input usually hurts time to first token because the model has to process the prompt before it can begin generating. Long output usually hurts total latency and capacity because the model stays in the generation loop longer. Streaming can make this feel better to the user, but streaming does not remove the backend work. It just lets the user watch the work happen.

This is why serious LLM serving metrics talk about time to first token, time per output token, inter-token latency, and tokens per second. NVIDIA's LLM benchmarking docs describe TTFT as the time before the first output token appears, and note that longer prompts generally increase TTFT because the input sequence has to be processed and the KV cache has to be created. Databricks' inference performance guidance also separates TTFT, TPOT, latency, and throughput instead of treating latency as one simple number.

A normal API request is one operation.

An LLM request is a sequence of token work.

One request can hide a huge amount of prompt assembly

The user does not usually send the real prompt.

The application builds it.

A user might type:

What is our refund policy for enterprise customers?

That looks tiny.

But by the time your application sends the request to the model, the prompt might include:

system prompt: 700 tokens
developer instructions: 400 tokens
chat history: 1,500 tokens
retrieved policy documents: 6,000 tokens
citations and metadata: 600 tokens
user question: 12 tokens
formatting instructions: 300 tokens

The user sent one short question. The model received more than 9,000 input tokens before it generated a single output token. That is one of the easiest mistakes to miss in production: teams measure the user message size, not the final assembled prompt size.

RAG makes this even more interesting. Retrieval-augmented generation is often described as a quality feature. The model gets relevant context, answers with better grounding, and can cite internal documents. That is true. But RAG is also an infrastructure multiplier.

Changing top_k from 4 chunks to 12 chunks may look like a harmless retrieval tuning change. No Kubernetes manifest changed. No model changed. Request count did not change. The product team may even see better answers. But now every request may carry thousands of extra input tokens. That can affect time to first token, GPU memory pressure, KV cache usage, batch composition, queueing delay, maximum concurrency, tail latency, and cost per interaction.

This is why prompt assembly needs observability. You do not only want to know that a request had 9,000 input tokens. You want to know where those tokens came from: chat history, retrieved documents, tool results, system instructions, verbose metadata, tenant documents, or an agent flow that appends every intermediate step.

Without that breakdown, token growth stays invisible until latency tells you something is wrong.

Long context is a capacity decision

Long-context models are useful. They let you analyze larger documents, keep longer conversations, handle more retrieval context, and build richer workflows. But a large context window is not a target. It is a limit.

This sounds obvious, but many teams behave as if "the model supports 128k context" means "we can casually send 128k context." That is like saying a node has 1 TB of memory, so every process should try to use it.

Long context changes the shape of serving. A small number of long-context requests can consume enough GPU memory and serving time to affect everyone else. A chat session can become more expensive as history grows. An agent can quietly append tool traces until each turn becomes much heavier than the first one. A summarization feature can go from "summarize this page" to "summarize this folder of documents" without the HTTP request count changing at all.

The failure mode is subtle because the old dashboard may still look calm. RPS is flat, but p95 input tokens moved from 2,000 to 18,000.

That is not flat traffic.

A useful platform practice is to bucket prompts by size: short prompts, medium prompts, long prompts, very long prompts, and batch or offline prompts. The exact numbers depend on your model and hardware, but the habit matters more than the bucket boundaries.

A 500-token chat and a 50,000-token document analysis should not be treated as the same class of work just because both entered through /v1/chat/completions.

Context windows are limits, not goals.

Output length is not just a UX choice

Input tokens get a lot of attention because long prompts are easy to blame. But output tokens can be just as important for capacity planning.

Two requests can have the same input prompt and completely different backend cost depending on output length.

Request A:

1,000 input tokens
100 output tokens

Request B:

1,000 input tokens
2,000 output tokens

Same route. Same user flow. Same prompt size. Very different serving time.

The second request keeps the model generating for much longer. If the response is streamed, the user may start seeing text quickly, which is good. But the GPU is still occupied while the model continues decoding token after token.

This is why max_tokens is not only a product parameter. It is a capacity control.

If every request is allowed to generate 4,000 tokens, you have created a worst-case capacity problem even if most responses are shorter. If a feature asks the model to "write a detailed report," that is not the same workload as "answer this chat question." If agents are allowed to produce long reasoning traces, tool plans, summaries, and final answers, output length can grow quickly.

You should track both requested maximum output tokens and actual generated output tokens. Requested max output tokens show the capacity risk your system accepted. Actual output tokens show the work the model really performed. If many requests hit the output cap, your users may be getting truncated answers. If very few requests use the available budget, your default might be too generous.

Output length is not formatting.

It is how long the request rents the GPU.

Same request count, completely different load

A useful dashboard should show when the same request count hides a different workload shape. For example:

Window A:

requests: 1,000
average input: 500 tokens
average output: 150 tokens
total token work: 650,000 tokens

Window B:

requests: 1,000
average input: 8,000 tokens
average output: 1,000 tokens
total token work: 9,000,000 tokens

Both windows show 1,000 requests. But the second window has almost 14x the token volume. If your dashboard only shows request count, it says traffic is flat. If your dashboard shows token volume, it says the workload changed completely.

That is why the useful question is not only:

How many requests are we serving?

It is also:

How many input tokens are arriving, how many output tokens are being generated, and where are those tokens coming from?

What Kubernetes sees and what the model server feels

Kubernetes is very good at managing containers. It can schedule pods, restart failed workloads, apply resource requests and limits, spread replicas, roll out deployments, and attach workloads to GPU nodes. But Kubernetes does not automatically understand the shape of an LLM request. A pod can be healthy while the model server is struggling. CPU can look uninteresting while GPU memory is the real limit. Generic memory can look fine while KV cache is under pressure. Request count can look flat while token volume has exploded.

This is where the division of responsibility matters. Kubernetes gives you the orchestration layer. The model server gives you the LLM execution layer. The application builds the prompt. The platform team has to connect the signals.

If those layers do not share the right metrics, you end up scaling the wrong thing. For example, CPU-based HPA may be useful around some parts of the stack, but it is not enough to understand LLM serving capacity. A model server may expose more relevant metrics such as prompt tokens, generation tokens, time to first token, time per output token, queue time, number of running requests, number of waiting requests, and KV cache usage.

vLLM's production metrics are a good example of where the industry is moving. It exposes metrics for prompt tokens, generation tokens, request prompt tokens, request generation tokens, time to first token, time per output token, request queue time, prefill time, decode time, KV cache usage, running requests, and waiting requests. That metric set tells you something important:

The production surface of LLM serving is already token-aware.

Your dashboard should be too.

Token-based observability is not optional

If you are running LLM workloads on Kubernetes, request count still matters. You still need API-level metrics. You still care about errors, availability, saturation, queueing, and latency. But those metrics need token context.

At minimum, every request should give you:

input tokens
output tokens
total tokens
requested max output tokens
time to first token
time per output token or inter-token latency
end-to-end latency
queue time
model name
model version
deployment
tenant or team
route or feature
finish reason

If possible, also track prompt composition:

system prompt tokens
chat history tokens
retrieved context tokens
tool result tokens
user message tokens
metadata or formatting tokens

That breakdown is where many production surprises hide.

For dashboards, averages are not enough. Average token count can look stable while the tail gets ugly. You want p50, p95, and p99 for input tokens and output tokens. You want latency by token bucket. You want TTFT by input size. You want end-to-end latency by output size. You want to know whether a tenant is sending mostly short prompts or occasionally sending giant ones.

Some useful views:

input token p50, p95, p99
output token p50, p95, p99
total tokens per minute by model
total tokens per tenant
TTFT by input token bucket
TPOT by output token bucket
queue time by token bucket
percentage of requests near context limits
percentage of requests hitting output cap
retrieved context tokens per request
chat history tokens per request
KV cache usage over time
waiting requests alongside waiting token estimates

The last point is important.

Do not only ask how many requests are waiting. Ask how many tokens are waiting. A queue of 20 short chat requests and a queue of 20 long document-analysis requests are not the same queue.

Product changes become infrastructure changes

One uncomfortable part of LLM platforms is that product changes can become infrastructure changes very quickly.

In a normal web app, adding a new field to a response may not matter much. In an LLM application, adding more context to a prompt can change capacity. Increasing retrieval depth can change latency. Keeping longer chat history can change memory pressure. Allowing longer outputs can change GPU occupancy.

The product team might say:

We only changed the prompt.

The platform team hears:

We changed the workload.

Both are true.

This does not mean product teams should be afraid of improving prompts. It means token impact should be visible before and after the change. If a new prompt improves answer quality but increases average input tokens by 3x, that may be a good tradeoff. But it should be a conscious tradeoff.

If a RAG change improves accuracy but pushes p99 prompts near the context limit, that should be visible before production users discover the latency problem. If a new report-generation mode produces 10x more output tokens than chat, it probably needs a different workload class and different expectations.

The platform question is not "are tokens bad?"

Tokens are the product.

The question is whether you know how many you are serving, where they come from, and what they do to your capacity.

Practical rules for platform teams

If you are starting to serve LLMs on Kubernetes, measure input and output tokens for every request. Do not wait until the first incident to add token metrics. Track the final assembled prompt, not just the user message. The model does not care what the user typed. It cares what your application sent.

Break input tokens down by source. Separate system prompt, chat history, retrieved context, tool results, and user message. Track requested max output tokens separately from actual output tokens. One tells you accepted risk. The other tells you real work.

Use token buckets in latency dashboards. A p95 latency graph without token buckets mixes small chat requests and huge document requests into one misleading line. Watch p95 and p99 token counts, not just averages. The tail is where LLM serving gets painful.

Put budgets on RAG retrieval. top_k is not only a relevance knob. It is a capacity knob. Treat context windows as limits, not targets. Just because a model accepts long context does not mean every request should use it. Set sane output defaults. Long answers should be intentional, not the accidental default for every route.

Separate workload classes when needed. Short interactive chat, long RAG, report generation, agent workflows, and batch summarization do not have the same shape. Review token growth after product changes. Prompt changes, retrieval changes, memory changes, and tool changes can all affect infrastructure.

These rules are not about making the system slower or less useful. They are about making the system understandable.

You cannot operate what you do not measure. And in LLM serving, measuring only requests means you are measuring the envelope while ignoring the work inside it.

The real unit of scale

The request is still useful at the API boundary. You need it for authentication, rate limits, logs, tracing, errors, and user flows. But it is not the right unit for LLM capacity.

It cannot tell you how much prompt the model processed, how long the model generated, how much KV cache was needed, or whether the workload was short chat, long-context RAG, report generation, or an agent loop.

Tokens get you closer to the truth. Input tokens explain much of the work before the first response appears. Output tokens explain how long the model keeps generating. Token distributions explain why averages lie. Token sources explain which product behavior changed the workload. Token-aware metrics explain why your Kubernetes deployment looks healthy while users still feel latency.

Part 1 was about letting go of the normal web app scaling model. Part 2 is about replacing one of its most misleading assumptions.

For LLMs on Kubernetes, you are not really scaling requests.

You are scaling token work across expensive, memory-constrained, latency-sensitive GPU systems.

Once you see that, the rest of the platform starts to make more sense.

Continue the series

I am writing this as a practical series on hosting large LLMs on Kubernetes, from GPU nodes and model servers to autoscaling, latency, cost, and production architecture. If you want the next part, subscribe to the newsletter.

I am also preparing a free LLM Serving on Kubernetes Production Readiness Checklist with the metrics, dashboard questions, and architecture review points platform teams should ask before putting an LLM workload in production. Subscribe and I will share it when it is ready.

Everything You Know About Scaling Web Apps Breaks When You Serve an LLM

Pawan Kumar — Thu, 14 May 2026 03:33:30 +0000

Most platform engineers already know how to scale a web app. Put it in a container. Deploy it on Kubernetes. Add CPU and memory requests. Put a Service or Ingress in front. Configure HPA. Watch p95 latency, error rate, CPU, memory, and request throughput. Add replicas when traffic goes up. This is Part 1 of a practical series on hosting large LLMs on Kubernetes.

That playbook works for a lot of services. Then you try to serve a large language model, and suddenly the old model starts cracking. A request is no longer just a request. Memory does not just mean RAM. Latency is not one number. Scaling a pod does not mean capacity appears instantly. One "replica" may need one GPU, eight GPUs, or several machines working together.

And the bottleneck may not be CPU at all. The first mental shift is simple:

LLM serving is not normal web serving.

The real unit of work is the token.

A request is no longer a request

In a normal web app, request count is often a useful planning signal. Not perfect, obviously. Some endpoints are heavier than others. Some queries are ugly. Some users manage to find the one path that melts the database. But request count still tells you something.

With LLMs, it can lie to your face.

One user asks:

Summarize this sentence.

Another user asks:

Analyze this 80-page contract, compare it with these policy documents, extract the risks, and generate a detailed memo.

Both are one request. They are not the same workload.

The second request may contain thousands of input tokens. It may generate thousands of output tokens. It may sit on GPU memory for longer. It may increase queueing delay for everyone behind it. It may consume far more KV cache. It may make your latency charts look haunted.

So if you only measure requests per second, you are almost blind. For LLMs, you need to care about:

input tokens
output tokens
tokens per second
time to first token
time per output token
queue depth
batch size
GPU memory
KV cache usage
model loading time

That is a very different world from normal HTTP throughput.

LLM inference has two phases, and they behave differently

When a user sends a prompt to an LLM, the model does not handle the request as one uniform block of work. At a high level, inference has two phases:

prefill
decode

Prefill is where the model processes the input prompt. If the prompt is long, prefill gets expensive. This is where the model reads the context and builds the internal state needed to start generating. Decode is where the model generates output tokens one at a time. This is the part users see when text starts streaming on the screen.

These phases stress the system differently. Prefill is more compute heavy. Decode is often more memory bandwidth heavy. Prefill depends heavily on input length. Decode depends heavily on output length. Both affect latency, throughput, cost, and capacity.

This distinction does not usually matter when you are scaling a normal API. You do not think of a checkout endpoint as having two GPU phases with different scheduling behavior. With LLMs, you have to.

If you ignore prefill and decode, you will struggle to explain why first token latency is slow, why long prompts hurt so much, or why the GPU looks busy but users still complain.

Latency is not one number anymore

For web services, we usually talk about latency as one number:

p50 latency
p95 latency
p99 latency
request duration

For LLMs, that is not enough. Two latency numbers matter a lot.

Time to first token

Time to first token, or TTFT, is how long the user waits before the model starts responding. This controls the feeling of responsiveness.

If nothing appears for five seconds, the product feels slow. It does not matter that the final answer is useful. The user has already started wondering if the system is stuck.

TTFT is affected by:

queueing delay
prompt length
prefill time
model routing
batch scheduling
GPU availability
cold starts
cache behavior

Users feel TTFT sharply because silence feels broken.

Time per output token

Time per output token, or TPOT, measures how fast the model generates each token after generation starts. This controls the streaming experience.

Good TTFT with bad TPOT feels like the model wakes up quickly and then crawls. Good TPOT makes the answer feel alive, even if the full response takes time.

TPOT is affected by:

decode efficiency
GPU memory bandwidth
batch size
KV cache pressure
model size
quantization
serving engine
hardware type

Normal web systems rarely force you to separate "time until the response starts" from "speed at which the rest of the response streams." LLM serving does.

Memory means GPU memory now

In a web app, memory usually means heap, runtime overhead, in-process cache, or connection pools. In LLM serving, memory often means GPU memory. And GPU memory is painful because it is limited, expensive, and easy to waste.

You need GPU memory for:

model weights
KV cache
runtime buffers
activations
batching overhead
framework overhead

Model weights are the obvious part. A 7 billion parameter model in FP16 or BF16 needs roughly 14 GB just for weights. A 70 billion parameter model needs roughly 140 GB just for weights at that precision. That already means one GPU may not be enough.

But weights are only the obvious cost. The hidden cost is KV cache.

KV cache stores the key and value tensors from previous tokens so the model does not recompute everything from scratch during generation. The longer the context and the more concurrent users you serve, the more KV cache you need.

This is why long context is not just a product feature. It is an infra bill. Every extra token you allow into the context window can come back as GPU memory pressure. Maximum context length is not only a model capability. It is a capacity planning decision.

Replicas are not always replicas

In a normal web app, one replica usually means one pod running one copy of the application. Traffic goes up, add pods. With LLMs, the word "replica" can hide a lot.

A small model may run inside one pod on one GPU.

A larger model may need:

multiple GPUs in one node
multiple pods on one node
multiple nodes
tensor parallelism
pipeline parallelism
a Ray cluster
a leader-worker setup
a group of pods that must start together

So when someone says, "scale the model to 10 replicas," the first question should be: what is one replica?

Is it one pod? One GPU? One tensor parallel group? One multi-node deployment? One endpoint backed by several workers? One prefill group plus one decode group?

This is where Kubernetes abstractions get interesting. A Deployment works nicely for simple stateless services. Serious LLM serving may need Ray, KServe, LeaderWorkerSet, Kueue, Volcano, or custom orchestration.

The model may not fit into the old "one pod equals one replica" picture.

Scaling a pod does not mean capacity is ready

In a normal web app, a new pod can become useful quickly. The image is pulled. The process starts. Readiness passes. Traffic flows. For LLMs, a new pod may sit there for a while before it can handle real traffic.

It may need to:

pull a large container image
download model weights
load hundreds of GBs from object storage or disk
initialize CUDA
allocate GPU memory
build or load optimized engines
warm up the model
join a distributed serving group

This can take minutes. Sometimes longer. So autoscaling is not just about deciding when to add replicas. It is about adding capacity early enough that it is ready before users feel the pain.

That is much harder than scaling a normal web app. This is why LLM platforms often use:

minimum warm replicas
preloaded models
local NVMe model cache
warm pools
separate GPU node pools
predictive scaling
queue based scaling
scheduled capacity for known peaks

Scale to zero sounds great until the first user waits for a giant model to load.

CPU autoscaling becomes a weak signal

CPU utilization is a decent signal for many Kubernetes workloads. Not perfect. But decent. For LLM serving, CPU can be almost irrelevant.

The expensive work happens on GPUs. More specifically, the bottleneck may be:

GPU memory
GPU memory bandwidth
KV cache capacity
decode throughput
queue depth
batch saturation
inter-GPU communication
model server scheduling
request length distribution

A model server can have low CPU usage and still be overloaded. It can have high GPU utilization and still deliver terrible latency. It can have enough compute but not enough KV cache capacity. It can be stuck serving long prompts while short prompts wait behind them.

So if you autoscale only on CPU, the platform may make the wrong decision at the worst possible time.

Better signals include:

queue depth
waiting requests
ongoing requests per replica
batch size
TTFT
TPOT
tokens per second
KV cache usage
GPU memory pressure
SLO burn rate

GPU utilization still matters. It just cannot be the only signal. LLM autoscaling has to understand the workload.

Round robin load balancing gets weird

For a normal web app, round robin load balancing is often fine. Request 1 goes to pod A. Request 2 goes to pod B. Request 3 goes to pod C. For LLMs, this can be wasteful.

A short prompt and a long prompt have completely different costs. A request with a cached prefix may be cheaper if it lands on the right worker. A long generation may occupy capacity much longer than the load balancer expects. One tenant may need lower latency than another. One model may need different hardware from another.

Naive load balancing can create strange failures:

one worker gets long prompts and slows down
another worker stays underused
KV cache locality is lost
prefix caching becomes less useful
tail latency gets worse
GPU utilization looks fine while users are unhappy

LLM serving needs smarter routing.

Good routing may consider:

model name
prompt length
estimated output length
tenant priority
cache locality
GPU availability
queue depth
hardware type
region
latency SLO

This is why inference gateways, model-aware routing, and cache-aware scheduling matter.

Cost changes shape

In a web app, cost usually grows with pods, CPU, memory, database load, and network traffic. In LLM serving, cost is shaped by GPU usage, and GPUs are expensive enough that small inefficiencies matter.

You can burn money through:

idle GPUs
poor batching
overprovisioned replicas
long context windows
bad routing
large models for simple tasks
no quantization
slow cold starts
inefficient KV cache usage
serving every request with the same model

The cost unit also changes. Instead of only thinking about cost per request, you start thinking about:

cost per input token
cost per output token
cost per million tokens
cost per model
cost per tenant
cost per GPU hour
cost per region
cost per latency tier

This is the cloud bill version of the first mental shift: a request is not a request. A token is the real unit of work.

Kubernetes is still useful. It is just not enough by itself

None of this means Kubernetes is the wrong platform for LLM serving. Kubernetes still gives you a lot:

scheduling
declarative deployment
resource management
isolation
service discovery
rollouts
observability integrations
autoscaling primitives
platform patterns for multiple teams

That is why many serious AI infrastructure platforms still use Kubernetes or something close to it. But Kubernetes does not automatically understand LLMs.

Out of the box, Kubernetes does not know:

what KV cache is
whether a model is loaded
whether a GPU group must be scheduled together
whether pods should land in the same rack
whether a request has 100 tokens or 100,000 tokens
whether TTFT is bad
whether a model server is overloaded despite low CPU
whether a new replica will take 10 minutes to warm up

You have to teach the platform these things through metrics, controllers, schedulers, serving frameworks, routing layers, and operational discipline. That is the real work.

The old scaling model breaks

The old web scaling model looks like this:

Traffic increases
        ↓
CPU increases
        ↓
HPA adds pods
        ↓
Load balancer spreads requests
        ↓
Latency improves

That still works for many stateless services. LLM serving looks more like this:

Traffic increases
        ↓
Input and output token mix changes
        ↓
Queue depth grows
        ↓
KV cache pressure increases
        ↓
Batching behavior changes
        ↓
TTFT and TPOT drift
        ↓
GPU memory or decode throughput becomes the bottleneck
        ↓
Autoscaler needs model-aware metrics
        ↓
New capacity may take minutes to warm up

If you bring only the old playbook, you will scale the wrong thing, at the wrong time, using the wrong signal.

The new mental model

To serve LLMs well, you need a different model in your head:

A request is not the unit. A token is.
Memory is not just RAM. GPU memory and KV cache matter more.
Latency is not one number. TTFT and TPOT matter separately.
A replica may be a distributed group, not a single pod.
Scaling is not instant because model loading is slow.
CPU is not a reliable autoscaling signal by itself.
Load balancing must understand request cost and cache locality.
Long context is an infrastructure cost decision.
Cost optimization starts with keeping expensive GPUs useful.
Kubernetes is the foundation, but LLM-aware systems must be built on top.

Once this clicks, the rest of LLM infrastructure becomes easier to reason about. You can understand why vLLM became popular. Why PagedAttention matters. Why KV cache dominates serving design. Why quantization is a capacity strategy. Why topology-aware scheduling matters. Why teams split prefill and decode. Why GPU cost optimization is its own discipline. Why normal autoscaling is not enough.

LLM serving is not "deploy a model behind an API." It is a new platform engineering problem.

Closing thought

For years, platform teams became very good at scaling stateless web services. We learned containers, Kubernetes, service meshes, autoscaling, observability, progressive delivery, and cloud cost optimization. That knowledge still matters, but LLM serving changes the shape of the problem.

The bottlenecks move. The metrics change. The cost model changes. The scheduler matters more. The load balancer needs to get smarter. The GPU becomes the scarce resource. The token becomes the unit of work.

So if you are trying to serve LLMs on Kubernetes, the first step is not installing a Helm chart. The first step is replacing the old mental model.

Because everything you know about scaling web apps starts to break the moment you serve an LLM.

Continue the series

I am also preparing a free LLM Serving on Kubernetes Production Readiness Checklist with the questions platform teams should ask before putting an LLM workload in production. Subscribe and I will share it when it is ready.

I Don't Want AI to Replace DevOps. I Want It to Read the Docs I'm Too Tired to Read

Pawan Kumar — Thu, 07 May 2026 06:32:51 +0000

Originally published at dheeth.blog

It's 2 AM. The pager went off eleven minutes ago. You're staring at a Kubernetes upgrade advisory that's forty-seven paragraphs long, and somewhere in paragraph thirty-one there's a breaking change about how EKS handles PodIdentity federation with IAM roles. You know it's in there. You read it three months ago. But right now your brain is running on caffeine and cortisol, and the words are blurring into each other.

You could run the upgrade now and hope for the best. Or you could spend forty minutes re-reading the entire changelog, the Terraform provider notes, the Helm chart migration guide, and three different Slack threads from the last time someone did this.

This is the part of DevOps nobody puts in conference talks. Not the elegant GitOps pipelines or the slick dashboards. The part where you're exhausted and you still have to make a decision that affects production, and the information you need is spread across nine browser tabs, a Confluence page from 2023, and a runbook that was last updated when your cluster was on 1.24.

This is where I want AI to help. Not by taking over. Not by running kubectl apply on my behalf while I sleep. By reading the damn docs for me.

The kind of tired that matters

The Google SRE Workbook has a word for what happens when engineers spend too much time on repetitive operational work: toil. They define it as "the repetitive, predictable, constant stream of tasks related to maintaining a service." Rollouts, upgrades, alert triage, manual repairs, ticket-driven provisioning. Google puts a hard cap on it: no more than 50% of an SRE's time should go to operational work.

The reasoning isn't just about efficiency. The workbook makes a point that has always stuck with me: time spent on toil is time not spent where human judgment, creativity, and design thinking matter.

Here's what I think the SRE Workbook doesn't fully capture, at least not in those exact words. There's a specific kind of toil that doesn't look like toil. It doesn't involve clicking buttons or running the same script for the hundredth time. It's cognitive. It's the mental cost of assembling context from scattered sources before you can make a decision.

Reading a Kubernetes release notes page that's 3,000 words long to find the one deprecation that affects your cluster. Comparing two versions of a Helm values.yaml to understand what changed between chart versions 4.2.1 and 5.0.0. Skimming a Terraform provider changelog to see if the aws_eks_cluster resource changed its default behavior. Correlating an incident timeline from last Thursday with the deployment that happened two hours before the spike in 5xx errors.

This work isn't glamorous. It doesn't produce artifacts. Nobody thanks you for spending an hour reading release notes. But if you skip it, you miss the breaking change that takes down a service at 3 AM on a Sunday.

Sometimes the most exhausting part of an incident is not fixing the issue. It is building enough context to feel safe fixing it.

I think of this as cognitive toil, and AI is unusually well suited to help with it.

Why I don't want an AI agent with production access

Before I talk about what I do want, let me be clear about what I don't.

I don't want an AI agent that has kubectl apply access by default. I don't want one that can merge PRs, push to main, modify IAM policies, or restart services without a human in the loop. I've seen enough production incidents caused by humans who were tired, rushed, or copy-pasting from the wrong terminal. Giving that same power to something that hallucinates API flags and invents Kubernetes resources that don't exist is not progress. It's a new category of incident.

In application code, an AI mistake might fail a test. In DevOps, an AI mistake might page five teams, drain the wrong node, rotate the wrong secret, or turn a small incident into a very educational afternoon.

The Stack Overflow 2025 Developer Survey backs this up. 76% of developers don't plan to use AI for deployment or monitoring tasks. Not because they're luddites. Because they know what's at stake. More developers actively distrust AI accuracy (46%) than trust it (33%). Only 3% highly trust it. That is the part that makes people nervous: AI can sound confident even when the answer still needs careful verification.

In DevOps, "almost right" isn't a minor inconvenience. An "almost right" IAM policy is a security incident. An "almost right" Kubernetes manifest is a workload that runs fine until it doesn't, and then you're debugging at 2 AM wondering why the liveness probe path changed. An "almost right" Terraform plan is a production resource that gets destroyed and recreated instead of updated in place.

The problem is not that AI is useless. The problem is that AI is useful enough to make dangerous workflows look reasonable. In DevOps, the gap between "sounds correct" and "safe to execute" is where incidents live.

The hard part of DevOps is rarely knowing the command. kubectl apply -f manifest.yaml isn't the hard part. The hard part is knowing whether that command is safe in this environment, with this version of Kubernetes, with these admission controllers, with this cluster autoscaler configuration, right after that EKS add-on got updated. That requires context, judgment, and accountability. AI is genuinely useful for the first two, but it can't own the third. Not yet. Maybe not ever.

Most production work is not blocked because nobody knows how to type kubectl. It is blocked because nobody is completely sure what is safe to do next.

What I actually want AI to do

I want AI to be the colleague who actually reads the release notes before standup. The one who highlights the three things that matter out of a forty-seven-paragraph changelog. The one who can look at a Terraform plan diff and tell you, in plain language, what's about to change and what might break.

Concretely, here's what that looks like.

When I'm going from Kubernetes 1.29 to 1.30, I want something that tells me what got deprecated, what changed in API versions, and what I need to act on before upgrading. Skip the boilerplate about "improved performance." Focus on the removals and behavioral changes.

Before I update the VPC CNI add-on, I want to know if this version is compatible with my current Kubernetes version, my node group AMI, and the Calico network policy version we're running. That compatibility matrix is spread across three AWS docs pages and it changes every quarter.

When the AWS Terraform provider goes from 5.x to 6.x, I don't want to read the entire migration guide. I want to know which resources I'm actually using that changed behavior. Focus on my code, not the universe of possibilities.

When I'm upgrading a Helm chart from 4.x to 5.x, show me what changed in the default values: which new keys were introduced, which old keys were removed, which ones changed their default behavior. Better yet, cross-reference my current values.yaml and tell me which of my overrides are now invalid.

If I inherit a cluster with 200 custom resources I've never seen before, help me understand what they do without reading CRD documentation for six hours.

When an incident happens, take the Slack thread, the PagerDuty timeline, and the post-mortem notes, and produce a runbook that the next on-call engineer can actually follow. One that isn't three years stale.

When the error rate spiked at 14:32 and something was deployed at 14:15, pull the deployment diff, the relevant log lines, and the metrics shift into one view so I can see the connection without switching between four tools.

When five services are throwing errors and the logs are a wall of stack traces, filter out the noise, group the unique errors, and tell me which one started first. That's the one I care about.

None of these require production access. None require the AI to execute anything. They require it to read, understand, summarize, compare, and present information so I can decide faster.

The best DevOps AI will not feel magical. It will feel like a senior engineer left clean notes before going on vacation.

The data says this approach works

GitHub's study on Copilot found something interesting beyond speed. 87% of developers said AI helped them preserve mental effort during repetitive tasks. 73% said it helped them stay in flow. 60-75% said it helped them focus on more satisfying work. One senior engineer put it simply: with AI, they had to think less about the boring stuff, and when they had to think, it was the fun stuff.

The DORA research on generative AI adds an important nuance. Developers who use AI extensively report higher job satisfaction, more time in flow state, and less burnout. But there's a catch: AI adoption didn't reduce time spent on toilsome, repetitive tasks. It sped up the valuable work developers already enjoyed, but didn't crack the code on automating drudgery. DORA also found that a 25% increase in AI adoption was associated with a decrease in delivery stability, because AI lets teams generate more code and more changes faster than their review and testing processes can handle.

Read that last sentence again. AI doesn't hurt stability because it writes bad code. It hurts stability because it lets teams produce more work than their feedback loops can safely absorb.

This is exactly why the read-summarize-suggest model is the right one for DevOps. It gives engineers better context without adding unreviewed changes to the pipeline. It accelerates understanding without bypassing approval. It reduces the time between "I need to figure this out" and "I understand enough to decide" without collapsing the distance between "I decided" and "it's done."

A boundary that matters

I'm not anti-agent. I think autonomous AI agents will eventually have a role in infrastructure operations. But the keyword is eventually, and the prerequisite is trust, and trust is earned slowly and lost quickly.

Stack Overflow also shows developers are much more cautious with high-responsibility work. Most respondents do not plan to use AI for deployment or monitoring. These are not people who hate AI. These are people who know where the blast radius lives.

The DORA report reinforces this: trust directly drives AI productivity. Developers who trust AI accept more suggestions, submit more changes, and spend less time searching for information. But DORA also found that 39% of developers still trust AI outputs "a little" or "not at all."

In DevOps, trust isn't about vibes. It's about being right when being wrong has consequences. An AI that summarizes a changelog and misses a breaking change is annoying but survivable. An AI that applies a change based on that incomplete summary is a production incident.

The line I draw is simple. AI should read, summarize, compare, draft, and suggest. Humans should approve, execute, and own.

Let AI read. Let AI summarize. Let AI compare. Let AI draft. Let AI suggest.

But make humans approve, execute, and own.

The fatigue I want to replace

DevOps has a burnout problem. This isn't news. The on-call rotations, the incident pressure, the constant context switching between ten different tools and three different cloud providers and a pile of documentation that's always slightly out of date.

The fatigue is real. It accumulates. It's not the dramatic kind where someone screams and quits. It's the quiet kind where you stop reading the full changelog because you've read forty of them and nothing ever breaks, until one Tuesday it does. Where you stop updating the runbook because nobody reads it anyway, including you. Where you start copy-pasting Terraform modules from the last project because you don't have the energy to check if the AWS provider changed the defaults again.

AI can't fix organizational dysfunction. It can't fix understaffed on-call rotations or unreasonable SLAs. But it can reduce the cognitive tax of the work that sits between "I got paged" and "I understand what is happening." It can give you back the thirty minutes you'd have spent re-reading docs you already read once. It can catch the breaking change you'd have missed at 2 AM.

I don't want AI to replace DevOps engineers. I want it to replace the exhaustion that makes us worse at the job we're good at. I want it to be the thing that reads the docs so I can focus on deciding what to do with what they say. I want it to handle the reading so I can handle the thinking.

That's not a smaller vision. It's a more honest one.

References:

Google SRE Workbook, "Eliminating Toil": sre.google/workbook/eliminating-toil/
DORA, "Impact of Generative AI in Software Development": dora.dev/ai/gen-ai-report/
Stack Overflow Developer Survey 2025, AI section: survey.stackoverflow.co/2025/ai/
GitHub Research, "Quantifying GitHub Copilot's Impact on Developer Productivity and Happiness": github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

DevOps to Platform Engineer: The Career Shift Nobody Explains Properly

Pawan Kumar — Thu, 30 Apr 2026 20:15:33 +0000

If you've been in DevOps long enough, you've probably seen the job postings by now. "Platform Engineer." "Internal Developer Platform." "Platform-as-a-Product." The titles are everywhere. Gartner says 80% of large engineering organizations will have dedicated platform teams by 2026. That's up from 45% in 2022.

But nobody really explains what changes. Not the buzzwords. The actual day job. The skills. The salary. The headaches.

I work as a DevOps Engineer at a company that builds Kubernetes application platforms. So I'm living in the middle of this transition every single day. Let me break down what's actually happening, what it means for your career, and whether you should care.

What's Actually Happening

Here's the short version: DevOps broke at scale. Not the philosophy. The practice.

When you have 5 teams and 20 services, DevOps works beautifully. Everyone knows everyone. You can walk over to someone's desk (or Slack them) and figure out why the pipeline broke. The "culture of collaboration" actually functions.

But at 50 teams? 500 services? Multiple clouds? That informal shared context collapses. Onboarding takes weeks instead of days. Every team builds slightly different CI/CD pipelines. Security reviews become bottlenecks. And you end up with 3 senior engineers who "know how things really work," and they're drowning.

Platform engineering is the response to that breakdown. Instead of relying on culture and tribal knowledge, you build a product, an Internal Developer Platform (IDP), that encodes best practices into self-service tooling.

The platform becomes the documentation. The guardrails become the governance. And the paved road becomes the easiest road.

DevOps vs Platform Engineering: The Real Differences

Let's skip the marketing fluff. Here's what actually changes:

Aspect	DevOps	Platform Engineering
Who you build for	Infrastructure, pipelines	Developers (your users)
How work comes to you	Tickets, Slack pings, "can you help me"	Platform feature requests, adoption metrics
Governance	PR reviews, approval gates, manual checks	Embedded into templates and workflows
Success metric	"Did the deploy work?"	"Are developers choosing to use the platform?"
Scale model	Linear (more teams = more DevOps)	Leverage (platform scales once, serves all)
Your mindset	"Let me fix this for you"	"Let me build it so you never need to ask"

That last row is the fundamental shift. DevOps is a service mindset. Platform engineering is a product mindset.

A Day in the Life

DevOps Engineer's typical day:

Build and maintain CI/CD pipelines (30%)
Write Terraform, manage infrastructure (25%)
Set up monitoring and alerting (15%)
Automate deployment processes (20%)
Help developers with infrastructure issues (10%)

Platform Engineer's typical day:

Build internal developer tools and abstractions (35%)
Improve self-service capabilities (25%)
Maintain the platform infrastructure itself (20%)
Developer support, education, onboarding (10%)
Platform documentation (10%)

Notice the shift: you're spending more time building things for developers to use independently and less time doing things for developers. It's the difference between being a chef and being someone who designs kitchen layouts.

The Salary Question (India-Focused)

Let's talk numbers. I cross-referenced data from AmbitionBox, Glassdoor, Levels.fyi, and real job postings across LinkedIn and Naukri. Here is the realistic range for India in 2026:

Experience	DevOps	Platform Engineer
3-5 years	₹10-28 LPA	₹20-40 LPA
6-10 years	₹20-45 LPA	₹35-60 LPA
Lead/Principal	₹35-65 LPA	₹55-90 LPA

Platform engineering commands a 30-60% premium over generalist DevOps, according to multiple 2026 India salary reports. The premium exists because the talent pool is much smaller, you need DevOps foundations plus product thinking plus software engineering depth.

Globally, platform engineers in North America average $160,000 USD, compared to DevOps roles that typically plateau around $140K. Not life-changing, but meaningful.

The Skills Gap: What You Need to Learn

If you're a DevOps engineer today, you already have most of the foundations. Here's what's missing:

1. Product Thinking

This is the biggest mindset shift. You're no longer building pipelines, you're building a product with users, feedback loops, and adoption metrics. That means:

Understanding developer pain points (user research)
Prioritizing features based on impact (product management)
Measuring adoption, not just uptime (analytics)
Iterating based on feedback (continuous improvement)

The #1 reason platform initiatives fail? Teams build technically excellent platforms that nobody uses. Voluntary adoption is the real metric.

2. API Design and Software Engineering

DevOps scripting (Bash, YAML, a bit of Python) doesn't cut it anymore. Platform engineers need:

API design - Your platform is consumed through APIs
Go or Rust - Most CNCF platform tooling is written in Go
Multi-tenancy patterns - Your platform serves multiple teams with different needs
Software engineering practices - Testing, versioning, deprecation strategies

# Example: A Golden Path template for a new microservice
# This is what platform engineers build - opinionated defaults
apiVersion: backstage.io/v1alpha1
kind: Template
metadata:
  name: microservice-template
  title: Standard Microservice
  description: Spin up a new Go microservice with CI/CD, monitoring, and security baked in
spec:
  parameters:
    - title: Service Details
      required:
        - name
        - team
      properties:
        name:
          title: Service Name
          type: string
        team:
          title: Owning Team
          type: string
          enum: [payments, auth, core, platform]
  steps:
    - id: scaffold
      name: Generate Service
      action: fetch:template
      input:
        url: ./templates/go-microservice
        values:
          name: ${{ parameters.name }}
          team: ${{ parameters.team }}

This is a simplified Backstage software template - one of the most common patterns in platform engineering. Developers fill in a few fields, and the platform generates a production-ready service with CI/CD, observability, and security pre-configured.

You can achieve the same with Devtron's Application Templates - capture CI/CD workflows, build configs, deployment templates, and environment overrides from an existing app, then reuse them to spin up new microservices in minutes instead of hours.

3. Developer Experience (DevEx)

You need to care about how developers feel using your platform. This includes:

Time to first deploy (how fast can a new dev ship?)
Self-service capabilities (can they do it without filing a ticket?)
Documentation quality (can they figure it out without asking you?)
Error messages (are they helpful or cryptic?)

The State of Platform Engineering Report recommends tracking DORA metrics (deployment frequency, lead time, change failure rate, MTTR) alongside SPACE metrics (developer productivity) and time-to-onboarding.

4. AI Literacy

This isn't optional anymore. 92% of CIOs plan AI integrations into their platforms. The recommendation is to reserve 20% of your time for AI skill development:

Using AI tools for platform operations (K8sGPT, AI-assisted troubleshooting)
Building AI-powered capabilities into your platform (intelligent autoscaling, anomaly detection)
Understanding how AI-generated code flows through your CI/CD

By 2028, platforms without AI capabilities will be considered outdated.

How to Actually Make the Transition

Here's a practical roadmap, assuming you have 3+ years of DevOps experience:

Month 1-2: Build Product Thinking

Read "Team Topologies" by Matthew Skelton and Manuel Pais
Start treating your current internal tools as products - add documentation, gather feedback, track usage
Learn about Backstage (CNCF project, 89% market share for IDPs)
Explore Devtron - an AI-native Kubernetes management platform to see how real IDPs work in practice

Month 3-4: Level Up Software Engineering

Pick up Go if you haven't already - most platform tooling is Go-based
Build a small internal tool with proper API design, tests, and documentation
Contribute to an open-source platform tool (Backstage, Crossplane, Port)

Month 5-6: Get Hands-On with IDPs

Deploy Backstage locally or in a sandbox cluster
Build a software template for your team's most common workflow
Add golden paths for your existing infrastructure patterns

Ongoing: Develop AI Competency

Experiment with K8sGPT for cluster troubleshooting
Explore AI-assisted CI/CD (GitHub Copilot in Actions, AI-powered code review)
Stay current with AI SRE tools (autonomous incident response is coming fast)

The Six Specialized Roles Within Platform Engineering

As the field matures, "platform engineer" is splitting into distinct specializations:

Head of Platform Engineering (HOPE) - Strategic direction, cross-functional coordination
Platform Product Manager (PPM) - Bridges technical teams and organizational needs
Infrastructure Platform Engineer (IPE) - Underlying infra (servers, networks, databases)
DevEx Platform Engineer (DPE) - Developer workflows, friction reduction, tool UX
Security Platform Engineer (SPE) - Security embedded into pipelines, policy-as-code
Reliability Platform Engineer (RPE) - Evolution of SRE, monitoring/observability plane

You don't need to pick one immediately. Most platform engineers touch multiple areas, especially in smaller teams. But knowing these exist helps you see where your career can go.

What I'm Seeing From the Inside

Working at Devtron, a company that literally builds a Kubernetes application platform, I get a front-row seat to this transition. Here's what I see daily:

Teams that adopted platform thinking are shipping faster with fewer incidents. They're not firefighting as much because the platform catches common mistakes before they reach production.

Teams that didn't are drowning in tickets. Every new microservice means another pipeline to build, another set of alerts to configure, another on-call rotation to manage. It doesn't scale.

The companies that get this right treat their platform as a product with a dedicated team, clear ownership, and actual user research. The ones that get it wrong rebrand their DevOps team as "Platform Engineering" and change nothing about how they work.

Don't be the second one.

The Honest Take

Platform engineering isn't replacing DevOps. It's DevOps growing up. The philosophy of collaboration, automation, and shared responsibility stays. What changes is the mechanism, from culture-dependent to platform-dependent.

Should you make the shift? If you enjoy building tools more than operating infrastructure, if you care about developer experience, and if you want to work on leverage (building something once that serves hundreds of developers) - yes.

The timing is right. Mid-level engineers with 3-5 years of experience are entering platform roles in growing numbers. You don't need to be a senior architect anymore. The field is democratizing, the salaries are competitive, and the demand is only going up.

Start by building one thing that removes friction for your team. Treat it like a product. See what happens.

Further Reading:

Team Topologies - the org design book behind platform thinking
Backstage.io - CNCF project for building developer portals
Devtron - AI-Native Kubernetes Management Platform
Platform Engineering community - reports, articles, and the annual State of Platform Engineering survey
DORA metrics - the standard for measuring software delivery performance