DEV Community: Daya Shankar

Best Managed Kubernetes Hosting in India in 2026

Daya Shankar — Wed, 17 Jun 2026 10:49:47 +0000

Most teams comparing managed Kubernetes hosting start with one question:

Which provider is the best?

But that depends on what you are trying to run.

A large enterprise may need deep IAM controls, multi-region recovery and integration with hundreds of cloud services.

A startup may care more about simple pricing, local support and a free control plane.

So instead of putting every provider into one list, it makes more sense to compare them by category.

First, the hyperscalers.

Then, India-based cloud providers.

And finally, global providers with Indian regions.

Managed Kubernetes Providers in India at a Glance

Provider	Category	Best suited for
Amazon EKS	Hyperscaler	AWS-based enterprise environments
Google Kubernetes Engine	Hyperscaler	Kubernetes automation and cloud-native teams
Azure Kubernetes Service	Hyperscaler	Microsoft and hybrid environments
Oracle Kubernetes Engine	Global cloud provider	Oracle and OCI workloads
AceCloud	India-based provider	Managed GPU Kubernetes and local support
Utho	India-based provider	Startups and cost-conscious teams
OVHcloud	Global provider with India region	Open cloud and network-heavy workloads
DigitalOcean Kubernetes	Global provider with India region	Simple developer-focused deployments

Best Hyperscaler Kubernetes Platforms in India

1. Amazon EKS: Best for AWS Environments

Amazon Elastic Kubernetes Service is usually the obvious choice when applications already use EC2, IAM, RDS, CloudWatch or other AWS services.

AWS manages the Kubernetes control plane, while teams can choose managed node groups, Fargate or EKS Auto Mode for worker infrastructure.

The biggest advantage is not EKS alone.

It is everything around EKS.

Networking, security, storage, databases and observability can remain inside the same cloud ecosystem.

But that flexibility also creates complexity. Control-plane charges, compute, NAT gateways, load balancers, storage and monitoring are billed separately.

Choose EKS when: your organisation already runs on AWS and needs deep enterprise integrations.

2. Google Kubernetes Engine: Best for Automation

Google Kubernetes Engine is one of the most mature managed Kubernetes platforms.

Standard mode gives infrastructure teams more control over nodes and cluster settings.

Autopilot manages more of the infrastructure, including node provisioning and scaling.

This makes GKE useful for teams that want Kubernetes flexibility without managing every underlying component.

Its strengths include release channels, workload identity, autoscaling and integration with Google Cloud’s data and AI services.

Choose GKE when: automation and a Kubernetes-first developer experience matter most.

3. Azure Kubernetes Service: Best for Microsoft Teams

Azure Kubernetes Service fits naturally into organisations using Microsoft Entra ID, Azure DevOps, Azure Monitor or Windows-based applications.

Its biggest advantage is ecosystem alignment.

Identity, policies, monitoring and CI/CD can remain connected to the Microsoft tools the organisation already uses.

AKS is also relevant for hybrid environments through Azure Arc.

Choose AKS when: Microsoft identity, development tools and hybrid infrastructure already shape your environment.

Best India-Based Managed Kubernetes Providers

4. AceCloud: Best for GPU Kubernetes and Managed Support

AceCloud Managed Kubernetes is designed for teams that want managed clusters without the operational overhead of running the control plane themselves.

Its offering includes a high-availability control plane, autoscaling, managed upgrades, monitoring and support.

The main differentiator is GPU infrastructure.

Teams can create GPU-enabled node groups for AI training, inference, analytics and other compute-heavy workloads.

This makes it relevant for Indian AI startups and enterprises that want Kubernetes and GPU capacity from the same provider.

Choose AceCloud when: GPU workloads, managed operations and local support are important.

5. Utho: Best for Cost-Conscious Indian Teams

Utho Managed Kubernetes focuses on simple deployment and transparent infrastructure pricing.

It offers a managed control plane, autoscaling, storage integrations and cloud infrastructure hosted in India.

Its smaller ecosystem can be an advantage for teams that do not want to navigate dozens of managed services before deploying a cluster.

But teams running complex enterprise workloads should compare support, integrations and service maturity carefully.

Choose Utho when: you want an Indian provider with straightforward pricing and simpler cluster management.

Other Global Providers with India Regions

6. OVHcloud: Best for Open Cloud Infrastructure

OVHcloud Managed Kubernetes is available through its Mumbai Public Cloud region.

The platform supports autoscaling, Terraform, managed load balancing and Cilium-based networking.

OVHcloud also positions its service around open standards and predictable networking costs.

It does not offer the same service breadth as AWS, Azure or Google Cloud.

But for teams trying to reduce dependency on a large hyperscaler, that may be a reasonable trade-off.

Choose OVHcloud when: open cloud infrastructure, Mumbai hosting and predictable networking matter.

7. DigitalOcean Kubernetes: Best for Simplicity

DigitalOcean Kubernetes is built for developers and smaller engineering teams that want to deploy clusters without a steep cloud-management learning curve.

It includes a managed control plane, autoscaling and integration with DigitalOcean storage, load balancers and container registries.

The service is available in Bangalore.

It offers fewer enterprise services than the hyperscalers, but its simpler platform can make deployments easier to understand and operate.

Choose DigitalOcean when: developer experience and fast deployment matter more than enterprise service breadth.

8. Oracle Kubernetes Engine: Best for OCI Workloads

Oracle Kubernetes Engine is most relevant when applications already depend on Oracle Database, Exadata or other OCI services.

OKE supports managed nodes, virtual nodes, autoscaling and OCI networking and security integrations.

It becomes a practical choice when Kubernetes is part of a broader Oracle architecture.

Choose OKE when: Oracle services already sit at the centre of your workload.

What Should You Compare Before Choosing?

Do not compare providers only on worker-node prices.

Also check:

Control-plane fees and SLA
Indian data-location requirements
Load balancer and public IP costs
Storage, snapshots and backup pricing
Network egress charges
GPU node availability
Upgrade and patching responsibility
Support response times

A cheap worker node can still produce an expensive cluster.

Which Managed Kubernetes Provider Should You Choose?

Choose EKS, GKE or AKS when you need the scale and ecosystem of a hyperscaler.

Choose AceCloud or Utho when local support, Indian infrastructure or simpler billing matters more.

Choose OVHcloud or DigitalOcean when you want a global platform with Indian hosting but less hyperscaler complexity.

Choose OKE when Oracle services already shape your architecture.

The best provider is not the one with the longest feature list.

It is the one that removes the Kubernetes work your team does not want to manage without taking away the control it still needs.

Stable Diffusion Inference: Memory Requirements, Speed and GPU Selection

Daya Shankar — Wed, 17 Jun 2026 10:16:44 +0000

When teams plan infrastructure for Stable Diffusion, the conversation usually starts with GPU speed.

Should we use an L40S?

Would an H100 generate images faster?

How many images can it produce per minute?

Those are useful questions.

But they often skip the constraint that decides whether the workload can run properly in the first place:

GPU memory.

A model may generate one image quickly during testing and still struggle in production.

Why?

Because production adds larger resolutions, concurrent requests, ControlNet models, LoRA adapters and multiple pipeline components competing for the same VRAM.

What looks like a slow GPU can actually be a workload that no longer fits comfortably in memory.

What Actually Uses VRAM During Stable Diffusion Inference?

The model weights are only part of the memory requirement.

VRAM is also used by:

The UNet or diffusion transformer
Text encoders
The VAE
Latent representations and intermediate tensors
Attention operations
Image buffers
Batch and concurrency overhead
ControlNet models and other adapters

Resolution matters because larger images create larger latent tensors and attention workloads.

Batch size matters because the GPU must process more images at the same time.

Concurrency matters because multiple active requests may require their own intermediate data.

And ControlNet matters because its weights and activations add another model component to the pipeline. The official Diffusers ControlNet documentation explains how these additional conditioning models work alongside the base diffusion model.

How Much VRAM Does Stable Diffusion Need?

There is no single correct number.

The requirement changes with the model, resolution, precision, framework, attention backend, batch size and memory optimisations.

Still, the following ranges provide a practical starting point for planning:

Deployment scenario	Practical VRAM range
Stable Diffusion 1.x at 512×512	6-8 GB
SDXL base at 1024×1024	12-16 GB
SDXL with LoRA or a light extended workflow	16-24 GB
SDXL with ControlNet or multiple pipeline components	20-24 GB or more
Concurrent production inference	24-48 GB or more

These are planning ranges, not fixed minimums. Memory-saving techniques can reduce VRAM use, while larger batches, multiple ControlNets and concurrent requests can push requirements higher.

For example, SDXL can run on lower-memory hardware by moving pipeline components to system memory. Hugging Face documents several options in its Diffusers memory optimisation guide.

But there is a trade-off.

CPU offloading saves VRAM by moving model components between the CPU and GPU. That movement can also increase generation time.

So fitting the model into memory and running it efficiently are not always the same thing.

Does More VRAM Make Stable Diffusion Faster?

Not directly.

This is where GPU selection often becomes confusing.

VRAM capacity determines whether the model, batch and active requests fit on the GPU.

Once they fit, generation speed depends more heavily on:

GPU compute performance
Memory bandwidth
Image resolution
Number of sampling steps
Batch size
Precision such as FP16, BF16 or FP8
Inference framework and kernel optimisations

Imagine two GPUs that finish one SDXL image in a similar amount of time.

One has 24 GB of VRAM. The other has 48 GB.

The 48 GB GPU may not generate that single image twice as fast.

But it may support larger batches, more complex pipelines or more concurrent requests before running out of memory.

That is the real value of additional VRAM.

It creates headroom.

Why Does Performance Change Under Concurrent Load?

A single-image benchmark answers one question:

How quickly can this GPU complete one controlled request?

A production service asks something different:

How many requests can it complete while keeping latency within an acceptable range?

Suppose one user generates a 1024×1024 image.

The GPU may look fast and lightly loaded.

Now add ten users, different LoRA adapters and a ControlNet workflow.

The hardware has not changed.

The memory requirement has.

When the workload no longer fits comfortably, the deployment may need to reduce batch size, queue requests, offload components to the CPU or reject requests with an out-of-memory error.

This is why production performance often looks very different from a benchmark.

How Does Batch Size Affect Memory and Speed?

Batching allows the GPU to process multiple prompts together.

This can improve throughput because the GPU does more work in each execution cycle.

But a larger batch also requires more VRAM.

The official Diffusers batch inference guide describes the same trade-off: batching can improve GPU utilisation, but it increases memory use and may increase latency.

So the largest possible batch is not automatically the best batch.

A batch service may prioritise maximum images per minute.

An interactive application may use smaller batches because users care more about how quickly each request starts and finishes.

Which GPU Should You Choose for Stable Diffusion?

There is no universal “best GPU.”

The better choice depends on whether you are optimising for single-user development, cost-efficient inference, concurrency or large shared environments.

Deployment goal	Suitable GPU class	Why it fits
Development and testing	16-24 GB GPU	Enough for common single-user SDXL workflows with sensible optimisation
Professional workstation workflows	RTX 6000 Ada, 48 GB	Large VRAM pool for complex local workflows and multiple extensions
Production image inference	L4, 24 GB or L40S, 48 GB	L4 suits lighter serving, while L40S adds headroom for larger batches and concurrency
High-concurrency inference	H100, 80 GB or 94 GB	Higher compute, bandwidth and memory capacity for demanding serving environments
Memory-heavy shared environments	H200, 141 GB	Large HBM3e capacity for high concurrency and larger mixed AI workloads

The L40S provides 48 GB of GDDR6 memory, while the H200 provides 141 GB of HBM3e. Those specifications are useful, but they do not mean every Stable Diffusion deployment should move directly to an H200.

For standard SDXL inference, an H200 may be unnecessary unless the environment also needs substantial concurrency, large batches or broader memory-heavy AI workloads.

Buying more headroom than the workload can use does not improve efficiency.

What Should You Measure Before Selecting a GPU?

Do not test only one image.

Test the workload you actually expect to operate.

Measure:

Peak VRAM usage
Average and p95 generation latency
Images generated per minute
Maximum stable concurrency
Queue length during peak traffic
Failure and out-of-memory rates
Cost per completed image

Use the same model, resolution, sampling steps, adapters and concurrency level expected in production.

Otherwise, the benchmark may tell you which GPU wins a test without telling you which GPU fits the deployment.

The Infrastructure Mistake Most Teams Make

The common mistake is selecting a GPU first and defining the workload later.

Teams see that an H100 is faster than an L4 and assume it must be the better choice.

But faster hardware only creates value when the workload uses that performance.

A low-volume internal tool may run efficiently on a 24 GB GPU.

A public image platform may need 48 GB or more because many users are generating images at once.

A large batch pipeline may care less about individual request latency and more about total images per GPU hour.

Same model.

Different operating conditions.

Different GPU decision.

Memory, Speed and Cost Are Connected

Stable Diffusion infrastructure is not only a GPU performance problem.

It is a resource-balancing problem.

VRAM capacity determines what can fit and how much work can run together.

Compute performance and memory bandwidth affect how quickly that work completes.

Concurrency and response-time targets determine how much spare capacity the deployment needs.

And all of those decisions affect cost.

So before asking, “Which GPU is fastest?” ask something more useful:

What must this GPU handle at the busiest point of the day?

That answer will tell you far more than a single-image benchmark.

Batch Processing vs Real-Time Inference: When to Use Each for Image Generation

Daya Shankar — Wed, 17 Jun 2026 10:01:18 +0000

Two companies use the same image generation model.

One needs 100,000 product images for an e-commerce catalogue. The other runs a design platform where users expect an image within seconds.

Same model. Possibly the same GPUs.

Completely different infrastructure.

Why?

Because one company needs the images completed. The other has users waiting for them.

Most teams begin by comparing models, inference frameworks and GPU specifications. Those choices matter, but another question often has a bigger effect on cost and GPU utilisation:

Does the image need to exist now, or can it be generated later?

The answer usually determines whether batch processing, real-time inference or a combination of both is the right approach.

Batch Processing vs Real-Time Inference at a Glance

Factor	Batch Processing	Real-Time Inference
Primary goal	Maximum throughput	Fast response time
User waiting	No	Yes
Queueing	Expected	Kept within a latency limit
GPU utilisation	Usually easier to maximise	Often requires spare capacity
Capacity planning	Based on job volume and deadlines	Based on traffic and latency targets
Cost priority	Lower cost per completed image	Consistent user experience
Infrastructure priority	Efficiency	Availability

The difference may look operational.

In reality, it shapes the entire deployment architecture.

When Does Batch Processing Make Sense?

Batch processing treats image generation as work that must be completed, not as a service that must respond immediately.

It works well for:

Product catalogue generation
Bulk image enhancement
Marketing asset production
Media rendering pipelines
Large-scale design automation

In these cases, the business cares about total output and delivery time. It does not usually matter whether every image appears seconds after the request.

That flexibility is useful.

Requests can wait in a queue. Compatible jobs can be grouped together. GPUs can continue processing without keeping capacity available for unpredictable user traffic.

The goal is simple:

Keep the GPU busy and complete as much work as possible.

Think of it like filling a delivery truck. When the delivery is not urgent, sending a full truck is more efficient than making several half-empty trips.

Batch image generation follows the same principle.

Technologies such as NVIDIA Triton dynamic batching can combine compatible inference requests into larger batches to improve throughput.

Here, the queue is not necessarily a bottleneck.

It is part of the optimisation strategy.

Why Can Batch Processing Cost Less?

Batch workloads give teams more control over when and how GPU capacity is used.

They can group similar requests, schedule jobs during available capacity and process work continuously for longer periods.

This can increase the number of images completed per GPU hour and reduce the effective cost per image.

But batching is not automatic magic.

It works best when requests use compatible settings such as the same model, resolution or inference configuration. Highly varied requests may require separate queues or scheduling rules.

Speed still matters, but the metric changes.

A batch pipeline may take several hours to generate 100,000 images. If the output is ready before the business deadline, it has done exactly what it was designed to do.

When Does Real-Time Inference Make Sense?

Now imagine a user entering a prompt and clicking Generate Image.

They are not thinking about GPU utilisation.

They are watching the loading screen.

The infrastructure must have capacity available when the request arrives. It cannot comfortably hold every request for several minutes while waiting to build a larger batch.

Every extra second becomes part of the product experience.

This makes real-time inference suitable for:

Interactive image generation tools
AI design platforms
Live photo-editing applications
Customer-facing content creation tools
Applications with strict response-time targets

Real-time infrastructure may need spare GPU capacity during quieter periods so it can handle sudden traffic increases.

From an infrastructure perspective, that capacity may look underused.

From a product perspective, it protects the user experience.

Does Real-Time Inference Mean No Batching?

No.

This is an important distinction.

Real-time systems can still use small or dynamic batches. The difference is that requests can only wait for a limited time.

For example, an inference server may hold a request for a few milliseconds to see whether another compatible request arrives. It can then process both together without creating a noticeable delay.

But here is the trade-off.

The longer the system waits to create a batch, the more throughput it may gain. It also adds more latency.

NVIDIA’s Triton optimisation guidance treats minimum latency and maximum throughput as different tuning goals. You rarely maximise both at the same time.

The Real Trade-Off: Utilisation vs Responsiveness

Many techniques that improve batch efficiency can make interactive applications feel slower.

Larger queues can improve throughput but increase waiting time.
Higher utilisation can lower idle capacity but leave less room for traffic spikes.
Aggressive scheduling can keep GPUs busy but delay interactive requests.

What looks like optimisation in a batch environment can become a bottleneck in a real-time one.

In batch processing, waiting can improve efficiency.

In real-time inference, waiting affects the customer experience.

Which Processing Model Should You Choose?

Ask one question:

What happens if the image arrives ten minutes later?

If the answer is “nothing important,” batch processing is probably the better choice.

If the delay interrupts a workflow or frustrates a waiting user, real-time inference may be justified.

Choose Batch Processing When:

No user is actively waiting for each image
The workload contains many similar requests
Images must meet a deadline rather than appear immediately
Cost per image matters more than individual request latency
Jobs can tolerate queueing or rescheduling

Choose Real-Time Inference When:

A customer is waiting for the result
Response time affects the product experience
Requests arrive unpredictably
The application has a clear latency target
Slow generation could cause users to abandon the workflow

What If Your Workload Needs Both?

Many production applications use a hybrid architecture.

Interactive requests go to infrastructure designed for low latency. Bulk tasks move to a queue and run on capacity optimised for throughput.

For example, a design platform may generate a preview in real time. Once the user approves it, high-resolution exports, different aspect ratios and additional variations can move to a batch pipeline.

The user gets a fast preview.

The infrastructure avoids treating every output as urgent.

Why Workload Behaviour Matters More Than GPU Size

Teams often begin by asking which GPU they should use.

But the fastest GPU does not automatically create the most cost-effective architecture.

A powerful GPU running at low utilisation in an oversized real-time environment may cost more per image than a smaller GPU running continuously in a batch pipeline.

The hardware matters.

But workload behaviour determines how efficiently that hardware is used.

Before selecting an GPU instance, define whether the workload needs maximum throughput, low latency or a balance of both.

You can then compare hourly and longer-term configurations through cloud GPU pricing instead of keeping unnecessary capacity active.

So, Who Is Waiting for the Image?

Choose batch processing when completion matters more than immediate delivery.

Choose real-time inference when the user experience depends on receiving the image quickly.

Use a hybrid architecture when only part of the workflow needs an instant response.

Before comparing GPUs or benchmarking inference frameworks, ask:

Who is waiting for the image?

If nobody is waiting, let the workload queue.

If a user is watching the screen, design the infrastructure around that moment.

Cold Starts, Model Loading, and Their Impact on Latency SLAs

Daya Shankar — Mon, 02 Mar 2026 06:37:05 +0000

Cold start latency breaks SLAs because “pod is Running” isn’t “model is ready.” In Kubernetes hookup with vLLM, cold start includes image pulls, weight downloads, tensor load into GPU memory, and warm-up work (often CUDA-graph-related). These events are rare but huge, so they dominate p95/p99—especially when you scale from zero.

The on-call version of this problem

Bridge: SLAs die on tails, and cold starts are tail generators.

You deploy a new vLLM revision. HPA scales up. Pods come up fast. Traffic shifts. p50 looks fine. p99 explodes.

Nothing “crashed.” You just routed users onto instances still doing model loading and warm-up. That’s not a bug. That’s physics plus orchestration.

If you run strict SLAs on a GPU fleet, you need to treat cold start like a first-class SLI.

What “cold start” actually contains for vLLM on Kubernetes

Bridge: Break the chain into phases so you can measure and fix the slowest link.

Cold start is not one thing. It’s a pipeline:

DIAGRAM 1 — Cold start timeline (what you must budget)

Scale event
|
v
[1] Image pull ---> [2] Container start ---> [3] Model fetch ---> [4] Weight load ---> [5] Warm-up ---> Ready
| | | | |
Registry Python init Network/FS Disk->RAM->GPU Graph/caches

The phase that usually dominates: model storage path

Bridge: If weights sit on slow shared storage, everything else is noise.

vLLM calls this out bluntly: loading large models from shared/network filesystems can be slow, and it’s better to store the model on local disk when possible. It also warns that CPU memory pressure can trigger swapping and slow the OS down.

Translation:

If your weights live on a slow network filesystem, you built a cold-start machine.

If you swap while loading weights, you built a cold-start machine that also hurts neighbors.

Warm-up is real work, not “nice to have”

Bridge: If you don’t pre-warm, the first user request becomes your warm-up job.

vLLM provides tooling specifically to benchmark cold vs warm startup, including model loading and compilation/cache operations.
If vLLM ships a benchmark for startup, that’s your sign: startup cost matters.

Why L40S changes the tuning you should do

Bridge: PCIe-only GPUs expose bad data paths immediately.

On NVIDIA L40S, you’re on PCIe Gen4 x16 (64GB/s bidirectional).
Also: NVLink: no and MIG: no.

What this means for cold starts:

Host↔GPU traffic rides PCIe. Extra copies hurt.

You can’t “hide” cold starts by slicing a big GPU into tiny always-warm MIG slices.

Your operational levers are boring: caching, warm replicas, and reducing churn.

SLA math: cold starts don’truin averages, they ruin p95/p99

Bridge: You can’t hand-wave tails with “but it’s rare.”

Cold starts are low-frequency, high-impact latency events. That’s exactly what percentiles punish.

If you allow scale-to-zero, your probability of cold starts after idle becomes close to 1 for the first request. Knative documents scale-to-zero as a feature and exposes config to enable/disable it.
Knative also documents Scale Down Delay specifically to keep containers around for a configurable time to avoid cold start penalties.

Even if you don’t use Knative, the principle holds:

Every time you delete a pod, you re-pay model load.

Every time you scale to zero, you guarantee a cold start on the next burst.

Fix cold start latency by attacking three things

Bridge: You reduce cold starts by moving fewer bytes, repeating less work, and avoiding scale-to-zero surprises.

1) Cache model artifacts on the node (prefer local disk)

Bridge: Put the bytes next to the GPU node or pay for network + FS latency on every churn event.

vLLM recommends local disk when shared filesystems are slow.
So do this:

Mirror model artifacts to a controlled location (object store, internal registry, or artifact repo).

Cache on node-local SSD/NVMe where possible.

Point vLLM/HF cache directories at that local path.

Practical rule for SREs: don’t download weights from the public internet in the hot path. vLLM itself recommends downloading first (via huggingface-cli) and passing the local path to isolate issues.

2) Pre-pull images on GPU nodes

Bridge: Image pulls are pure waste during a scale event.

Use a DaemonSet that pins to GPU nodes and pulls your serving image. Keep it dumb.

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: vllm-prepull
namespace: kube-system
spec:
selector:
matchLabels: { app: vllm-prepull }
template:
metadata:
labels: { app: vllm-prepull }
spec:
nodeSelector:
accelerator: nvidia
tolerations:
- key: "accelerator"
operator: "Equal"
value: "nvidia"
effect: "NoSchedule"
containers:
- name: sleep
image: your-registry/vllm-serving:TAG
command: ["sh","-c","sleep 365000"]
resources:
requests: { cpu: "10m", memory: "32Mi" }
limits: { cpu: "50m", memory: "64Mi" }

3) Keep a warm floor for SLA-bound services

Bridge: If your SLA can’t tolerate cold starts, don’t scale to zero.

Set:

min replicas > 0 (HPA floor or Deployment replicas)

a “warm pool” per model

separate burst capacity if you need it

Scale-to-zero is a cost tool. It is not an SLA tool. Knative’s own docs bake in knobs to control that behavior for a reason.

Two diagrams that match how this should be deployed

Bridge: These are the shapes that keep p99 sane without paying for always-on waste.

Reference deployment YAML (vLLM on L40S with readiness gating + node-local cache)

Bridge: This is the “copy/paste then edit” block you can review in PRs.

This example does three things:

pins onto GPU nodes

caches model files on a node-local path

gates readiness until a warm-up completes

Important: This assumes you control the container entrypoint and can add a small wrapper script. That’s the cleanest way to tie readiness to “model is hot.”

apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-serve
namespace: inference
spec:
replicas: 2 # warm floor for SLA
selector:
matchLabels:
app: vllm-serve
template:
metadata:
labels:
app: vllm-serve
spec:
nodeSelector:
accelerator: nvidia
gpu: l40s
tolerations:
- key: "accelerator"
operator: "Equal"
value: "nvidia"
effect: "NoSchedule"
containers:
- name: vllm
image: your-registry/vllm-serving:TAG
ports:
- containerPort: 8000
resources:
requests:
cpu: "4"
memory: "24Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
env:
- name: HF_HOME
value: /models/hf
- name: MODEL_PATH
value: /models/hf/my-model # pre-downloaded or mirrored
command: ["/bin/bash","-lc"]
args:
- |
set -euo pipefail
rm -f /tmp/ready

# Start vLLM in background
vllm serve "${MODEL_PATH}" \
--host 0.0.0.0 --port 8000 \
--dtype auto \
--max-model-len 8192 \
--tensor-parallel-size 1 \
2>&1 | tee /var/log/vllm.log &
VLLM_PID=$!

# Wait for the server socket, then trigger a warm-up request.
# Replace the warm-up call with your own internal probe if needed.
for i in {1..120}; do
(echo > /dev/tcp/127.0.0.1/8000) >/dev/null 2>&1 && break
sleep 1
done

# Minimal warm-up: hit the server once (your internal client/probe here).
# If you can’t curl the API, run a lightweight local script instead.
curl -sf http://127.0.0.1:8000/ >/dev/null || true

# Mark ready only after warm-up.
touch /tmp/ready

# Keep foreground
wait $VLLM_PID
readinessProbe:
exec:
command: ["/bin/sh","-c","test -f /tmp/ready"]
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 30
startupProbe:
exec:
command: ["/bin/sh","-c","test -f /var/log/vllm.log"]
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 300 # allow long first load
volumeMounts:
- name: model-cache
mountPath: /models
volumes:
- name: model-cache
hostPath:
path: /var/lib/model-cache
type: DirectoryOrCreate

Notes for senior SREs

hostPath is powerful and dangerous. In a managed Kubernetes environment, you may prefer node-local ephemeral SSD mounts that the platform team controls, or a LocalPV setup with strict node affinity.

Set replicas to your SLA floor. Use HPA for burst, but don’t let it go to zero if p99 matters.

Measure it like an SRE: phase timings and startup benchmarks

Bridge: You can’t improve what you can’t attribute.

Use vLLM’s startup benchmark tooling

Bridge: Benchmark cold vs warm startup and block regressions.

vLLM ships a startup benchmark module to measure cold and warm startup times, including model loading and compilation/cache operations.

Run it against:

your container image

your model

your storage backend

your L40S node class

Then fail CI when cold start time regresses.

Log phase timestamps

Bridge: Turn “it’s slow” into numbers you can grep.

Log these timestamps per pod:

image pulled (node event)

process start

model fetch complete

weights loaded

warm-up complete

readiness true

Then build histograms:

cold_start_seconds{phase="fetch"}

cold_start_seconds{phase="load"}

cold_start_seconds{phase="warmup"}

This tells you where to spend effort.

Managed Kubernetes: what it helps, what it doesn’t

Bridge: Managed Kubernetes runs the plumbing. You still own the SLA path.

Managed Kubernetes can:

keep control plane stable

manage node lifecycle and autoscaler hygiene

standardize storage classes and node pools

It will not:

pick your cache strategy

keep models warm for your SLA

prevent you from scaling to zero and cold-starting on every burst

On AceCloud managed Kubernetes, the clean play is: dedicated GPU node pools for vLLM, pre-pull images, cache weights on node-local storage, set warm floors for SLA services, and Script warm-up into readiness. Keep your cold path measured and boring.

Checklist for PR reviews

Bridge: This is the short list that prevents “p99 spikes after deploy.”

Model artifacts are local or cached. Not pulled from the public internet at runtime.

GPU node pools are pinned (L40S), tainted, and isolated.

Image pre-pull exists on GPU nodes.

Readiness gates on “model is hot,” not “process started.”

Warm floor exists (min replicas > 0) for SLA services.

Cold vs warm startup is benchmarked in CI.

If you want, I can also add a short Prometheus section (cold-start phase histograms + alert rules) so the on-call page tells you which phase is burning your SLA.

Operational Risks of Running Large Multi-Tenant Kubernetes Clusters

Daya Shankar — Mon, 02 Mar 2026 06:30:39 +0000

Large multi-tenant Kubernetes clusters concentrate risk. Tenants share the control plane, core add-ons (CNI/CSI/Ingress/DNS), and scheduling capacity, so one bad deployment or “safe” upgrade can hit everyone.

The common failures are noisy neighbors, weak isolation, quota starvation, identity drift, and upgrade blast radius. Managed Kubernetes helps, but it won’t design tenancy for you.

What “multi-tenancy” means when you’re on call

If you don’t define the tenant boundary, you can’t defend it.

Multi-tenant Kubernetes usually means “many teams share one cluster.” The boundary is often a namespace. Sometimes it’s stronger: separate node pools, stricter network policy, workload identity, dedicated ingress, dedicated GPUs, etc.

Operationally, multi-tenancy is shared failure domains:

One API server.

One DNS stack (CoreDNS).

One CNI and conntrack table.

One CSI and storage path.

One ingress layer (or a few shared controllers).

One scheduler and one pool of allocatable capacity.

If you want a cluster to survive at scale, you need to decide which failures are allowed to be shared and which are not.

Noisy neighbors aren’t a “performance issue” they’re an outage pattern

Shared capacity turns small mistakes into cluster-level incidents.

CPU: throttling, request inflation, and scheduler lies

CPU is compressible, so people abuse it.

Two classic problems:

No limits + bursty workloads → one tenant burns cores and everyone’s latency climbs.

Overstated requests → scheduler thinks nodes are full → cluster autoscaler spins up nodes → real CPU sits idle.

If you size everything to p95 requests, you don’t just waste money. You also block bin-packing and create “Pending pods” incidents that look like infra failure.

Minimum guardrail

Enforce requests on CPU.

Be cautious with CPU limits for latency-sensitive services (throttling is real).

Use HPA with a real scaling signal. Don’t “set and forget.”

Memory: eviction storms and node death spirals

Memory is not compressible; it fails hard.

One tenant can trigger:

node memory pressure

kubelet evictions

cascading restarts

thundering herds as pods all re-pull images and rebuild caches

Minimum guardrail

Set memory requests and limits for all tenant workloads.

Alert on OOMKilled and eviction rates per namespace.

Keep headroom on nodes so eviction doesn’t become a cluster-wide reboot loop.

Disk/inode: the silent killer

Disk fills don’t page until they page everybody.

Common multi-tenant disk failures:

log storms filling /var/log or container runtime storage

inode exhaustion from small-file spam

image cache churn under high pod turnover

Minimum guardrail

Per-namespace log volume controls (don’t let one team spam logs).

Node alerts on disk/inode usage.

Runtime storage quotas where available.

Network: saturation and conntrack exhaustion

Networking failures hit all tenants because the kernel tables are shared.

When one tenant opens too many connections or you get a traffic spike:

conntrack table fills

packets drop

“random” timeouts appear across unrelated services

Minimum guardrail

Rate-limit at ingress.

Enforce egress policies.

Watch conntrack, dropped packets, and retransmits on nodes.

Isolation failures become security incidents

Namespaces are a convenience boundary, not a security boundary.

RBAC drift and privilege creep

RBAC starts clean and rots fast in shared clusters.

The failure mode is predictable:

a team needs one permission

someone grants a broad ClusterRole

later, nobody remembers it exists

now the tenant can list secrets cluster-wide or mutate critical resources

Minimum guardrail

Centralize ClusterRole creation.

Lint RBAC in CI (Script it; don’t “review in Slack”).

Ban wildcard verbs/resources for tenant roles.

Workload identity and cloud IAM misbinding

The fastest way to leak data is to bind the wrong identity to the right pod.

In multi-tenant, identity mistakes propagate:

a shared service account gets reused

a workload identity binding is too broad

pods gain access to buckets/queues they should never see

Minimum guardrail

One workload identity per service, not per namespace.

Deny “default” service account usage for real workloads.

Audit “who can assume what” regularly and pipe it to alerts.

Pod security exceptions that never die

The exception list grows until it becomes the policy.

If you allow privileged pods, hostPath mounts, or host networking for one team, you’ve opened a side door for everyone unless you lock it down.

Minimum guardrail

Use Pod Security Admission (baseline/restricted) as default.

Require an exception workflow with expiry.

Grep for privileged/hostPath usage weekly.

Network policy gaps turn “one bad app” into “everyone is down”

Flat networks are how tenant bugs become tenant outages.

Default-allow is the default failure

If everything can talk to everything, blast radius is automatic.

A single noisy service can:

hammer shared dependencies

trigger thundering herds

overload DNS

spike cross-namespace traffic

Minimum guardrail

Default-deny egress and ingress per namespace.

Explicit allowlists for shared services.

Treat policies like code (PRs, review, tests).

Shared ingress controllers amplify mistakes

One bad ingress change can break unrelated tenants.

Failure patterns:

config reload loops

bad annotations triggering expensive behaviors

certificate mis-rotation

Minimum guardrail

Separate ingress controllers by tenancy tier (shared/dev vs prod/critical).

Canary ingress changes.

Enforce annotation allowlists.

DNS is a shared single point of failure

CoreDNS is the cluster’s heartbeat; overload it and nothing resolves.

In big clusters, DNS load grows with:

pod count

churn

retries during incidents

Minimum guardrail

Scale CoreDNS for QPS and cache settings.

Alert on CoreDNS latency/errors.

During incidents: Grep logs for upstream timeouts and SERVFAIL.

Scheduling and quota pathologies at scale

“Fair scheduling” is policy you must configure, not something Kubernetes gifts you.

Quota starvation and priority inversion

One tenant can starve others without “breaking rules.”

Common patterns:

Tenant A uses up shared node pool capacity with big requests.

Tenant B is within quota but can’t schedule due to fragmentation.

Everyone blames the scheduler. It did what you told it.

Minimum guardrail

ResourceQuotas per namespace (CPU/memory/pods).

LimitRanges to prevent “no requests” and “ridiculous limits.”

Separate node pools for noisy/bursty tenants.

Preemption can save prod or murder batch

Preemption is a knife. Use it like one.

If you enable priority + preemption:

your critical services can recover capacity

your batch jobs can get killed repeatedly unless they checkpoint

Minimum guardrail

PriorityClasses: “prod-critical”, “prod”, “batch”.

For batch: checkpoint or accept loss.

Measure eviction rates after enabling.

Upgrade and change-management risk is multiplied by tenant count

In big clusters, “safe changes” are the biggest outage source.

The shared add-ons are the sharp edges:

CNI upgrades

CSI upgrades

ingress controller upgrades

API deprecations breaking controllers/operators

node patching + drains deadlocking on PDBs

Failure mode you will hit: PDB deadlock.
A drain starts, pods can’t evict due to strict budgets, the rollout stalls, capacity shrinks, and unrelated tenants get squeezed.

Minimum guardrail

Canary upgrades in a smaller cluster or a dedicated pool.

Script rollback paths.

Set realistic PDBs (protect availability, don’t freeze the cluster).

Observability and incident response get harder as the cluster gets bigger

If you can’t attribute load to a tenant, you can’t run multi-tenant.

Per-tenant attribution is mandatory

“The cluster is slow” is not an actionable alert.

You need:

dashboards by namespace (CPU/mem/network/disk)

request rates at ingress by tenant

top talkers (network) and top allocators (memory)

Cardinality will bite you. Don’t ship every label. Decide which labels you can afford, then enforce it.

Audit logs and “who did what”

Multi-tenant incidents often start as “someone applied something.”

Enable audit logs and make them searchable. When you’re debugging a weird outage, you should be able to answer:

who changed the Deployment?

who changed the NetworkPolicy?

who updated the ingress?

Managed Kubernetes changes the risk profile, not the physics

Managed Kubernetes reduces control-plane toil, not tenant blast radius.

Managed Kubernetes usually helps with:

control plane uptime/patching

some upgrade orchestration

basic integrations

It does not automatically give you:

tenant isolation

safe defaults for quotas and policies

sane RBAC boundaries

disciplined change control for shared add-ons

If you’re on a managed Kubernetes offering like AceCloud, use the managed layer for what it’s good at (platform plumbing), then enforce tenancy guardrails at the cluster policy layer (quotas, PSA defaults, network policies, and tiered node pools). That’s where multi-tenancy succeeds or fails.

Mitigation playbook (week 1 controls that actually reduce incidents)

These are the controls you can deploy fast and feel immediately.

1) Create tenancy tiers

Not every workload deserves to share the same failure domain.

Shared/dev tier: many tenants, lower guarantees

Prod/shared tier: stricter policies, more guardrails

Prod/dedicated tier: separate node pools or separate clusters for the truly critical

2) Enforce default-deny networking

Flat networks are the default blast radius.

Deploy default-deny policies per namespace. Add explicit allow rules.

3) Lock down RBAC and pod security

Security drift is guaranteed unless you block it.

central RBAC templates

Pod Security Admission defaults

expiring exceptions

4) Quotas + LimitRanges everywhere

Multi-tenant without quotas is “first team to deploy wins.”

ResourceQuota per namespace

LimitRange to prevent “no requests” and “unbounded limits”

Alerts on quota saturation and Pending pods

5) Safer change management for shared components

Your add-ons are shared infrastructure. Treat them like prod.

canary upgrades

rollback scripts

PDB sanity checks before drains

runbooks for CNI/CSI/Ingress/DNS failures

Bottom line

Large multi-tenant clusters work when you treat them like a shared operating system.

A big shared Kubernetes cluster isn’t “just more nodes.” It’s a bigger shared failure domain. The operational risks are predictable: noisy neighbors, weak isolation, quota starvation, identity drift, and upgrade blast radius.

If you want reliability, you must Configure guardrails, Script rollouts, and verify attribution per tenant whether you run it yourself or on managed Kubernetes.

Hosted control plane: when it simplifies operations and when it adds complexity

Daya Shankar — Fri, 27 Feb 2026 06:13:52 +0000

A hosted control plane moves Kubernetes control-plane components off your worker fleet either into a provider-managed boundary (EKS) or onto a separate hosting cluster as pods (HyperShift).

It simplifies ops when you want predictable upgrades, less per-cluster snowflake work, and cleaner separation between “management” and “workloads.”

It adds complexity when control-plane connectivity, IAM, and shared blast radius become your new failure modes especially with private clusters.

Define hosted control plane in concrete terms

If you can’t say where the API server and etcd live, you can’t model risk.

“Hosted control plane” is a placement decision.

EKS: hosted by AWS in an EKS-managed VPC

AWS owns the masters; you own nodes and workloads.

AWS documents that the EKS-managed control plane runs inside an AWS-managed VPC and includes Kubernetes API server nodes and an etcd cluster. API server nodes run in an Auto Scaling group across at least two AZs; etcd nodes span three AZs.

What that means operationally:

You don’t patch control-plane instances.

You don’t rebuild etcd.

You do still own access, RBAC, node lifecycle, and add-ons.

kubeadm on EC2: not hosted, you host it

You run the masters, the etcd, the upgrades, and the recovery drills.

Kubeadm HA requires you to pick a topology (stacked etcd vs external etcd) and wire up the endpoints (often via a load balancer DNS name). External etcd needs explicit endpoint configuration; stacked etcd is “managed automatically” by kubeadm’s topology.

What that means operationally:

You patch and upgrade the control plane.

You own etcd snapshots and restore tests.

You own certificates and rotation edge cases.

HyperShift (hosted control planes): control planes as pods on a hosting cluster

You consolidate many control planes onto one management cluster.

Red Hat’s hosted control planes model runs control planes as pods on a management/hosting cluster, without dedicated VMs per control plane.

HyperShift then introduces a new question: where do those control plane pods land? Docs show “shared everything” by default, and you can dedicate nodes for control plane workloads via labels/taints.

Side-by-side: what gets simpler, what gets harder

Feature lists lie. Ownership and failure modes don’t.

Model	What simplifies	What gets harder	The new “pager line”
EKS hosted control plane	Control plane HA, scaling, replacement; less etcd babysitting	Endpoint access + SG design for private clusters; version planning	“Can we reach the API endpoint from the right networks?”
kubeadm on EC2	Full control; no managed constraints	Everything: HA wiring, etcd ops, upgrades, certs	“etcd is sick” is your incident
HyperShift	Reduce per-cluster control-plane VMs; faster cluster churn; multi-tenant mgmt	Hosting cluster becomes shared blast radius; two-layer debugging	“Hosting cluster health” pages everyone

When a hosted control plane simplifies operations

Hosted control planes help when your bottleneck is “running too many control planes.”

1) You operate many clusters (multi-tenant SaaS, env sprawl)

Cluster count is the multiplier.

If you run 20+ clusters, self-managed control planes become a tax:

patch windows multiply

certificate and etcd risk multiplies

“one-off cluster drift” becomes normal

EKS removes the control plane instances from your fleet and gives you a standardized control plane architecture across AZs.

HyperShift goes further: it removes dedicated control-plane machines per cluster and runs them as pods on a hosting cluster.

2) You want predictable control-plane availability without building an etcd practice

etcd is not hard until it’s hard at 3 AM.

kubeadm HA docs are clear: external etcd adds configuration surface area (explicit endpoints); stacked etcd is simpler but still your operational problem.

If your team doesn’t want to own etcd restores as a practiced drill, a hosted control plane removes that class of work from your team’s backlog.

3) You need fast cluster create/delete (ephemeral clusters, tenant clusters)

Provisioning speed is operational leverage.

HyperShift is designed around the concept of creating control planes as pods on a management cluster, which reduces the need to “spin up” dedicated control-plane machines per hosted cluster.

That’s useful when:

you create short-lived clusters for CI

you provision tenant clusters and churn them

you want cluster lifecycle to look like Deploying an app

4) You’re private-cluster-heavy and want a supported endpoint model

Private changes the operational shape more than any “feature.”

EKS lets you run a private-only API server endpoint (no public access), where kubectl must come from within the VPC or connected networks. Access to the private endpoint is controlled by rules on the cluster security group.

That’s not “simpler” in absolute terms. It’s simpler because it’s a supported, documented pattern with fewer moving parts than self-hosting your own API endpoint VIP/LB and cert story.

When a hosted control plane adds complexity

You trade “masters on VMs” for “network + IAM + shared blast radius.”

1) Control-plane connectivity becomes a first-class dependency

The API server is now “across a boundary,” and boundaries fail.

With EKS private-only clusters:

your kubectl, CI runners, and controllers must live inside the VPC or connected networks

your security group rules become part of cluster availability

With public endpoint access, the default behavior has historically been public enabled / private disabled (and you can toggle both).
Either way, endpoint mode is now a design choice you must document, test, and audit.

What changes for on-call:

“API is down” might really be “route to endpoint is broken”

DNS, TGW/peering, SG rules, and client network become suspects

2) Identity boundaries get sharper (and easier to misconfigure)

Hosted control planes push you into “who can reach what” decisions.

Private endpoint + security group control is good. It’s also easy to get wrong:

over-broad SG rules turn “private endpoint” into “private but reachable from everything”

too-tight rules break controllers and CI/CD in weird ways

Hosted doesn’t remove IAM work. It moves it to the center of the blast radius.

3) HyperShift’s hosting cluster becomes shared infrastructure

You didn’t delete control planes. You consolidated them.

HyperShift runs control planes as pods on a hosting cluster.
Docs show that hosted control plane pods can be scheduled broadly (“shared everything”), and you can taint/label nodes to dedicate capacity.

This is the operational trade:

Pro: fewer dedicated control-plane machines per tenant cluster

Con: hosting cluster saturation, upgrades, or outages can hit multiple hosted clusters at once

If you adopt HyperShift, treat the hosting cluster like tier-0 infrastructure:

separate node pools

aggressive monitoring

strict change control

tested disaster recovery

4) Debug becomes two-layer

Symptoms show up in the guest cluster; root cause can live elsewhere.

With EKS, control plane is managed. You troubleshoot via endpoint reachability, AWS telemetry, and cluster behavior. You can’t SSH into masters, and that’s the point.

With HyperShift, you can often inspect control plane pods on the hosting cluster. That’s powerful and it means your runbooks must cover two clusters:

guest cluster symptoms

hosting cluster root cause

Private clusters: the “hosted” decision that matters most

Private mode turns networking into part of the control plane.

EKS private endpoint: supported, but policy-heavy

SG rules are now part of cluster uptime.

AWS states that for private-only API servers:

there is no public access from the internet

kubectl must come from the VPC or connected network

cluster security group rules control private endpoint access

This is clean if you already run:

TGW / VPC peering / Direct Connect

private DNS resolution patterns

locked-down egress

It’s messy if your ops tooling lives outside the network boundary and you aren’t ready to move it.

kubeadm private: you own the endpoint and its failure modes

You don’t get a managed endpoint; you build one.

kubeadm HA guides assume you Configure a load balancer in front of the control plane nodes and wire up DNS names and endpoints.

That’s flexible. It’s also more work:

API endpoint LB health checks

TLS/cert rotation

routing changes during upgrades

HyperShift private: you design exposure between hosting and guest clusters

Hosted control planes still need reachable endpoints.

Hosted control plane pods live on the hosting cluster. That’s good for consolidation. It also means you must design:

how guest nodes reach the hosted API server

how admins reach it (private networks, bastions, CI runners)

how you segment tenants

The exact networking patterns vary by environment, but the invariant is: private hosted control planes increase the importance of network design.

Terraform: what you actually manage in each model

IaC doesn’t disappear. The resource graph changes.

EKS Terraform surface area

You configure endpoint modes, SGs, node groups, and IAM.

Minimum Terraform concerns:

endpoint access mode (public/private/both)

cluster security group rules for private access

node groups and AMI strategy

IRSA and IAM boundaries

Hosted control plane simplifies the “masters” part. It does not simplify the access-control part.

kubeadm Terraform surface area

Terraform becomes your control-plane installer, not just a cluster creator.

You end up managing:

control plane EC2 instances

LB/VIP in front of API servers (common HA pattern)

etcd instances (external) or colocated etcd (stacked)

bootstrap scripts, cert distribution, upgrade workflows

This can be clean if you have mature automation. If not, it’s a lot of state to keep consistent.

HyperShift Terraform surface area

You manage the hosting cluster like a platform, then declaratively create hosted clusters.

HyperShift adds:

hosting cluster lifecycle (upgrade, capacity, resilience)

hosted cluster objects and their infra mappings

scheduling policies for control plane pods (dedicated nodes via labels/taints)

Terraform can drive parts of this, but you’ll also lean on cluster-native controllers.

Prometheus: what you need to watch so hosted doesn’t surprise you

Hosted control planes move failure modes. Your dashboards must follow.

At minimum, split monitoring into two planes:

Workload plane (guest cluster apps)
request rates, latency, errors
node saturation
queue depth / retries
Control plane plane
API server availability/latency from where your clients run
controller health signals
for HyperShift: hosting cluster resource pressure, because control planes are pods

For private clusters, add synthetic checks from the networks that matter:

from CI runner network

from admin network

from in-cluster controllers

If the API endpoint is unreachable from your automation network, you don’t have a cluster. You have a museum exhibit.

Decision checklist for SaaS and platform teams

Answer these honestly and the right model usually falls out.

How many clusters will you run in 12 months?
If the number is growing fast, hosted control plane saves toil.
Do you have an etcd practice?
If “restore drill” isn’t something you run quarterly, kubeadm HA is a risk trade.
Is private-only mandatory?
If yes, model endpoint reachability and SG rules as part of uptime.
Can you tolerate shared blast radius?
HyperShift consolidates control planes. Treat hosting cluster as tier-0.
What do you want to debug at 3 AM: VMs or networks?
kubeadm tends toward VM-level debugging.
hosted control planes tend toward network/identity debugging.

Where AceCloud fits

Hosted control plane only helps if the day-2 loop is owned and scripted.

If you’re buying hosted control plane benefits but don’t want to run the surrounding ops (endpoint policies, Terraform hygiene, Prometheus wiring, upgrade runbooks), a managed Kubernetes provider like AceCloud can own that platform loop while your team focuses on workload correctness and SLOs.

Bottom line

Hosted control plane is not “less complexity.” It’s different complexity.

Pick a hosted control plane (EKS) when you want AWS to own control plane HA, scaling, and replacement across AZs.
Pick kubeadm when you need maximum control and you’re willing to own HA topology, etcd ops, and endpoint plumbing.
Pick HyperShift when you need to run many clusters and you’re ready to operate a tier-0 hosting cluster that runs control planes as pods.

The correct choice is the one that gives every failure mode a clear owner—and keeps your pager quiet for the right reasons.

Serving LLMs on IaaS: throughput vs latency tuning with practical guardrails

Daya Shankar — Fri, 27 Feb 2026 06:11:05 +0000

Serving LLMs on IaaS is queueing plus memory pressure dressed up as ML. Every request has a prefill phase (prompt → KV cache) and a decode phase (token-by-token output).

Throughput tuning pushes batching and concurrency. Latency tuning caps them to protect TTFT and ITL. With vLLM on a single L40S (PCIe), you win by setting hard limits and enforcing admission control.

TTFT, ITL, TPS: stop mixing the metrics

If you tune the wrong metric, you’ll ship a fast benchmark and a slow product.

You need three numbers, and they mean different things:

TTFT (time to first token): how long the user waits before anything shows up. Interactive UX lives here.
ITL (inter-token latency): the “smoothness” of streaming output once decoding starts. Chat feels broken when this jitters.
Throughput (tokens/sec): the finance metric. It decides cost per request.

One important detail: E2E latency includes queueing + prefill + decode. TTFT is where queueing hides when you’re overloaded.

Practical measurement rule: measure TTFT and ITL at the client (or gateway), not inside the GPU server. Internal timings miss queueing in front of vLLM.

Hardware reality check: single L40S on PCIe

You can’t tune around a bus you don’t have.

An L40S is a strong inference GPU, but it’s not an NVLink box. It’s 48GB GDDR6 on PCIe Gen4 x16.
That matters because:

You have one GPU’s worth of memory for weights + KV cache.
You don’t get multi-GPU model parallel tricks for free.
Your main enemies are KV-cache pressure and batch/concurrency overshoot, not “GPU topology.”

On a single GPU server, latency failures usually look like:

TTFT spikes because the prefill queue grows.
ITL spikes because decode gets starved or the batch gets too big.
OOM/restarts because KV cache math was wishful thinking.

vLLM’s default behavior: TTFT-first scheduling (and the trade)

vLLM already picks a side; your job is to set guardrails around it.

By default, vLLM’s scheduler prioritizes prefills and does not batch prefill and decode into the same batch. That typically optimizes TTFT, but can worsen ITL and GPU utilization.

Translation: out of the box, vLLM tries to be responsive. You can still break it by feeding it mixed traffic with no limits.

The knobs that actually move TTFT, ITL, and OOM risk

You don’t “optimize latency.” You Configure concurrency and KV-cache headroom.

These four knobs do most of the work in production vLLM.

1) --max-num-seqs caps concurrency

This is your “how many requests can be active” ceiling.

--max-num-seqs is the maximum number of sequences per iteration.
Lowering it:

reduces concurrent KV cache usage
reduces queue contention inside the engine
usually helps tail latency (until you underutilize the GPU)

2) --max-num-batched-tokens controls batch size per iteration

This is where you trade throughput for TTFT/ITL stability.

--max-num-batched-tokens limits batched tokens per iteration.
Lowering it:

reduces “one huge prefill” events
reduces KV cache demand per cycle
can protect TTFT and prevent decode jitter

Raising it:

can increase throughput
can increase queueing and tail spikes if your traffic is bursty or prompts are long

3) --gpu-memory-utilization sets KV-cache headroom

This decides how much VRAM vLLM pre-allocates for cache.

vLLM pre-allocates GPU cache using gpu_memory_utilization. Increase it to provide more KV cache space.
If you set it too high, you risk fragmentation and less room for everything else. If you set it too low, you’ll hit KV cache limits early and TTFT will spike under concurrency.

4) --enable-chunked-prefill tames long prompts

Long prompts are TTFT killers; chunking makes them less explosive.

When enabled, vLLM can chunk prefill requests based on max_num_batched_tokens.
This is a practical guardrail when you can’t control prompt length perfectly.

A sane starting config for your SLA (p95 TTFT 250ms, p99 800ms)

Start conservative, hit the TTFT target, then earn throughput back.

On a single L40S, don’t begin with “maximum throughput.” Begin with “stable TTFT.”

Example vllm serve baseline (single GPU):

vllm serve /models/your-llm \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 64 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill

Why these shapes:

max_num_seqs prevents unlimited concurrency blowups.
max_num_batched_tokens prevents one batch from ballooning.
gpu_memory_utilization keeps cache headroom explicit.
chunked prefill reduces “one giant prompt ruins the minute.”

You will tune these. But you need a stable base first.

Practical guardrails for mixed chat + batch traffic

Throughput tuning is easy. Guardrails are what keep p99 alive.

Mixed traffic (interactive + batch) is where systems get weird. Batch clients tend to:

send long prompts
request long generations
retry aggressively
keep load constant

Interactive chat needs:

fast TTFT
consistent ITL
predictable tail behavior

So you need admission control in front of vLLM. Not “best effort.”

Guardrail table (start here)

These caps stop one client from torching everyone else.

Guardrail	Default starting point	Why it exists
Max prompt tokens	4k–8k (per request)	Long prefills blow TTFT and batch size
Max output tokens	256–512 (interactive), 1k+ (batch)	Protect tail latency for chat
Max in-flight requests	Tie to max_num_seqs	Prevent internal queue explosion
Max queue depth	1–2× in-flight	If queue > that, reject/429 fast
Request timeout	Slightly above p99 target	Don’t let zombie requests clog decode
Retry policy	capped + jitter	Stops retry storms multiplying load

These aren’t theoretical. They’re how you keep a single GPU server usable.

Two-lane routing (interactive vs batch)

If you mix traffic in one FIFO queue, batch wins and chat loses.

On one GPU, the clean pattern is two lanes at the gateway:

Interactive lane: strict caps (short prompts, short outputs), low queue depth.
Batch lane: looser caps, but it yields when interactive is busy.

You can implement this with a thin gateway that:

inspects request size (prompt tokens + requested output tokens)
routes “interactive” to the main lane
routes “batch” to a background lane with stricter admission

Even if both lanes hit the same vLLM backend, the queue policy changes outcomes.

Concrete rule that works:
If interactive queue depth > N, reject batch (429) instead of letting it sit and inflate TTFT.

The tuning loop that converges (without cargo cult)

Tune one knob at a time and measure TTFT and ITL separately.

Here’s the loop to run on a GPU cloud server before you call it “production.”

Step 1: Fix the workload mix

Your traffic generator must match reality.

Build two test profiles:

Chat: short prompts, short outputs, bursty concurrency.
Batch: longer prompts and outputs, steady concurrency.

If you benchmark only one, you’ll tune only one.

Step 2: Lock SLOs first

You already have targets; enforce them.

Targets:

TTFT p95 ≤ 250ms
TTFT p99 ≤ 800ms

Keep a red line on the dashboard. If a tuning change crosses it, roll back.

Step 3: Set limits, then raise carefully

Earn throughput; don’t steal it from p99.

Order of operations:

Set max_num_seqs low enough that you never OOM under your worst prompt mix.
Set max_num_batched_tokens to prevent giant prefills from blocking decode.
Adjust gpu_memory_utilization to give KV cache room.
Enable chunked prefill if long prompts exist in real traffic.

Then:

increase max_num_seqs until TTFT p95 hits the edge of your budget
increase max_num_batched_tokens only if ITL stays stable and TTFT doesn’t spike

Step 4: Add overload behavior on purpose

A good system fails fast, not slowly.

Define overload mode:

when queue depth exceeds threshold → return 429 with Retry-After
when prompt/output exceeds limits → return 400 with a clear message
when batch lane is busy → shed batch first

If you don’t define this, your system will “define it” by melting.

Dashboards that catch trouble before users do

You can’t grep production. You need signals that predict tail spikes.

Track:

TTFT p50/p95/p99 (interactive lane, batch lane)
ITL distribution (interactive lane)
queue depth and reject rate (the guardrail is working if it fires)
GPU memory usage and cache pressure (OOM risk proxy)

vLLM already frames TTFT/ITL as the core performance story, and its scheduler tradeoffs explain why TTFT can look good while ITL suffers (or vice versa).

Where AceCloud fits (one honest paragraph)

IaaS isn’t the problem; inconsistency is.

If you’re serving on an IaaS gpu cloud server from a provider like AceCloud, treat it like any other VM: bake a known image, pin driver/CUDA versions, and script your vLLM flags so every node behaves the same. The tuning work above only sticks when the box is predictable.

Bottom line

Throughput is what you brag about. Latency is what users feel.

On vLLM + single L40S, you don’t win by chasing max tokens/sec. You win by controlling concurrency and batch size, allocating KV cache intentionally, and enforcing guardrails that keep mixed traffic from turning into a queueing disaster. Hit TTFT p95/p99 first. Then scale throughput without stealing it from your tail.

Memory Ballooning Effects in Virtualized Cloud Environments

Daya Shankar — Thu, 19 Feb 2026 14:55:13 +0000

Memory ballooning is a host memory reclaim method used during VM overcommit. The hypervisor inflates a balloon driver inside a VM to claw back RAM.

It can avoid host swapping, but it also shrinks guest page cache and can trigger paging. In Kubernetes, you see MemoryPressure, pod evictions, and tail-latency spikes.

What memory ballooning is

Ballooning is cooperative reclaim. It’s not “free memory.”

On VMware, the balloon driver (vmmemctl) works with the host to reclaim pages the guest considers least valuable.

VMware’s own perf guidance is blunt: avoid overcommit that forces regular host swapping, because that’s where performance collapses.

What you actually see in a managed Kubernetes service

You don’t see “ballooned MB.” You see consequences.

Kubelet enforces node-pressure eviction. Default hard threshold on Linux is memory.available<100Mi, and hard evictions have no grace period.
So any reclaim event that drops memory.available can turn into kills.

How ballooning pressure turns into outages on K8s nodes

This is the failure chain you should expect under overcommit.

Cache gets punched → more disk reads → p95 climbs.
Paging starts → jitter rises.
Kubelet evicts → restarts + thundering herd.

You don’t need hypervisor access to catch this. You just need node metrics and events.

What AceCloud gives you to control blast radius

You control node sizing and node-group policy, not the host reclaim knobs.

AceCloud Managed Kubernetes exposes worker node configurations like 2 vCPU/4 GiB, 4 vCPU/8 GiB, 8 vCPU/16 GiB (their published comparison table).
If you need bigger worker nodes, AceCloud’s flavor catalog shows Standard Instance options like S3a.2xlarge (8 vCPU/32 GiB) up through S3a.8xlarge (32 vCPU/128 GiB) and beyond.

Guardrails that work when overcommit is “yes”

These are defaults you can deploy without cluster-specific tuning.

Split worker node groups by risk

One node pool for prod latency. One for batch.

Protected pool: ingress, API, user-facing services.

Best-effort pool: ETL, async jobs, rebuilds.

This keeps batch from turning your prod nodes into the provider’s pressure valve.

Enforce requests/limits everywhere

The scheduler packs based on requests. If you don’t set them, you’re gambling.

Use Kubernetes resource requests/limits for CPU and memory.
For latency pods, run Guaranteed QoS: requests == limits.

resources:
requests:
cpu: "1"
memory: "2Gi"
limits:
cpu: "1"
memory: "2Gi"

Keep headroom by design

If you can’t tune Node Allocatable, you simulate it with request budgets.

Kubernetes calls this concept Node Allocatable (reserving resources for system daemons).
In a managed service, you may not get to set kube-reserved / system-reserved, so leave headroom in pod requests.

Baseline rule (protected nodes): don’t schedule more than 75% of node RAM by requested memory.

Pod density: don’t chase 110

Kubernetes “supported scale” guidance says no more than 110 pods per node.
Some platforms can configure higher, but pod IP and CNI limits usually bite first.

Use caps that match memory, not bragging rights.

Starting caps for AceCloud-sized worker nodes

Assumptions: typical daemonsets, no hugepages/DPDK, overcommit exists somewhere upstream.

Worker node	Role	Max total pod memory requests	Pod cap	Why
4 GiB	best-effort	2.5–3.0 GiB	15–25	leaves OS+kube headroom
8 GiB	protected	5.5–6.0 GiB	25–40	avoids eviction on small dips
16 GiB	protected	11–12 GiB	40–70	room for spikes + cache
32 GiB	mixed	24–26 GiB	70–110	only if requests are real

Anchor: the 110-pods/node guidance is a ceiling, not a target.

Evictions: make them predictable

If you can’t set kubelet flags, you still control which pods die first.

Assign PriorityClasses.

Put best-effort on best-effort nodes.

Put strict limits on batch so it can’t eat the node.

Know the kubelet defaults: memory.available<100Mi is the hard tripwire on Linux.

Swap: pick a stance and document it

Swap support exists now, but it’s not “turn it on and pray.”

Kubernetes documents swap memory management and node swap behaviors (including LimitedSwap).

Practical policy:

Protected nodes: swap off unless you’ve load-tested tail latency with swap on.

Best-effort nodes: consider LimitedSwap if you accept slower jobs.

What to alert on (works in any managed K8s)

You don’t need vCenter. You need signals.

Kubernetes-level

Node condition: MemoryPressure=True

Events: eviction messages

(Default eviction behavior is documented upstream.)

Node-level (Prometheus / node-exporter)

Alert on:

sustained low MemAvailable

paging activity (pgmajfault, pswpin, pswpout)

memory PSI pressure rising

If those light up during latency spikes, you’re in reclaim/paging territory.

Where AceCloud fits in this story

This is how you use their catalog without lying to yourself.

Start with AceCloud’s published worker sizes (4/8/16 GiB) for general pools.

For memory-heavy services (Kafka, JVM heaps, model servers), move the protected pool to bigger flavors from the standard catalog (ex: 8 vCPU/32 GiB and up).

Scale node groups earlier instead of packing nodes to the cliff. Node-group autoscaling is part of their managed Kubernetes offering.

If you want the “tight” version for your cluster

You can do it later from any terminal with cluster access, but you don’t need it to start.

Use the caps table above.

Enforce requests/limits + PriorityClasses.

Split node groups.

Keep 20–25% memory headroom on protected nodes.

That stops the common eviction storm even when the provider is running overcommit behind the scenes.

Hybrid Orchestration Basics: Avoiding Single-Provider Risks in 2026

Daya Shankar — Thu, 19 Feb 2026 14:49:58 +0000

Hybrid orchestration in 2026 means you can deploy the same workload across on-prem + AWS (and a second cloud if needed) using Kubernetes + Terraform + Argo CD as the common layer.

Keep Git as source of truth. Standardize identity, DNS, ingress, and observability. Then test failover like it’s a feature, not a promise.

What “single-provider risk” looks like

You can’t mitigate what you won’t name.

Risk	What breaks first	What it looks like at 2AM
Region/control-plane dependency	Deploy pipeline, cluster ops	“Can’t roll back. API calls time out.”
IAM lock-in	Workload identity, secrets access	“Pods can’t auth off-cloud.”
Network primitives	Ingress/LB, DNS	“Traffic won’t steer. Health checks lie.”
Data gravity/egress	DR, migration	“Failover works, but costs explode.”
Managed service coupling	DB/cache/queue	“App is portable. State is not.”

Rule: If your deploy and auth only work inside AWS, you don’t have “hybrid.” You have “AWS with extra steps.”

Pick a hybrid shape that matches reality

Topology decides your failure modes.

Option A: Two independent clusters (recommended default)

This is the boring one. It works.

Cluster 1: EKS in AWS

Cluster 2: on-prem Kubernetes (or another provider)

Argo CD fans out apps to both. Terraform builds both. You can fail one without taking the other’s control plane with it.

Option B: “Stretched cluster” (know the connectivity tax)

This is EKS Hybrid Nodes territory: control plane in AWS Region, nodes on-prem.

AWS calls this a “stretched/extended” cluster architecture.
AWS also publishes best practices that assume redundant, resilient connectivity to avoid disconnections.

Use it when:

you want one control plane

you can engineer reliable private connectivity

Avoid it when:

your on-prem is intermittently connected

you need disconnected/air-gapped operations

Option C: Disconnected/air-gapped on-prem

If “internet might not exist,” treat it as a hard requirement.

AWS documents EKS Anywhere as capable of running in air-gapped/disconnected environments.

Reference architecture

Every subsystem needs a home.

Git (source of truth)
|
v
Argo CD (GitOps)
(runs on-prem or neutral)
/ \
v v
On-prem K8s cluster AWS EKS cluster
(apps + addons) (apps + addons)
\ /
\ /
v v
Shared services: DNS, OIDC, logging/metrics,
container registry (mirrors), secrets KMS strategy

Rule: Put the GitOps control plane where a provider outage can’t strand you. Argo CD is a Kubernetes controller that continuously compares live state to Git and reports drift as OutOfSync.

Terraform: build infra once, not by hand

Terraform is for infra. Argo is for convergence. Don’t mix them.

Terraform responsibilities

VPC/VPN/Direct Connect edge

EKS cluster + node groups

On-prem cluster primitives (or the platform that hosts it)

IAM/OIDC scaffolding

Base DNS zones / records (if you must)

Repo layout that survives day-2

Keep it simple:

infra/
aws/
eks/
network/
onprem/
k8s/
apps/
base/
overlays/
aws/
onprem/
gitops/
applicationsets/

Argo CD: one template, many clusters

Multi-cluster GitOps is the whole point.

Argo CD supports ApplicationSet for multi-cluster automation.
The Cluster generator can auto-discover clusters registered in Argo CD and expose their metadata as template parameters.

Example: ApplicationSet that deploys to both clusters

Label your clusters in Argo (env=aws, env=onprem), then:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-addons
spec:
generators:
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: ["aws", "onprem"]
template:
metadata:
name: "addons-{{name}}"
spec:
project: platform
source:
repoURL: https://git.example.com/platform.git
targetRevision: main
path: "apps/overlays/{{metadata.labels.env}}/addons"
destination:
server: "{{server}}"
namespace: platform
syncPolicy:
automated:
prune: true
selfHeal: true

This gives you:

one definition

two targets

drift correction

Portability boundary: decide what must stay portable

Hybrid fails when you pretend everything is portable.

Portable by default

Kubernetes APIs (Deployments, Services, Ingress)

Helm/Kustomize overlays

Argo CD delivery mechanics

OpenTelemetry-based app telemetry

Not portable unless you plan it

Managed databases

Provider IAM-only auth

Provider-specific LBs and DNS behavior

Storage classes with provider-only semantics

Rule: If state can’t move, failover is theater.

Identity: stop wiring apps to one cloud’s IAM

Auth is the first thing that breaks off-cloud.

Baseline pattern:

Use OIDC for human and workload identity.

Use Kubernetes service accounts mapped to your identity provider.

Keep secrets strategy consistent (Vault, SOPS, external secret operators) across clusters.

If AWS IAM is your only workload identity story, your on-prem cluster becomes a second-class citizen.

Networking: make DNS and routing boring

Hybrid is mostly DNS and routes.

Minimum requirements:

deterministic routing between on-prem and AWS (VPN/Direct Connect)

clear ownership of egress/ingress paths

DNS resolution both directions (forward + reverse if needed)

If you choose “stretched EKS,” AWS’s docs push you to engineer resilient connectivity and plan for disconnections.

Operations: avoid doubling your surface area

Two clusters means two of everything unless you standardize.

One observability pipeline

one metrics backend

one log backend

consistent labels: cluster, env, region, service

One upgrade policy

version skew rules

maintenance windows

rollback runbooks

One incident drill

Run this quarterly:

break AWS ingress (simulate region/LB outage)

fail traffic to on-prem

verify auth, DNS, data correctness

roll back cleanly

If you can’t rehearse it, don’t claim it.

Where AceCloud fits in a “don’t bet on one provider” plan

If you want a second cloud without rewriting your platform, add it as another Kubernetes target.

AceCloud’s docs show a managed Kubernetes flow built around worker node groups, where you pick Flavor Type/Name, worker count, per-node volume, and security group.
That maps cleanly to the same GitOps model:

Terraform (or API) builds the cluster/node groups

Argo CD registers the cluster

ApplicationSet deploys the same overlays

This gives you a practical hedge:

AWS EKS as primary

on-prem as locality/compliance anchor

AceCloud as a secondary cloud target for burst, DR rehearsals, or an exit ramp

CTO checklist

Print this and use it in reviews.

GitOps control plane is provider-neutral (or at least not single-region)

Two independent clusters exist (on-prem + AWS), not just a stretched cluster

Argo CD multi-cluster deployment is automated (ApplicationSet)

Identity works off-cloud (OIDC strategy, not AWS-only IAM)

DNS and routing are deterministic (and tested)

Failover drill is scripted and run regularly

State portability is explicitly defined (what can fail over, what can’t)

GPU Scheduling Deep Dive: How Cloud Providers Allocate GPUs for Multi-Tenant AI Workloads

Daya Shankar — Thu, 19 Feb 2026 14:46:30 +0000

Cloud GPU “scheduling” is a chain of gates: quota decides if you’re allowed to ask, capacity reservations decide if GPUs exist in a zone, placement bins you onto physical hosts, and partitioning decides how many tenants share silicon (full GPU, MIG, time-slicing, vGPU).

Kubernetes sits at the end and consumes whatever the platform exposes.

GPU allocation is a pipeline, not one scheduler

If you can’t name the layer that said “no,” you can’t fix it.

Request (API / YAML)
-> Account quota (by region + purchasing option)
-> Zonal capacity (reservation / capacity block / spot pool)
-> Placement (host + network topology)
-> Partitioning (full GPU | MIG | time-slicing | vGPU)
-> Cluster scheduler (Kubernetes / Slurm)
-> Kubelet + device plugin exposes devices to containers

What each layer controls

This table routes incidents to the right team fast.

Layer	Who controls it	What failure looks like	What to check
Quota	Provider control plane	“quota exceeded” / “limit exceeded”	Instance type quotas + service quotas docs
Capacity	Provider control plane	“insufficient capacity” / stuck provisioning	Capacity Reservations / Capacity Blocks
Placement	Provider control plane	GPUs launch, topology is wrong	Placement groups / cluster strategy
Partitioning	Provider + NVIDIA stack	noisy neighbors / unfair sharing	MIG vs time-slicing vs vGPU
Cluster scheduling	You	Pods Pending	GPU resources + device plugin plumbing

Partitioning decides multi-tenancy behavior

This is where you pick isolation vs utilization.

Mode	What it does	Isolation	What breaks first	Best fit
Full GPU	One workload per GPU	High	utilization	training, big batch
MIG	Hardware partitions with dedicated compute/memory	High	fragmentation by profile	inference, fine-tune with QoS
Time-slicing	Oversubscribe GPUs; workloads interleave	Low	noisy neighbor	burst inference, dev/test
vGPU	Virtual GPU slices via hypervisor stack	Medium–High	licensing + ops	shared VM fleets, VDI, controlled slices

Two blunt truths:

Time-slicing is not memory/fault isolation. It’s interleaving.

MIG is real partitioning with fixed profiles. That creates profile fragmentation if you don’t standardize.

How major cloud providers gate GPU allocation

You can’t “autoscale GPUs” if the provider won’t hand you any.

Amazon Web Services

Quota and capacity are separate checks.

Instance type quotas are grouped by purchasing option (On-Demand, Spot, Dedicated, Capacity Blocks).

Capacity Reservations reserve compute capacity in a specific AZ.

Capacity Blocks for ML reserve GPU instances for a future time window for short-duration ML workloads.

Google Cloud

Reservations are zonal, and they validate capacity up front.

When you create a reservation, Compute Engine verifies capacity in the specified zone, then reserves it.

GPU machine types include A3 variants backed by H100 SKUs (A3 High/Mega/Edge).

Reservation types exist for ensuring optional resources like GPUs are available when you need them.

Microsoft Azure

Quota often shows up as vCPU-family limits plus capacity reservation.

Azure VM quotas are tiered (total regional vCPUs + VM-family cores). If either is exceeded, deployment fails.

Azure capacity reservation reserves compute capacity in a region or AZ for any duration.

ND H100 v5 starts at 8× H100 GPUs per VM (Azure docs).

Kubernetes GPU scheduling is device-plugin driven

The scheduler can’t place what the node doesn’t advertise.

Kubernetes exposes GPUs through device plugins; workloads request resources like nvidia.com/gpu, and the scheduler places Pods on nodes with allocatable capacity.

Full GPU request

This is the baseline contract.

apiVersion: v1
kind: Pod
metadata:
name: gpu-smoke
spec:
restartPolicy: Never
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base
command: ["bash","-lc","nvidia-smi && sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1

MIG request

You request a profile resource, not “a GPU.”

apiVersion: v1
kind: Pod
metadata:
name: mig-infer
spec:
containers:
- name: infer
image: your-infer-image
resources:
limits:
nvidia.com/mig-1g.10gb: 1

MIG is explicitly described as partitioning supported GPUs into isolated instances with dedicated compute/memory.

Time-slicing (oversubscription)

This raises utilization and raises blast radius.

version: v1
sharing:
timeSlicing:
renameByDefault: true
resources:
- name: nvidia.com/gpu
replicas: 10

Time-slicing is documented as oversubscription where workloads interleave on the same GPU.

Installing the NVIDIA stack

You need one canonical way to install drivers + device plugin + toolkit + monitoring.

The NVIDIA GPU Operator automates driver + device plugin + container toolkit + labeling + DCGM monitoring components.

Queue GPUs at the job layer or you’ll drown in Pending Pods

Pods Pending is not a scheduling policy.

Kueue manages quotas and decides when a job waits, when it’s admitted (Pods can be created), and when it’s preempted.

Practical pattern:

One shared cluster.

Separate GPU node pools by workload class.

Kueue ClusterQueues per tenant/team.

Admission control before Pods exist.

GKE publishes a multi-tenant Kueue tutorial that matches this model.

Reference architecture for multi-tenant AI workloads

This is a setup you can run for months without babysitting.

Split pools by workload class

This prevents MIG profile churn from breaking training placement.

Pool	GPU strategy	Controls	Notes
Training	Full GPUs	Kueue + quotas	avoids MIG fragmentation
Inference	MIG	Kueue or HPA	predictable slices
Dev/Test	Time-slicing	loose quotas	accept noisy neighbors

Use taints/tolerations to keep workloads honest

This stops inference from landing on training nodes “because it fit.”

# training nodes
spec:
taints:
- key: gpu-class
value: training
effect: NoSchedule

# training pods
spec:
tolerations:
- key: gpu-class
operator: Equal
value: training
effect: NoSchedule

Where AceCloud.ai fits

You keep the same scheduling stack. You change where the capacity comes from.

AceCloud publishes:

Cloud GPU instances with H100/A100/L40S listed as available options.

Managed Kubernetes GPU clusters as a first-class service offering.

Spot GPU pricing pages with per-hour rates and “saving” percentages by SKU (example: L40S in Mumbai).

Managed control plane claims, including a stated 99.99% uptime SLA on the managed control plane page.

How teams wire AceCloud into multi-tenant scheduling

This is the boring path. It works.

Deploy a managed Kubernetes cluster plus GPU node pools (training vs inference).

Install NVIDIA GPU Operator once per cluster.

Enable MIG on inference pools; keep training pools full GPU.

Add Kueue for tenant quotas and job admission.

Use spot GPUs for interruptible inference/batch where it fits your SLOs.

Ops checklist for debugging “we can’t get GPUs”

This is what you run before you open a ticket.

1) Confirm the provider gate that blocked you

Quota errors and capacity errors look similar in dashboards. They aren’t.

AWS: instance type quotas by purchasing option; capacity reservations / capacity blocks.

GCP: reservations validate capacity at creation time.

Azure: vCPU quota tiers plus capacity reservation.

2) Confirm Kubernetes sees allocatable GPUs

If the node doesn’t advertise it, the scheduler can’t place it.

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
kubectl describe node <node> | grep -E "nvidia.com/gpu|nvidia.com/mig"

Device plugin plumbing is the core dependency here.

3) Confirm you didn’t fragment MIG profiles

MIG failures are often self-inflicted.

If your inference pool is carved into small profiles, large-profile jobs won’t place until you reconfigure the GPU. MIG profiles are fixed.

Conclusion

Cloud providers allocate GPUs through quota + capacity + placement + partitioning before Kubernetes schedules anything. Multi-tenant reliability comes from picking the right sharing primitive: full GPU for training, MIG for isolated inference, time-slicing only for best-effort. Add Kueue so jobs queue instead of stalling Pods. Use AceCloud when capacity gating is your bottleneck, while keeping the same Kubernetes + NVIDIA Operator model.

How to Set Up Edge Infrastructure for Low-Latency Production Apps in India

Daya Shankar — Thu, 12 Feb 2026 05:28:15 +0000

Edge infrastructure India work comes down to one thing: cut round trips.

Put Cloudflare CDN in front, run Cloudflare Workers for routing/auth short-circuits and cache control, keep your origin in AWS Mumbai, and use Redis for hot state.

Measure p95 by city/ISP, then tighten cache keys, warm critical paths, and cap retry storms.

Start with a latency budget you can defend

If you don’t set a budget per hop, you’ll “optimize” the wrong layer.

Define your target like an SRE, not a slide deck:

User SLO: p95 end-to-end latency (and p99 if you have real SLAs).
Breakdown: DNS + TLS + TTFB + payload.
Scope: split by metro and ISP. India is not one network.

Minimum measurement plan

RUM (Real User Monitoring) from browsers/apps. Tag requests with city, asn, isp if your RUM tool supports it.
Synthetics from at least: Delhi NCR, Mumbai, Bengaluru, Chennai, Hyderabad, Kolkata.
Server timing headers from origin so you can isolate backend time vs network time.

What you want to see on a single chart

Edge TTFB (Cloudflare)
Origin TTFB (Mumbai)
Redis time (if used)
App compute time

If you can’t see those separately, you’re flying blind.

Reference architecture: Cloudflare → Workers → AWS Mumbai → Redis

Lock the shape first so each knob has a clear home.

Here’s the stack you picked, with crisp ownership boundaries.

Client (India ISP)
|
| DNS + TLS + HTTP
v
Cloudflare Edge (CDN)
|
| Worker (routing, cache policy, auth short-circuit)
v
AWS Mumbai Origin (ALB/NLB -> app)
|
| hot state / rate limits / sessions
v
Redis (ElastiCache or self-managed)

What runs where (don’t mix this up)

Layer	Do	Don’t
Cloudflare CDN	cache static + cacheable API responses, terminate TLS, absorb spikes	run business logic that needs DB writes
Workers	route, normalize headers, enforce cache keys, cheap auth gates, redirects	call 5 downstream services from the edge
AWS Mumbai origin	serve uncached requests, durable logic, writes	depend on “edge will save us” if origin is slow
Redis	sessions, rate limits, feature flags, hot lookups	treat it like a source of truth

Configure Cloudflare CDN like you mean it

CDN defaults are generic; your production app needs explicit cache rules.

The #1 reason “edge didn’t help” is this: you didn’t make responses cacheable. The difference between a generic CDN setup and a secure CDN solution is intentional cache design and strict isolation.Step 1: Classify endpoints

You can’t cache what you haven’t categorized.

Make three buckets:

Static: JS/CSS/images/fonts. Cache hard.
Semi-static: config, feature flags, catalog, “home feed” variants. Cache with short TTL + SWR.
Dynamic: personalized, writes, payments. No cache.

Step 2: Control cache keys

Bad cache keys destroy hit ratio and spike origin load.

Rules of thumb:

Strip tracking params (utm_*, fbclid, gclid) from cache keys.
Don’t vary on cookies unless you must.
If you must vary, vary on a small whitelist (e.g., plan_tier, locale), not the full cookie blob.

Step 3: Turn on “stale while revalidate” behavior

SWR converts origin spikes into background refresh.

If Cloudflare features are available in your plan, configure:

short TTL for semi-static API responses (e.g., 30–120s)
allow stale serve during refresh

This is how you keep p95 stable during origin deploys and brief Mumbai hiccups.

Step 4: Avoid cache poisoning and auth leakage

One sloppy header can cache private data for strangers.

Hard rules:

Never cache responses that depend on Authorization unless you fully control the cache key and isolation model.
Set Cache-Control: private for truly user-specific payloads.
For “public but user-aware” endpoints, issue explicit cache keys based on a safe token (not raw auth headers).

Use Workers for short-circuits and cache policy, not heroics

Workers are the glue. Keep them small so you can reason for failure.

Workers shine for:

request normalization (headers, query params)
cheap routing decisions
edge auth gating (basic, not deep)
explicit cache behavior via caches.default

A Worker pattern that helps latency

Route and cache at the edge, then fall back cleanly to Mumbai.

export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);

// Normalize cache-busting junk.
["utm_source","utm_medium","utm_campaign","gclid","fbclid"].forEach(p => url.searchParams.delete(p));

// Cheap gate. Block obvious abuse before it hits Mumbai.
const apiKey = request.headers.get("x-api-key");
if (url.pathname.startsWith("/api/") && !apiKey) {
return new Response("missing api key", { status: 401 });
}

// Only cache safe GETs.
if (request.method !== "GET") {
return fetch(new Request(url.toString(), request));
}

// Cache semi-static API endpoints for short TTL.
const isCacheableApi = url.pathname.startsWith("/api/catalog") || url.pathname.startsWith("/api/config");
if (!isCacheableApi) {
return fetch(new Request(url.toString(), request));
}

const cache = caches.default;

// Build a safe cache key. Keep it small.
const locale = request.headers.get("accept-language")?.split(",")[0] ?? "en";
const cacheKey = new Request(url.toString(), {
method: "GET",
headers: { "x-locale": locale }
});

let resp = await cache.match(cacheKey);
if (resp) return resp;

// Fetch origin, then cache it.
resp = await fetch(new Request(url.toString(), request), {
cf: { cacheTtl: 60, cacheEverything: true }
});

// Don’t cache errors.
if (resp.status >= 200 && resp.status < 300) {
ctx.waitUntil(cache.put(cacheKey, resp.clone()));
}
return resp;
}
}

What this does

It routes only what’s safe.
It caches only what you explicitly allow.
It avoids caching auth-bound content.
It keeps origin clean for truly dynamic calls.

Canary Workers safely

Edge bugs are global bugs.

Canary patterns that work:

enable Worker only on a subset of paths
enable by header: x-edge-canary: 1
enable by % rollout based on a stable hash (cookie/session id)

Keep a kill switch. Script it. Don’t rely on “we can revert fast” during an incident.

Redis: keep hot state hot, and make it disposable

Redis saves RTT when used right; it becomes your bottleneck when used wrong.

Put the right data in Redis

Cache the things that are read-heavy and safe to lose.

Good Redis candidates:

session tokens / session metadata
rate limiting counters
feature flags/config snapshots
small lookup tables (tenant → plan, user → segment)

Bad Redis candidates:

“primary database but faster”
large blobs
unbounded key growth

TTL strategy decides p95

TTL is a latency control knob.

Rules:

Use short TTL for rapidly changing data (seconds to minutes).
Use long TTL only if invalidation is correct.
Never ship “no TTL” in a multi-tenant production system unless you enjoy OOM incidents.

Avoid hot keys and stampedes

One hot key can take down your whole cache layer.

Fixes:

shard counters (key:{user}:{bucket}) instead of one global counter
add jitter to TTL to avoid synchronized expiry
use a single-flight pattern (only one origin fetch per key)

Add circuit breakers

When Redis is slow, fail fast and move on.

Set client timeouts.
Cap retries.
If Redis is down, serve from edge cache or hit origin directly depending on endpoint criticality.

Don’t let Redis become a distributed queue by accident.

Harden the AWS Mumbai origin so edge doesn’t mask real slowness

Edge can cut distance; it can’t fix a slow backend.

Connection reuse matters

Most “Mumbai is slow” tickets are handshake overhead plus pool starvation.

Do the basics:

keep-alive between ALB and pods/instances
HTTP/2 where it makes sense
right-size connection pools to DB and Redis

Make origin cacheable too

CDN misses still happen. Origin should be fast on repeat reads.

Add in-process caches for ultra-hot config.
Cache DB reads where correctness allows.
Precompute expensive aggregates.

Scale the right layer

Scaling pods won’t fix a saturated DB pool.

Watch:

request queue depth at load balancer
DB connection wait time
Redis latency percentiles
CPU throttling if you set tight limits

Scale compute only when compute is the constraint. Everything else is noise.

Observability that catches edge failures fast

“Cache hit ratio” is not an SLO.

Track these as first-class metrics:

p95/p99 latency at the edge (client-facing)
origin TTFB for uncached routes
cache hit/miss per route group
Redis p95 latency and error rate
retry rate (gRPC/HTTP clients)
5xx rate at edge and origin

Correlation trick that saves hours

Add a request ID at the edge.
Pass it to origin as a header.
Log it in app + Redis calls.
Now you can grep one request across layers.

Rollout plan that won’t torch production

Edge changes ship fast; that’s good until it isn’t.

A safe rollout looks like this:

Deploy Worker behind a header flag.
Canary with internal traffic and one low-risk route group.
Measure p95 and origin offload.
Scale rollout by % in steps.
Rollback instantly if p95 moves the wrong way.

Also: test from India, not from your laptop in Europe. Pipe synthetic checks from real metros.

India-specific gotchas you should design for

India traffic punishes extra round trips and large payloads.

Common realities:

Mobile networks with variable loss and jitter.
ISPs with inconsistent peering.
DNS resolution variance.

Practical fixes:

keep payloads small (compress, trim JSON, avoid chatty endpoints)
avoid multi-call fanout on the critical path
cache aggressively where safe
fail fast on slow dependencies to protect p99

The build checklist

If you can’t tick these off, you don’t have an edge setup—you have a proxy.

RUM + synthetics split by metro/ISP
Explicit cache rules for static + semi-static endpoints
Worker only does routing/cache/auth short-circuit
Redis keys have TTL, no hot-key stampedes
Origin in AWS Mumbai is tuned for keep-alive and fast reads
Kill switch for Worker rollout
Dashboards show edge p95/p99 + origin TTFB + Redis p95

Managed Cloud Infrastructure: What’s Included, What’s Not, and Why It Matters

Daya Shankar — Thu, 12 Feb 2026 05:11:33 +0000

Managed cloud infrastructure means your provider runs day-2 ops for defined layers patching, monitoring, backups, and incident response while you still own identities, data, application config, and misconfig risk.

Read the responsibility matrix and SLA, then script runbooks and restore tests around the boundary. If the scope is vague, outages drag on.

“Managed” is a scope boundary

If you can’t point to the boundary in writing, you don’t have a managed service.

Cloud providers frame this as shared responsibility: the provider secures the underlying cloud platform; you secure what you deploy and configure on top of it.

What’s included in managed cloud infrastructure

These are the tasks you’re paying to stop doing by hand.

1) Uptime for provider-owned components

The provider should publish an SLA for the layer they run (control plane, storage service, DR service). Don’t assume “whole stack” uptime unless the SLA says so.

2) Patching for the managed layer

Providers typically patch what they own (platform, managed control planes, managed service runtimes). Your OS, node pools, and app dependencies may still be yours unless the contract states otherwise.

3) Monitoring + alerting for their layer

A real managed offering ships:

platform metrics and health checks
alert routing + escalation
a support boundary that says what they touch and what they don’t

4) Backup/DR primitives

You usually get replication and failover mechanics. You still sharpie in app consistency, restore validation, and recovery drills.

5) Change management on their layer

Expect documented maintenance windows, version policy, and an upgrade path for provider-owned components. Managed Kubernetes control planes are common cases.

What’s not included

This is where most “managed” expectations break.

1) Identity and access configuration

You own identities and access policy across cloud models. If IAM is wrong, “managed” won’t save you.

2) Your data and how it’s protected

Providers give encryption features. You choose classification, key custody, access patterns, and retention.

3) Your network intent

Providers run the physical network. You still configure routes, firewall rules, private connectivity, and segmentation. Misconfig here still drops prod.

4) Your application behavior

Providers keep the platform alive. They won’t fix your deployment config, bad queries, or memory leaks.

5) Restore testing (often missed)

Some DR offerings explicitly don’t include routine test drills as a managed feature. If you don’t test restores, you don’t have recovery just storage.

The responsibility matrix you should put in the contract

This is the table you paste into the SOW and grep during incidents.

Layer	Provider owns (typical)	You own (typical)
Datacenter + hardware	power, racks, physical security	nothing
Virtualization	host/hypervisor baseline	guest OS if you run VMs
Managed Kubernetes control plane	API server/etcd/control-plane upgrades	RBAC, admission, policies, namespaces, workloads
Worker nodes	varies by service tier	node OS patching, runtime, add-ons (unless explicitly managed)
Backups/DR engine	replication + orchestration	restore tests, app consistency, recovery validation
Security	“of the cloud” controls	“in the cloud” configuration, identities, data

Why it matters

This is incident math, not procurement fluff.

Faster incident routing

If the boundary is clear, you don’t spend 45 minutes arguing about whose problem it is. You open the right ticket and move on.

Cleaner RTO/RPO planning

Providers can offer targets like <15 min RTO and <5 min RPO for DR, but your app still needs to restore validation and cutover steps you can execute under stress.

Fewer “surprise” costs

Managed services reduce ops toil. They don’t remove engineering work caused by fragile app design or bad change control.

What this looks like with AceCloud.ai

This is how to map “managed” scope to actual service pages and enforceable statements.

Managed Kubernetes Control Plane: states HA operation and a 99.99% uptime SLA for production workloads. Treat that as “provider owns the control plane” in your RACI.
Cloud uptime SLA statement: AceCloud also publishes a 99.99% uptime claim with an explicit downtime math note (“52 minutes per year”). Use that language in your SOW if it matches your scope.
Disaster Recovery service: documents DR orchestration and publishes RTO/RPO claims (including the <15/<5 figures on a replication page). Also calls out limitations like DR test capabilities. Put both the capability and the limitation in your runbook.
Fully managed private cloud: if your requirement is isolation + “someone else runs the platform,” this is the managed-infra pattern without public multi-tenant tradeoffs.

Buying checklist

Ask these questions. Get the answers in writing.

What is the SLA, and which component? (control plane vs nodes vs storage vs network)
Who patches what? (control plane, node OS, CNI, ingress, runtime)
What is the support boundary? (what’s excluded, what voids support, what is “best effort”)
How do backups and restores work? (RTO/RPO, restore steps, restore testing cadence)
How do upgrades work? (maintenance windows, rollback, version policy)

Conclusion

Managed cloud infrastructure works when the boundary is explicit. You outsource day-2 ops for the layers the provider controls control plane upkeep, platform patching, monitoring, and DR mechanics. You still own identities, data protection choices, network intent, and workload config. Put the RACI in the contract, then script restores and incident runbooks around it.