DEV Community: Daya Shankar

Cold Starts, Model Loading, and Their Impact on Latency SLAs

Daya Shankar — Mon, 02 Mar 2026 06:37:05 +0000

Cold start latency breaks SLAs because “pod is Running” isn’t “model is ready.” In Kubernetes hookup with vLLM, cold start includes image pulls, weight downloads, tensor load into GPU memory, and warm-up work (often CUDA-graph-related). These events are rare but huge, so they dominate p95/p99—especially when you scale from zero.

The on-call version of this problem

Bridge: SLAs die on tails, and cold starts are tail generators.

You deploy a new vLLM revision. HPA scales up. Pods come up fast. Traffic shifts. p50 looks fine. p99 explodes.

Nothing “crashed.” You just routed users onto instances still doing model loading and warm-up. That’s not a bug. That’s physics plus orchestration.

If you run strict SLAs on a GPU fleet, you need to treat cold start like a first-class SLI.

What “cold start” actually contains for vLLM on Kubernetes

Bridge: Break the chain into phases so you can measure and fix the slowest link.

Cold start is not one thing. It’s a pipeline:

DIAGRAM 1 — Cold start timeline (what you must budget)

Scale event
|
v
[1] Image pull ---> [2] Container start ---> [3] Model fetch ---> [4] Weight load ---> [5] Warm-up ---> Ready
| | | | |
Registry Python init Network/FS Disk->RAM->GPU Graph/caches

The phase that usually dominates: model storage path

Bridge: If weights sit on slow shared storage, everything else is noise.

vLLM calls this out bluntly: loading large models from shared/network filesystems can be slow, and it’s better to store the model on local disk when possible. It also warns that CPU memory pressure can trigger swapping and slow the OS down.

Translation:

If your weights live on a slow network filesystem, you built a cold-start machine.

If you swap while loading weights, you built a cold-start machine that also hurts neighbors.

Warm-up is real work, not “nice to have”

Bridge: If you don’t pre-warm, the first user request becomes your warm-up job.

vLLM provides tooling specifically to benchmark cold vs warm startup, including model loading and compilation/cache operations.
If vLLM ships a benchmark for startup, that’s your sign: startup cost matters.

Why L40S changes the tuning you should do

Bridge: PCIe-only GPUs expose bad data paths immediately.

On NVIDIA L40S, you’re on PCIe Gen4 x16 (64GB/s bidirectional).
Also: NVLink: no and MIG: no.

What this means for cold starts:

Host↔GPU traffic rides PCIe. Extra copies hurt.

You can’t “hide” cold starts by slicing a big GPU into tiny always-warm MIG slices.

Your operational levers are boring: caching, warm replicas, and reducing churn.

SLA math: cold starts don’truin averages, they ruin p95/p99

Bridge: You can’t hand-wave tails with “but it’s rare.”

Cold starts are low-frequency, high-impact latency events. That’s exactly what percentiles punish.

If you allow scale-to-zero, your probability of cold starts after idle becomes close to 1 for the first request. Knative documents scale-to-zero as a feature and exposes config to enable/disable it.
Knative also documents Scale Down Delay specifically to keep containers around for a configurable time to avoid cold start penalties.

Even if you don’t use Knative, the principle holds:

Every time you delete a pod, you re-pay model load.

Every time you scale to zero, you guarantee a cold start on the next burst.

Fix cold start latency by attacking three things

Bridge: You reduce cold starts by moving fewer bytes, repeating less work, and avoiding scale-to-zero surprises.

1) Cache model artifacts on the node (prefer local disk)

Bridge: Put the bytes next to the GPU node or pay for network + FS latency on every churn event.

vLLM recommends local disk when shared filesystems are slow.
So do this:

Mirror model artifacts to a controlled location (object store, internal registry, or artifact repo).

Cache on node-local SSD/NVMe where possible.

Point vLLM/HF cache directories at that local path.

Practical rule for SREs: don’t download weights from the public internet in the hot path. vLLM itself recommends downloading first (via huggingface-cli) and passing the local path to isolate issues.

2) Pre-pull images on GPU nodes

Bridge: Image pulls are pure waste during a scale event.

Use a DaemonSet that pins to GPU nodes and pulls your serving image. Keep it dumb.

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: vllm-prepull
namespace: kube-system
spec:
selector:
matchLabels: { app: vllm-prepull }
template:
metadata:
labels: { app: vllm-prepull }
spec:
nodeSelector:
accelerator: nvidia
tolerations:
- key: "accelerator"
operator: "Equal"
value: "nvidia"
effect: "NoSchedule"
containers:
- name: sleep
image: your-registry/vllm-serving:TAG
command: ["sh","-c","sleep 365000"]
resources:
requests: { cpu: "10m", memory: "32Mi" }
limits: { cpu: "50m", memory: "64Mi" }

3) Keep a warm floor for SLA-bound services

Bridge: If your SLA can’t tolerate cold starts, don’t scale to zero.

Set:

min replicas > 0 (HPA floor or Deployment replicas)

a “warm pool” per model

separate burst capacity if you need it

Scale-to-zero is a cost tool. It is not an SLA tool. Knative’s own docs bake in knobs to control that behavior for a reason.

Two diagrams that match how this should be deployed

Bridge: These are the shapes that keep p99 sane without paying for always-on waste.

Reference deployment YAML (vLLM on L40S with readiness gating + node-local cache)

Bridge: This is the “copy/paste then edit” block you can review in PRs.

This example does three things:

pins onto GPU nodes

caches model files on a node-local path

gates readiness until a warm-up completes

Important: This assumes you control the container entrypoint and can add a small wrapper script. That’s the cleanest way to tie readiness to “model is hot.”

apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-serve
namespace: inference
spec:
replicas: 2 # warm floor for SLA
selector:
matchLabels:
app: vllm-serve
template:
metadata:
labels:
app: vllm-serve
spec:
nodeSelector:
accelerator: nvidia
gpu: l40s
tolerations:
- key: "accelerator"
operator: "Equal"
value: "nvidia"
effect: "NoSchedule"
containers:
- name: vllm
image: your-registry/vllm-serving:TAG
ports:
- containerPort: 8000
resources:
requests:
cpu: "4"
memory: "24Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
env:
- name: HF_HOME
value: /models/hf
- name: MODEL_PATH
value: /models/hf/my-model # pre-downloaded or mirrored
command: ["/bin/bash","-lc"]
args:
- |
set -euo pipefail
rm -f /tmp/ready

# Start vLLM in background
vllm serve "${MODEL_PATH}" \
--host 0.0.0.0 --port 8000 \
--dtype auto \
--max-model-len 8192 \
--tensor-parallel-size 1 \
2>&1 | tee /var/log/vllm.log &
VLLM_PID=$!

# Wait for the server socket, then trigger a warm-up request.
# Replace the warm-up call with your own internal probe if needed.
for i in {1..120}; do
(echo > /dev/tcp/127.0.0.1/8000) >/dev/null 2>&1 && break
sleep 1
done

# Minimal warm-up: hit the server once (your internal client/probe here).
# If you can’t curl the API, run a lightweight local script instead.
curl -sf http://127.0.0.1:8000/ >/dev/null || true

# Mark ready only after warm-up.
touch /tmp/ready

# Keep foreground
wait $VLLM_PID
readinessProbe:
exec:
command: ["/bin/sh","-c","test -f /tmp/ready"]
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 30
startupProbe:
exec:
command: ["/bin/sh","-c","test -f /var/log/vllm.log"]
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 300 # allow long first load
volumeMounts:
- name: model-cache
mountPath: /models
volumes:
- name: model-cache
hostPath:
path: /var/lib/model-cache
type: DirectoryOrCreate

Notes for senior SREs

hostPath is powerful and dangerous. In a managed Kubernetes environment, you may prefer node-local ephemeral SSD mounts that the platform team controls, or a LocalPV setup with strict node affinity.

Set replicas to your SLA floor. Use HPA for burst, but don’t let it go to zero if p99 matters.

Measure it like an SRE: phase timings and startup benchmarks

Bridge: You can’t improve what you can’t attribute.

Use vLLM’s startup benchmark tooling

Bridge: Benchmark cold vs warm startup and block regressions.

vLLM ships a startup benchmark module to measure cold and warm startup times, including model loading and compilation/cache operations.

Run it against:

your container image

your model

your storage backend

your L40S node class

Then fail CI when cold start time regresses.

Log phase timestamps

Bridge: Turn “it’s slow” into numbers you can grep.

Log these timestamps per pod:

image pulled (node event)

process start

model fetch complete

weights loaded

warm-up complete

readiness true

Then build histograms:

cold_start_seconds{phase="fetch"}

cold_start_seconds{phase="load"}

cold_start_seconds{phase="warmup"}

This tells you where to spend effort.

Managed Kubernetes: what it helps, what it doesn’t

Bridge: Managed Kubernetes runs the plumbing. You still own the SLA path.

Managed Kubernetes can:

keep control plane stable

manage node lifecycle and autoscaler hygiene

standardize storage classes and node pools

It will not:

pick your cache strategy

keep models warm for your SLA

prevent you from scaling to zero and cold-starting on every burst

On AceCloud managed Kubernetes, the clean play is: dedicated GPU node pools for vLLM, pre-pull images, cache weights on node-local storage, set warm floors for SLA services, and Script warm-up into readiness. Keep your cold path measured and boring.

Checklist for PR reviews

Bridge: This is the short list that prevents “p99 spikes after deploy.”

Model artifacts are local or cached. Not pulled from the public internet at runtime.

GPU node pools are pinned (L40S), tainted, and isolated.

Image pre-pull exists on GPU nodes.

Readiness gates on “model is hot,” not “process started.”

Warm floor exists (min replicas > 0) for SLA services.

Cold vs warm startup is benchmarked in CI.

If you want, I can also add a short Prometheus section (cold-start phase histograms + alert rules) so the on-call page tells you which phase is burning your SLA.

Operational Risks of Running Large Multi-Tenant Kubernetes Clusters

Daya Shankar — Mon, 02 Mar 2026 06:30:39 +0000

Large multi-tenant Kubernetes clusters concentrate risk. Tenants share the control plane, core add-ons (CNI/CSI/Ingress/DNS), and scheduling capacity, so one bad deployment or “safe” upgrade can hit everyone.

The common failures are noisy neighbors, weak isolation, quota starvation, identity drift, and upgrade blast radius. Managed Kubernetes helps, but it won’t design tenancy for you.

What “multi-tenancy” means when you’re on call

If you don’t define the tenant boundary, you can’t defend it.

Multi-tenant Kubernetes usually means “many teams share one cluster.” The boundary is often a namespace. Sometimes it’s stronger: separate node pools, stricter network policy, workload identity, dedicated ingress, dedicated GPUs, etc.

Operationally, multi-tenancy is shared failure domains:

One API server.

One DNS stack (CoreDNS).

One CNI and conntrack table.

One CSI and storage path.

One ingress layer (or a few shared controllers).

One scheduler and one pool of allocatable capacity.

If you want a cluster to survive at scale, you need to decide which failures are allowed to be shared and which are not.

Noisy neighbors aren’t a “performance issue” they’re an outage pattern

Shared capacity turns small mistakes into cluster-level incidents.

CPU: throttling, request inflation, and scheduler lies

CPU is compressible, so people abuse it.

Two classic problems:

No limits + bursty workloads → one tenant burns cores and everyone’s latency climbs.

Overstated requests → scheduler thinks nodes are full → cluster autoscaler spins up nodes → real CPU sits idle.

If you size everything to p95 requests, you don’t just waste money. You also block bin-packing and create “Pending pods” incidents that look like infra failure.

Minimum guardrail

Enforce requests on CPU.

Be cautious with CPU limits for latency-sensitive services (throttling is real).

Use HPA with a real scaling signal. Don’t “set and forget.”

Memory: eviction storms and node death spirals

Memory is not compressible; it fails hard.

One tenant can trigger:

node memory pressure

kubelet evictions

cascading restarts

thundering herds as pods all re-pull images and rebuild caches

Minimum guardrail

Set memory requests and limits for all tenant workloads.

Alert on OOMKilled and eviction rates per namespace.

Keep headroom on nodes so eviction doesn’t become a cluster-wide reboot loop.

Disk/inode: the silent killer

Disk fills don’t page until they page everybody.

Common multi-tenant disk failures:

log storms filling /var/log or container runtime storage

inode exhaustion from small-file spam

image cache churn under high pod turnover

Minimum guardrail

Per-namespace log volume controls (don’t let one team spam logs).

Node alerts on disk/inode usage.

Runtime storage quotas where available.

Network: saturation and conntrack exhaustion

Networking failures hit all tenants because the kernel tables are shared.

When one tenant opens too many connections or you get a traffic spike:

conntrack table fills

packets drop

“random” timeouts appear across unrelated services

Minimum guardrail

Rate-limit at ingress.

Enforce egress policies.

Watch conntrack, dropped packets, and retransmits on nodes.

Isolation failures become security incidents

Namespaces are a convenience boundary, not a security boundary.

RBAC drift and privilege creep

RBAC starts clean and rots fast in shared clusters.

The failure mode is predictable:

a team needs one permission

someone grants a broad ClusterRole

later, nobody remembers it exists

now the tenant can list secrets cluster-wide or mutate critical resources

Minimum guardrail

Centralize ClusterRole creation.

Lint RBAC in CI (Script it; don’t “review in Slack”).

Ban wildcard verbs/resources for tenant roles.

Workload identity and cloud IAM misbinding

The fastest way to leak data is to bind the wrong identity to the right pod.

In multi-tenant, identity mistakes propagate:

a shared service account gets reused

a workload identity binding is too broad

pods gain access to buckets/queues they should never see

Minimum guardrail

One workload identity per service, not per namespace.

Deny “default” service account usage for real workloads.

Audit “who can assume what” regularly and pipe it to alerts.

Pod security exceptions that never die

The exception list grows until it becomes the policy.

If you allow privileged pods, hostPath mounts, or host networking for one team, you’ve opened a side door for everyone unless you lock it down.

Minimum guardrail

Use Pod Security Admission (baseline/restricted) as default.

Require an exception workflow with expiry.

Grep for privileged/hostPath usage weekly.

Network policy gaps turn “one bad app” into “everyone is down”

Flat networks are how tenant bugs become tenant outages.

Default-allow is the default failure

If everything can talk to everything, blast radius is automatic.

A single noisy service can:

hammer shared dependencies

trigger thundering herds

overload DNS

spike cross-namespace traffic

Minimum guardrail

Default-deny egress and ingress per namespace.

Explicit allowlists for shared services.

Treat policies like code (PRs, review, tests).

Shared ingress controllers amplify mistakes

One bad ingress change can break unrelated tenants.

Failure patterns:

config reload loops

bad annotations triggering expensive behaviors

certificate mis-rotation

Minimum guardrail

Separate ingress controllers by tenancy tier (shared/dev vs prod/critical).

Canary ingress changes.

Enforce annotation allowlists.

DNS is a shared single point of failure

CoreDNS is the cluster’s heartbeat; overload it and nothing resolves.

In big clusters, DNS load grows with:

pod count

churn

retries during incidents

Minimum guardrail

Scale CoreDNS for QPS and cache settings.

Alert on CoreDNS latency/errors.

During incidents: Grep logs for upstream timeouts and SERVFAIL.

Scheduling and quota pathologies at scale

“Fair scheduling” is policy you must configure, not something Kubernetes gifts you.

Quota starvation and priority inversion

One tenant can starve others without “breaking rules.”

Common patterns:

Tenant A uses up shared node pool capacity with big requests.

Tenant B is within quota but can’t schedule due to fragmentation.

Everyone blames the scheduler. It did what you told it.

Minimum guardrail

ResourceQuotas per namespace (CPU/memory/pods).

LimitRanges to prevent “no requests” and “ridiculous limits.”

Separate node pools for noisy/bursty tenants.

Preemption can save prod or murder batch

Preemption is a knife. Use it like one.

If you enable priority + preemption:

your critical services can recover capacity

your batch jobs can get killed repeatedly unless they checkpoint

Minimum guardrail

PriorityClasses: “prod-critical”, “prod”, “batch”.

For batch: checkpoint or accept loss.

Measure eviction rates after enabling.

Upgrade and change-management risk is multiplied by tenant count

In big clusters, “safe changes” are the biggest outage source.

The shared add-ons are the sharp edges:

CNI upgrades

CSI upgrades

ingress controller upgrades

API deprecations breaking controllers/operators

node patching + drains deadlocking on PDBs

Failure mode you will hit: PDB deadlock.
A drain starts, pods can’t evict due to strict budgets, the rollout stalls, capacity shrinks, and unrelated tenants get squeezed.

Minimum guardrail

Canary upgrades in a smaller cluster or a dedicated pool.

Script rollback paths.

Set realistic PDBs (protect availability, don’t freeze the cluster).

Observability and incident response get harder as the cluster gets bigger

If you can’t attribute load to a tenant, you can’t run multi-tenant.

Per-tenant attribution is mandatory

“The cluster is slow” is not an actionable alert.

You need:

dashboards by namespace (CPU/mem/network/disk)

request rates at ingress by tenant

top talkers (network) and top allocators (memory)

Cardinality will bite you. Don’t ship every label. Decide which labels you can afford, then enforce it.

Audit logs and “who did what”

Multi-tenant incidents often start as “someone applied something.”

Enable audit logs and make them searchable. When you’re debugging a weird outage, you should be able to answer:

who changed the Deployment?

who changed the NetworkPolicy?

who updated the ingress?

Managed Kubernetes changes the risk profile, not the physics

Managed Kubernetes reduces control-plane toil, not tenant blast radius.

Managed Kubernetes usually helps with:

control plane uptime/patching

some upgrade orchestration

basic integrations

It does not automatically give you:

tenant isolation

safe defaults for quotas and policies

sane RBAC boundaries

disciplined change control for shared add-ons

If you’re on a managed Kubernetes offering like AceCloud, use the managed layer for what it’s good at (platform plumbing), then enforce tenancy guardrails at the cluster policy layer (quotas, PSA defaults, network policies, and tiered node pools). That’s where multi-tenancy succeeds or fails.

Mitigation playbook (week 1 controls that actually reduce incidents)

These are the controls you can deploy fast and feel immediately.

1) Create tenancy tiers

Not every workload deserves to share the same failure domain.

Shared/dev tier: many tenants, lower guarantees

Prod/shared tier: stricter policies, more guardrails

Prod/dedicated tier: separate node pools or separate clusters for the truly critical

2) Enforce default-deny networking

Flat networks are the default blast radius.

Deploy default-deny policies per namespace. Add explicit allow rules.

3) Lock down RBAC and pod security

Security drift is guaranteed unless you block it.

central RBAC templates

Pod Security Admission defaults

expiring exceptions

4) Quotas + LimitRanges everywhere

Multi-tenant without quotas is “first team to deploy wins.”

ResourceQuota per namespace

LimitRange to prevent “no requests” and “unbounded limits”

Alerts on quota saturation and Pending pods

5) Safer change management for shared components

Your add-ons are shared infrastructure. Treat them like prod.

canary upgrades

rollback scripts

PDB sanity checks before drains

runbooks for CNI/CSI/Ingress/DNS failures

Bottom line

Large multi-tenant clusters work when you treat them like a shared operating system.

A big shared Kubernetes cluster isn’t “just more nodes.” It’s a bigger shared failure domain. The operational risks are predictable: noisy neighbors, weak isolation, quota starvation, identity drift, and upgrade blast radius.

If you want reliability, you must Configure guardrails, Script rollouts, and verify attribution per tenant whether you run it yourself or on managed Kubernetes.

Hosted control plane: when it simplifies operations and when it adds complexity

Daya Shankar — Fri, 27 Feb 2026 06:13:52 +0000

A hosted control plane moves Kubernetes control-plane components off your worker fleet either into a provider-managed boundary (EKS) or onto a separate hosting cluster as pods (HyperShift).

It simplifies ops when you want predictable upgrades, less per-cluster snowflake work, and cleaner separation between “management” and “workloads.”

It adds complexity when control-plane connectivity, IAM, and shared blast radius become your new failure modes especially with private clusters.

Define hosted control plane in concrete terms

If you can’t say where the API server and etcd live, you can’t model risk.

“Hosted control plane” is a placement decision.

EKS: hosted by AWS in an EKS-managed VPC

AWS owns the masters; you own nodes and workloads.

AWS documents that the EKS-managed control plane runs inside an AWS-managed VPC and includes Kubernetes API server nodes and an etcd cluster. API server nodes run in an Auto Scaling group across at least two AZs; etcd nodes span three AZs.

What that means operationally:

You don’t patch control-plane instances.

You don’t rebuild etcd.

You do still own access, RBAC, node lifecycle, and add-ons.

kubeadm on EC2: not hosted, you host it

You run the masters, the etcd, the upgrades, and the recovery drills.

Kubeadm HA requires you to pick a topology (stacked etcd vs external etcd) and wire up the endpoints (often via a load balancer DNS name). External etcd needs explicit endpoint configuration; stacked etcd is “managed automatically” by kubeadm’s topology.

What that means operationally:

You patch and upgrade the control plane.

You own etcd snapshots and restore tests.

You own certificates and rotation edge cases.

HyperShift (hosted control planes): control planes as pods on a hosting cluster

You consolidate many control planes onto one management cluster.

Red Hat’s hosted control planes model runs control planes as pods on a management/hosting cluster, without dedicated VMs per control plane.

HyperShift then introduces a new question: where do those control plane pods land? Docs show “shared everything” by default, and you can dedicate nodes for control plane workloads via labels/taints.

Side-by-side: what gets simpler, what gets harder

Feature lists lie. Ownership and failure modes don’t.

Model	What simplifies	What gets harder	The new “pager line”
EKS hosted control plane	Control plane HA, scaling, replacement; less etcd babysitting	Endpoint access + SG design for private clusters; version planning	“Can we reach the API endpoint from the right networks?”
kubeadm on EC2	Full control; no managed constraints	Everything: HA wiring, etcd ops, upgrades, certs	“etcd is sick” is your incident
HyperShift	Reduce per-cluster control-plane VMs; faster cluster churn; multi-tenant mgmt	Hosting cluster becomes shared blast radius; two-layer debugging	“Hosting cluster health” pages everyone

When a hosted control plane simplifies operations

Hosted control planes help when your bottleneck is “running too many control planes.”

1) You operate many clusters (multi-tenant SaaS, env sprawl)

Cluster count is the multiplier.

If you run 20+ clusters, self-managed control planes become a tax:

patch windows multiply

certificate and etcd risk multiplies

“one-off cluster drift” becomes normal

EKS removes the control plane instances from your fleet and gives you a standardized control plane architecture across AZs.

HyperShift goes further: it removes dedicated control-plane machines per cluster and runs them as pods on a hosting cluster.

2) You want predictable control-plane availability without building an etcd practice

etcd is not hard until it’s hard at 3 AM.

kubeadm HA docs are clear: external etcd adds configuration surface area (explicit endpoints); stacked etcd is simpler but still your operational problem.

If your team doesn’t want to own etcd restores as a practiced drill, a hosted control plane removes that class of work from your team’s backlog.

3) You need fast cluster create/delete (ephemeral clusters, tenant clusters)

Provisioning speed is operational leverage.

HyperShift is designed around the concept of creating control planes as pods on a management cluster, which reduces the need to “spin up” dedicated control-plane machines per hosted cluster.

That’s useful when:

you create short-lived clusters for CI

you provision tenant clusters and churn them

you want cluster lifecycle to look like Deploying an app

4) You’re private-cluster-heavy and want a supported endpoint model

Private changes the operational shape more than any “feature.”

EKS lets you run a private-only API server endpoint (no public access), where kubectl must come from within the VPC or connected networks. Access to the private endpoint is controlled by rules on the cluster security group.

That’s not “simpler” in absolute terms. It’s simpler because it’s a supported, documented pattern with fewer moving parts than self-hosting your own API endpoint VIP/LB and cert story.

When a hosted control plane adds complexity

You trade “masters on VMs” for “network + IAM + shared blast radius.”

1) Control-plane connectivity becomes a first-class dependency

The API server is now “across a boundary,” and boundaries fail.

With EKS private-only clusters:

your kubectl, CI runners, and controllers must live inside the VPC or connected networks

your security group rules become part of cluster availability

With public endpoint access, the default behavior has historically been public enabled / private disabled (and you can toggle both).
Either way, endpoint mode is now a design choice you must document, test, and audit.

What changes for on-call:

“API is down” might really be “route to endpoint is broken”

DNS, TGW/peering, SG rules, and client network become suspects

2) Identity boundaries get sharper (and easier to misconfigure)

Hosted control planes push you into “who can reach what” decisions.

Private endpoint + security group control is good. It’s also easy to get wrong:

over-broad SG rules turn “private endpoint” into “private but reachable from everything”

too-tight rules break controllers and CI/CD in weird ways

Hosted doesn’t remove IAM work. It moves it to the center of the blast radius.

3) HyperShift’s hosting cluster becomes shared infrastructure

You didn’t delete control planes. You consolidated them.

HyperShift runs control planes as pods on a hosting cluster.
Docs show that hosted control plane pods can be scheduled broadly (“shared everything”), and you can taint/label nodes to dedicate capacity.

This is the operational trade:

Pro: fewer dedicated control-plane machines per tenant cluster

Con: hosting cluster saturation, upgrades, or outages can hit multiple hosted clusters at once

If you adopt HyperShift, treat the hosting cluster like tier-0 infrastructure:

separate node pools

aggressive monitoring

strict change control

tested disaster recovery

4) Debug becomes two-layer

Symptoms show up in the guest cluster; root cause can live elsewhere.

With EKS, control plane is managed. You troubleshoot via endpoint reachability, AWS telemetry, and cluster behavior. You can’t SSH into masters, and that’s the point.

With HyperShift, you can often inspect control plane pods on the hosting cluster. That’s powerful and it means your runbooks must cover two clusters:

guest cluster symptoms

hosting cluster root cause

Private clusters: the “hosted” decision that matters most

Private mode turns networking into part of the control plane.

EKS private endpoint: supported, but policy-heavy

SG rules are now part of cluster uptime.

AWS states that for private-only API servers:

there is no public access from the internet

kubectl must come from the VPC or connected network

cluster security group rules control private endpoint access

This is clean if you already run:

TGW / VPC peering / Direct Connect

private DNS resolution patterns

locked-down egress

It’s messy if your ops tooling lives outside the network boundary and you aren’t ready to move it.

kubeadm private: you own the endpoint and its failure modes

You don’t get a managed endpoint; you build one.

kubeadm HA guides assume you Configure a load balancer in front of the control plane nodes and wire up DNS names and endpoints.

That’s flexible. It’s also more work:

API endpoint LB health checks

TLS/cert rotation

routing changes during upgrades

HyperShift private: you design exposure between hosting and guest clusters

Hosted control planes still need reachable endpoints.

Hosted control plane pods live on the hosting cluster. That’s good for consolidation. It also means you must design:

how guest nodes reach the hosted API server

how admins reach it (private networks, bastions, CI runners)

how you segment tenants

The exact networking patterns vary by environment, but the invariant is: private hosted control planes increase the importance of network design.

Terraform: what you actually manage in each model

IaC doesn’t disappear. The resource graph changes.

EKS Terraform surface area

You configure endpoint modes, SGs, node groups, and IAM.

Minimum Terraform concerns:

endpoint access mode (public/private/both)

cluster security group rules for private access

node groups and AMI strategy

IRSA and IAM boundaries

Hosted control plane simplifies the “masters” part. It does not simplify the access-control part.

kubeadm Terraform surface area

Terraform becomes your control-plane installer, not just a cluster creator.

You end up managing:

control plane EC2 instances

LB/VIP in front of API servers (common HA pattern)

etcd instances (external) or colocated etcd (stacked)

bootstrap scripts, cert distribution, upgrade workflows

This can be clean if you have mature automation. If not, it’s a lot of state to keep consistent.

HyperShift Terraform surface area

You manage the hosting cluster like a platform, then declaratively create hosted clusters.

HyperShift adds:

hosting cluster lifecycle (upgrade, capacity, resilience)

hosted cluster objects and their infra mappings

scheduling policies for control plane pods (dedicated nodes via labels/taints)

Terraform can drive parts of this, but you’ll also lean on cluster-native controllers.

Prometheus: what you need to watch so hosted doesn’t surprise you

Hosted control planes move failure modes. Your dashboards must follow.

At minimum, split monitoring into two planes:

Workload plane (guest cluster apps)
request rates, latency, errors
node saturation
queue depth / retries
Control plane plane
API server availability/latency from where your clients run
controller health signals
for HyperShift: hosting cluster resource pressure, because control planes are pods

For private clusters, add synthetic checks from the networks that matter:

from CI runner network

from admin network

from in-cluster controllers

If the API endpoint is unreachable from your automation network, you don’t have a cluster. You have a museum exhibit.

Decision checklist for SaaS and platform teams

Answer these honestly and the right model usually falls out.

How many clusters will you run in 12 months?
If the number is growing fast, hosted control plane saves toil.
Do you have an etcd practice?
If “restore drill” isn’t something you run quarterly, kubeadm HA is a risk trade.
Is private-only mandatory?
If yes, model endpoint reachability and SG rules as part of uptime.
Can you tolerate shared blast radius?
HyperShift consolidates control planes. Treat hosting cluster as tier-0.
What do you want to debug at 3 AM: VMs or networks?
kubeadm tends toward VM-level debugging.
hosted control planes tend toward network/identity debugging.

Where AceCloud fits

Hosted control plane only helps if the day-2 loop is owned and scripted.

If you’re buying hosted control plane benefits but don’t want to run the surrounding ops (endpoint policies, Terraform hygiene, Prometheus wiring, upgrade runbooks), a managed Kubernetes provider like AceCloud can own that platform loop while your team focuses on workload correctness and SLOs.

Bottom line

Hosted control plane is not “less complexity.” It’s different complexity.

Pick a hosted control plane (EKS) when you want AWS to own control plane HA, scaling, and replacement across AZs.
Pick kubeadm when you need maximum control and you’re willing to own HA topology, etcd ops, and endpoint plumbing.
Pick HyperShift when you need to run many clusters and you’re ready to operate a tier-0 hosting cluster that runs control planes as pods.

The correct choice is the one that gives every failure mode a clear owner—and keeps your pager quiet for the right reasons.

Serving LLMs on IaaS: throughput vs latency tuning with practical guardrails

Daya Shankar — Fri, 27 Feb 2026 06:11:05 +0000

Serving LLMs on IaaS is queueing plus memory pressure dressed up as ML. Every request has a prefill phase (prompt → KV cache) and a decode phase (token-by-token output).

Throughput tuning pushes batching and concurrency. Latency tuning caps them to protect TTFT and ITL. With vLLM on a single L40S (PCIe), you win by setting hard limits and enforcing admission control.

TTFT, ITL, TPS: stop mixing the metrics

If you tune the wrong metric, you’ll ship a fast benchmark and a slow product.

You need three numbers, and they mean different things:

TTFT (time to first token): how long the user waits before anything shows up. Interactive UX lives here.
ITL (inter-token latency): the “smoothness” of streaming output once decoding starts. Chat feels broken when this jitters.
Throughput (tokens/sec): the finance metric. It decides cost per request.

One important detail: E2E latency includes queueing + prefill + decode. TTFT is where queueing hides when you’re overloaded.

Practical measurement rule: measure TTFT and ITL at the client (or gateway), not inside the GPU server. Internal timings miss queueing in front of vLLM.

Hardware reality check: single L40S on PCIe

You can’t tune around a bus you don’t have.

An L40S is a strong inference GPU, but it’s not an NVLink box. It’s 48GB GDDR6 on PCIe Gen4 x16.
That matters because:

You have one GPU’s worth of memory for weights + KV cache.
You don’t get multi-GPU model parallel tricks for free.
Your main enemies are KV-cache pressure and batch/concurrency overshoot, not “GPU topology.”

On a single GPU server, latency failures usually look like:

TTFT spikes because the prefill queue grows.
ITL spikes because decode gets starved or the batch gets too big.
OOM/restarts because KV cache math was wishful thinking.

vLLM’s default behavior: TTFT-first scheduling (and the trade)

vLLM already picks a side; your job is to set guardrails around it.

By default, vLLM’s scheduler prioritizes prefills and does not batch prefill and decode into the same batch. That typically optimizes TTFT, but can worsen ITL and GPU utilization.

Translation: out of the box, vLLM tries to be responsive. You can still break it by feeding it mixed traffic with no limits.

The knobs that actually move TTFT, ITL, and OOM risk

You don’t “optimize latency.” You Configure concurrency and KV-cache headroom.

These four knobs do most of the work in production vLLM.

1) --max-num-seqs caps concurrency

This is your “how many requests can be active” ceiling.

--max-num-seqs is the maximum number of sequences per iteration.
Lowering it:

reduces concurrent KV cache usage
reduces queue contention inside the engine
usually helps tail latency (until you underutilize the GPU)

2) --max-num-batched-tokens controls batch size per iteration

This is where you trade throughput for TTFT/ITL stability.

--max-num-batched-tokens limits batched tokens per iteration.
Lowering it:

reduces “one huge prefill” events
reduces KV cache demand per cycle
can protect TTFT and prevent decode jitter

Raising it:

can increase throughput
can increase queueing and tail spikes if your traffic is bursty or prompts are long

3) --gpu-memory-utilization sets KV-cache headroom

This decides how much VRAM vLLM pre-allocates for cache.

vLLM pre-allocates GPU cache using gpu_memory_utilization. Increase it to provide more KV cache space.
If you set it too high, you risk fragmentation and less room for everything else. If you set it too low, you’ll hit KV cache limits early and TTFT will spike under concurrency.

4) --enable-chunked-prefill tames long prompts

Long prompts are TTFT killers; chunking makes them less explosive.

When enabled, vLLM can chunk prefill requests based on max_num_batched_tokens.
This is a practical guardrail when you can’t control prompt length perfectly.

A sane starting config for your SLA (p95 TTFT 250ms, p99 800ms)

Start conservative, hit the TTFT target, then earn throughput back.

On a single L40S, don’t begin with “maximum throughput.” Begin with “stable TTFT.”

Example vllm serve baseline (single GPU):

vllm serve /models/your-llm \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 64 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill

Why these shapes:

max_num_seqs prevents unlimited concurrency blowups.
max_num_batched_tokens prevents one batch from ballooning.
gpu_memory_utilization keeps cache headroom explicit.
chunked prefill reduces “one giant prompt ruins the minute.”

You will tune these. But you need a stable base first.

Practical guardrails for mixed chat + batch traffic

Throughput tuning is easy. Guardrails are what keep p99 alive.

Mixed traffic (interactive + batch) is where systems get weird. Batch clients tend to:

send long prompts
request long generations
retry aggressively
keep load constant

Interactive chat needs:

fast TTFT
consistent ITL
predictable tail behavior

So you need admission control in front of vLLM. Not “best effort.”

Guardrail table (start here)

These caps stop one client from torching everyone else.

Guardrail	Default starting point	Why it exists
Max prompt tokens	4k–8k (per request)	Long prefills blow TTFT and batch size
Max output tokens	256–512 (interactive), 1k+ (batch)	Protect tail latency for chat
Max in-flight requests	Tie to max_num_seqs	Prevent internal queue explosion
Max queue depth	1–2× in-flight	If queue > that, reject/429 fast
Request timeout	Slightly above p99 target	Don’t let zombie requests clog decode
Retry policy	capped + jitter	Stops retry storms multiplying load

These aren’t theoretical. They’re how you keep a single GPU server usable.

Two-lane routing (interactive vs batch)

If you mix traffic in one FIFO queue, batch wins and chat loses.

On one GPU, the clean pattern is two lanes at the gateway:

Interactive lane: strict caps (short prompts, short outputs), low queue depth.
Batch lane: looser caps, but it yields when interactive is busy.

You can implement this with a thin gateway that:

inspects request size (prompt tokens + requested output tokens)
routes “interactive” to the main lane
routes “batch” to a background lane with stricter admission

Even if both lanes hit the same vLLM backend, the queue policy changes outcomes.

Concrete rule that works:
If interactive queue depth > N, reject batch (429) instead of letting it sit and inflate TTFT.

The tuning loop that converges (without cargo cult)

Tune one knob at a time and measure TTFT and ITL separately.

Here’s the loop to run on a GPU cloud server before you call it “production.”

Step 1: Fix the workload mix

Your traffic generator must match reality.

Build two test profiles:

Chat: short prompts, short outputs, bursty concurrency.
Batch: longer prompts and outputs, steady concurrency.

If you benchmark only one, you’ll tune only one.

Step 2: Lock SLOs first

You already have targets; enforce them.

Targets:

TTFT p95 ≤ 250ms
TTFT p99 ≤ 800ms

Keep a red line on the dashboard. If a tuning change crosses it, roll back.

Step 3: Set limits, then raise carefully

Earn throughput; don’t steal it from p99.

Order of operations:

Set max_num_seqs low enough that you never OOM under your worst prompt mix.
Set max_num_batched_tokens to prevent giant prefills from blocking decode.
Adjust gpu_memory_utilization to give KV cache room.
Enable chunked prefill if long prompts exist in real traffic.

Then:

increase max_num_seqs until TTFT p95 hits the edge of your budget
increase max_num_batched_tokens only if ITL stays stable and TTFT doesn’t spike

Step 4: Add overload behavior on purpose

A good system fails fast, not slowly.

Define overload mode:

when queue depth exceeds threshold → return 429 with Retry-After
when prompt/output exceeds limits → return 400 with a clear message
when batch lane is busy → shed batch first

If you don’t define this, your system will “define it” by melting.

Dashboards that catch trouble before users do

You can’t grep production. You need signals that predict tail spikes.

Track:

TTFT p50/p95/p99 (interactive lane, batch lane)
ITL distribution (interactive lane)
queue depth and reject rate (the guardrail is working if it fires)
GPU memory usage and cache pressure (OOM risk proxy)

vLLM already frames TTFT/ITL as the core performance story, and its scheduler tradeoffs explain why TTFT can look good while ITL suffers (or vice versa).

Where AceCloud fits (one honest paragraph)

IaaS isn’t the problem; inconsistency is.

If you’re serving on an IaaS gpu cloud server from a provider like AceCloud, treat it like any other VM: bake a known image, pin driver/CUDA versions, and script your vLLM flags so every node behaves the same. The tuning work above only sticks when the box is predictable.

Bottom line

Throughput is what you brag about. Latency is what users feel.

On vLLM + single L40S, you don’t win by chasing max tokens/sec. You win by controlling concurrency and batch size, allocating KV cache intentionally, and enforcing guardrails that keep mixed traffic from turning into a queueing disaster. Hit TTFT p95/p99 first. Then scale throughput without stealing it from your tail.

Memory Ballooning Effects in Virtualized Cloud Environments

Daya Shankar — Thu, 19 Feb 2026 14:55:13 +0000

Memory ballooning is a host memory reclaim method used during VM overcommit. The hypervisor inflates a balloon driver inside a VM to claw back RAM.

It can avoid host swapping, but it also shrinks guest page cache and can trigger paging. In Kubernetes, you see MemoryPressure, pod evictions, and tail-latency spikes.

What memory ballooning is

Ballooning is cooperative reclaim. It’s not “free memory.”

On VMware, the balloon driver (vmmemctl) works with the host to reclaim pages the guest considers least valuable.

VMware’s own perf guidance is blunt: avoid overcommit that forces regular host swapping, because that’s where performance collapses.

What you actually see in a managed Kubernetes service

You don’t see “ballooned MB.” You see consequences.

Kubelet enforces node-pressure eviction. Default hard threshold on Linux is memory.available<100Mi, and hard evictions have no grace period.
So any reclaim event that drops memory.available can turn into kills.

How ballooning pressure turns into outages on K8s nodes

This is the failure chain you should expect under overcommit.

Cache gets punched → more disk reads → p95 climbs.
Paging starts → jitter rises.
Kubelet evicts → restarts + thundering herd.

You don’t need hypervisor access to catch this. You just need node metrics and events.

What AceCloud gives you to control blast radius

You control node sizing and node-group policy, not the host reclaim knobs.

AceCloud Managed Kubernetes exposes worker node configurations like 2 vCPU/4 GiB, 4 vCPU/8 GiB, 8 vCPU/16 GiB (their published comparison table).
If you need bigger worker nodes, AceCloud’s flavor catalog shows Standard Instance options like S3a.2xlarge (8 vCPU/32 GiB) up through S3a.8xlarge (32 vCPU/128 GiB) and beyond.

Guardrails that work when overcommit is “yes”

These are defaults you can deploy without cluster-specific tuning.

Split worker node groups by risk

One node pool for prod latency. One for batch.

Protected pool: ingress, API, user-facing services.

Best-effort pool: ETL, async jobs, rebuilds.

This keeps batch from turning your prod nodes into the provider’s pressure valve.

Enforce requests/limits everywhere

The scheduler packs based on requests. If you don’t set them, you’re gambling.

Use Kubernetes resource requests/limits for CPU and memory.
For latency pods, run Guaranteed QoS: requests == limits.

resources:
requests:
cpu: "1"
memory: "2Gi"
limits:
cpu: "1"
memory: "2Gi"

Keep headroom by design

If you can’t tune Node Allocatable, you simulate it with request budgets.

Kubernetes calls this concept Node Allocatable (reserving resources for system daemons).
In a managed service, you may not get to set kube-reserved / system-reserved, so leave headroom in pod requests.

Baseline rule (protected nodes): don’t schedule more than 75% of node RAM by requested memory.

Pod density: don’t chase 110

Kubernetes “supported scale” guidance says no more than 110 pods per node.
Some platforms can configure higher, but pod IP and CNI limits usually bite first.

Use caps that match memory, not bragging rights.

Starting caps for AceCloud-sized worker nodes

Assumptions: typical daemonsets, no hugepages/DPDK, overcommit exists somewhere upstream.

Worker node	Role	Max total pod memory requests	Pod cap	Why
4 GiB	best-effort	2.5–3.0 GiB	15–25	leaves OS+kube headroom
8 GiB	protected	5.5–6.0 GiB	25–40	avoids eviction on small dips
16 GiB	protected	11–12 GiB	40–70	room for spikes + cache
32 GiB	mixed	24–26 GiB	70–110	only if requests are real

Anchor: the 110-pods/node guidance is a ceiling, not a target.

Evictions: make them predictable

If you can’t set kubelet flags, you still control which pods die first.

Assign PriorityClasses.

Put best-effort on best-effort nodes.

Put strict limits on batch so it can’t eat the node.

Know the kubelet defaults: memory.available<100Mi is the hard tripwire on Linux.

Swap: pick a stance and document it

Swap support exists now, but it’s not “turn it on and pray.”

Kubernetes documents swap memory management and node swap behaviors (including LimitedSwap).

Practical policy:

Protected nodes: swap off unless you’ve load-tested tail latency with swap on.

Best-effort nodes: consider LimitedSwap if you accept slower jobs.

What to alert on (works in any managed K8s)

You don’t need vCenter. You need signals.

Kubernetes-level

Node condition: MemoryPressure=True

Events: eviction messages

(Default eviction behavior is documented upstream.)

Node-level (Prometheus / node-exporter)

Alert on:

sustained low MemAvailable

paging activity (pgmajfault, pswpin, pswpout)

memory PSI pressure rising

If those light up during latency spikes, you’re in reclaim/paging territory.

Where AceCloud fits in this story

This is how you use their catalog without lying to yourself.

Start with AceCloud’s published worker sizes (4/8/16 GiB) for general pools.

For memory-heavy services (Kafka, JVM heaps, model servers), move the protected pool to bigger flavors from the standard catalog (ex: 8 vCPU/32 GiB and up).

Scale node groups earlier instead of packing nodes to the cliff. Node-group autoscaling is part of their managed Kubernetes offering.

If you want the “tight” version for your cluster

You can do it later from any terminal with cluster access, but you don’t need it to start.

Use the caps table above.

Enforce requests/limits + PriorityClasses.

Split node groups.

Keep 20–25% memory headroom on protected nodes.

That stops the common eviction storm even when the provider is running overcommit behind the scenes.

Hybrid Orchestration Basics: Avoiding Single-Provider Risks in 2026

Daya Shankar — Thu, 19 Feb 2026 14:49:58 +0000

Hybrid orchestration in 2026 means you can deploy the same workload across on-prem + AWS (and a second cloud if needed) using Kubernetes + Terraform + Argo CD as the common layer.

Keep Git as source of truth. Standardize identity, DNS, ingress, and observability. Then test failover like it’s a feature, not a promise.

What “single-provider risk” looks like

You can’t mitigate what you won’t name.

Risk	What breaks first	What it looks like at 2AM
Region/control-plane dependency	Deploy pipeline, cluster ops	“Can’t roll back. API calls time out.”
IAM lock-in	Workload identity, secrets access	“Pods can’t auth off-cloud.”
Network primitives	Ingress/LB, DNS	“Traffic won’t steer. Health checks lie.”
Data gravity/egress	DR, migration	“Failover works, but costs explode.”
Managed service coupling	DB/cache/queue	“App is portable. State is not.”

Rule: If your deploy and auth only work inside AWS, you don’t have “hybrid.” You have “AWS with extra steps.”

Pick a hybrid shape that matches reality

Topology decides your failure modes.

Option A: Two independent clusters (recommended default)

This is the boring one. It works.

Cluster 1: EKS in AWS

Cluster 2: on-prem Kubernetes (or another provider)

Argo CD fans out apps to both. Terraform builds both. You can fail one without taking the other’s control plane with it.

Option B: “Stretched cluster” (know the connectivity tax)

This is EKS Hybrid Nodes territory: control plane in AWS Region, nodes on-prem.

AWS calls this a “stretched/extended” cluster architecture.
AWS also publishes best practices that assume redundant, resilient connectivity to avoid disconnections.

Use it when:

you want one control plane

you can engineer reliable private connectivity

Avoid it when:

your on-prem is intermittently connected

you need disconnected/air-gapped operations

Option C: Disconnected/air-gapped on-prem

If “internet might not exist,” treat it as a hard requirement.

AWS documents EKS Anywhere as capable of running in air-gapped/disconnected environments.

Reference architecture

Every subsystem needs a home.

Git (source of truth)
|
v
Argo CD (GitOps)
(runs on-prem or neutral)
/ \
v v
On-prem K8s cluster AWS EKS cluster
(apps + addons) (apps + addons)
\ /
\ /
v v
Shared services: DNS, OIDC, logging/metrics,
container registry (mirrors), secrets KMS strategy

Rule: Put the GitOps control plane where a provider outage can’t strand you. Argo CD is a Kubernetes controller that continuously compares live state to Git and reports drift as OutOfSync.

Terraform: build infra once, not by hand

Terraform is for infra. Argo is for convergence. Don’t mix them.

Terraform responsibilities

VPC/VPN/Direct Connect edge

EKS cluster + node groups

On-prem cluster primitives (or the platform that hosts it)

IAM/OIDC scaffolding

Base DNS zones / records (if you must)

Repo layout that survives day-2

Keep it simple:

infra/
aws/
eks/
network/
onprem/
k8s/
apps/
base/
overlays/
aws/
onprem/
gitops/
applicationsets/

Argo CD: one template, many clusters

Multi-cluster GitOps is the whole point.

Argo CD supports ApplicationSet for multi-cluster automation.
The Cluster generator can auto-discover clusters registered in Argo CD and expose their metadata as template parameters.

Example: ApplicationSet that deploys to both clusters

Label your clusters in Argo (env=aws, env=onprem), then:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-addons
spec:
generators:
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: ["aws", "onprem"]
template:
metadata:
name: "addons-{{name}}"
spec:
project: platform
source:
repoURL: https://git.example.com/platform.git
targetRevision: main
path: "apps/overlays/{{metadata.labels.env}}/addons"
destination:
server: "{{server}}"
namespace: platform
syncPolicy:
automated:
prune: true
selfHeal: true

This gives you:

one definition

two targets

drift correction

Portability boundary: decide what must stay portable

Hybrid fails when you pretend everything is portable.

Portable by default

Kubernetes APIs (Deployments, Services, Ingress)

Helm/Kustomize overlays

Argo CD delivery mechanics

OpenTelemetry-based app telemetry

Not portable unless you plan it

Managed databases

Provider IAM-only auth

Provider-specific LBs and DNS behavior

Storage classes with provider-only semantics

Rule: If state can’t move, failover is theater.

Identity: stop wiring apps to one cloud’s IAM

Auth is the first thing that breaks off-cloud.

Baseline pattern:

Use OIDC for human and workload identity.

Use Kubernetes service accounts mapped to your identity provider.

Keep secrets strategy consistent (Vault, SOPS, external secret operators) across clusters.

If AWS IAM is your only workload identity story, your on-prem cluster becomes a second-class citizen.

Networking: make DNS and routing boring

Hybrid is mostly DNS and routes.

Minimum requirements:

deterministic routing between on-prem and AWS (VPN/Direct Connect)

clear ownership of egress/ingress paths

DNS resolution both directions (forward + reverse if needed)

If you choose “stretched EKS,” AWS’s docs push you to engineer resilient connectivity and plan for disconnections.

Operations: avoid doubling your surface area

Two clusters means two of everything unless you standardize.

One observability pipeline

one metrics backend

one log backend

consistent labels: cluster, env, region, service

One upgrade policy

version skew rules

maintenance windows

rollback runbooks

One incident drill

Run this quarterly:

break AWS ingress (simulate region/LB outage)

fail traffic to on-prem

verify auth, DNS, data correctness

roll back cleanly

If you can’t rehearse it, don’t claim it.

Where AceCloud fits in a “don’t bet on one provider” plan

If you want a second cloud without rewriting your platform, add it as another Kubernetes target.

AceCloud’s docs show a managed Kubernetes flow built around worker node groups, where you pick Flavor Type/Name, worker count, per-node volume, and security group.
That maps cleanly to the same GitOps model:

Terraform (or API) builds the cluster/node groups

Argo CD registers the cluster

ApplicationSet deploys the same overlays

This gives you a practical hedge:

AWS EKS as primary

on-prem as locality/compliance anchor

AceCloud as a secondary cloud target for burst, DR rehearsals, or an exit ramp

CTO checklist

Print this and use it in reviews.

GitOps control plane is provider-neutral (or at least not single-region)

Two independent clusters exist (on-prem + AWS), not just a stretched cluster

Argo CD multi-cluster deployment is automated (ApplicationSet)

Identity works off-cloud (OIDC strategy, not AWS-only IAM)

DNS and routing are deterministic (and tested)

Failover drill is scripted and run regularly

State portability is explicitly defined (what can fail over, what can’t)

GPU Scheduling Deep Dive: How Cloud Providers Allocate GPUs for Multi-Tenant AI Workloads

Daya Shankar — Thu, 19 Feb 2026 14:46:30 +0000

Cloud GPU “scheduling” is a chain of gates: quota decides if you’re allowed to ask, capacity reservations decide if GPUs exist in a zone, placement bins you onto physical hosts, and partitioning decides how many tenants share silicon (full GPU, MIG, time-slicing, vGPU).

Kubernetes sits at the end and consumes whatever the platform exposes.

GPU allocation is a pipeline, not one scheduler

If you can’t name the layer that said “no,” you can’t fix it.

Request (API / YAML)
-> Account quota (by region + purchasing option)
-> Zonal capacity (reservation / capacity block / spot pool)
-> Placement (host + network topology)
-> Partitioning (full GPU | MIG | time-slicing | vGPU)
-> Cluster scheduler (Kubernetes / Slurm)
-> Kubelet + device plugin exposes devices to containers

What each layer controls

This table routes incidents to the right team fast.

Layer	Who controls it	What failure looks like	What to check
Quota	Provider control plane	“quota exceeded” / “limit exceeded”	Instance type quotas + service quotas docs
Capacity	Provider control plane	“insufficient capacity” / stuck provisioning	Capacity Reservations / Capacity Blocks
Placement	Provider control plane	GPUs launch, topology is wrong	Placement groups / cluster strategy
Partitioning	Provider + NVIDIA stack	noisy neighbors / unfair sharing	MIG vs time-slicing vs vGPU
Cluster scheduling	You	Pods Pending	GPU resources + device plugin plumbing

Partitioning decides multi-tenancy behavior

This is where you pick isolation vs utilization.

Mode	What it does	Isolation	What breaks first	Best fit
Full GPU	One workload per GPU	High	utilization	training, big batch
MIG	Hardware partitions with dedicated compute/memory	High	fragmentation by profile	inference, fine-tune with QoS
Time-slicing	Oversubscribe GPUs; workloads interleave	Low	noisy neighbor	burst inference, dev/test
vGPU	Virtual GPU slices via hypervisor stack	Medium–High	licensing + ops	shared VM fleets, VDI, controlled slices

Two blunt truths:

Time-slicing is not memory/fault isolation. It’s interleaving.

MIG is real partitioning with fixed profiles. That creates profile fragmentation if you don’t standardize.

How major cloud providers gate GPU allocation

You can’t “autoscale GPUs” if the provider won’t hand you any.

Amazon Web Services

Quota and capacity are separate checks.

Instance type quotas are grouped by purchasing option (On-Demand, Spot, Dedicated, Capacity Blocks).

Capacity Reservations reserve compute capacity in a specific AZ.

Capacity Blocks for ML reserve GPU instances for a future time window for short-duration ML workloads.

Google Cloud

Reservations are zonal, and they validate capacity up front.

When you create a reservation, Compute Engine verifies capacity in the specified zone, then reserves it.

GPU machine types include A3 variants backed by H100 SKUs (A3 High/Mega/Edge).

Reservation types exist for ensuring optional resources like GPUs are available when you need them.

Microsoft Azure

Quota often shows up as vCPU-family limits plus capacity reservation.

Azure VM quotas are tiered (total regional vCPUs + VM-family cores). If either is exceeded, deployment fails.

Azure capacity reservation reserves compute capacity in a region or AZ for any duration.

ND H100 v5 starts at 8× H100 GPUs per VM (Azure docs).

Kubernetes GPU scheduling is device-plugin driven

The scheduler can’t place what the node doesn’t advertise.

Kubernetes exposes GPUs through device plugins; workloads request resources like nvidia.com/gpu, and the scheduler places Pods on nodes with allocatable capacity.

Full GPU request

This is the baseline contract.

apiVersion: v1
kind: Pod
metadata:
name: gpu-smoke
spec:
restartPolicy: Never
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base
command: ["bash","-lc","nvidia-smi && sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1

MIG request

You request a profile resource, not “a GPU.”

apiVersion: v1
kind: Pod
metadata:
name: mig-infer
spec:
containers:
- name: infer
image: your-infer-image
resources:
limits:
nvidia.com/mig-1g.10gb: 1

MIG is explicitly described as partitioning supported GPUs into isolated instances with dedicated compute/memory.

Time-slicing (oversubscription)

This raises utilization and raises blast radius.

version: v1
sharing:
timeSlicing:
renameByDefault: true
resources:
- name: nvidia.com/gpu
replicas: 10

Time-slicing is documented as oversubscription where workloads interleave on the same GPU.

Installing the NVIDIA stack

You need one canonical way to install drivers + device plugin + toolkit + monitoring.

The NVIDIA GPU Operator automates driver + device plugin + container toolkit + labeling + DCGM monitoring components.

Queue GPUs at the job layer or you’ll drown in Pending Pods

Pods Pending is not a scheduling policy.

Kueue manages quotas and decides when a job waits, when it’s admitted (Pods can be created), and when it’s preempted.

Practical pattern:

One shared cluster.

Separate GPU node pools by workload class.

Kueue ClusterQueues per tenant/team.

Admission control before Pods exist.

GKE publishes a multi-tenant Kueue tutorial that matches this model.

Reference architecture for multi-tenant AI workloads

This is a setup you can run for months without babysitting.

Split pools by workload class

This prevents MIG profile churn from breaking training placement.

Pool	GPU strategy	Controls	Notes
Training	Full GPUs	Kueue + quotas	avoids MIG fragmentation
Inference	MIG	Kueue or HPA	predictable slices
Dev/Test	Time-slicing	loose quotas	accept noisy neighbors

Use taints/tolerations to keep workloads honest

This stops inference from landing on training nodes “because it fit.”

# training nodes
spec:
taints:
- key: gpu-class
value: training
effect: NoSchedule

# training pods
spec:
tolerations:
- key: gpu-class
operator: Equal
value: training
effect: NoSchedule

Where AceCloud.ai fits

You keep the same scheduling stack. You change where the capacity comes from.

AceCloud publishes:

Cloud GPU instances with H100/A100/L40S listed as available options.

Managed Kubernetes GPU clusters as a first-class service offering.

Spot GPU pricing pages with per-hour rates and “saving” percentages by SKU (example: L40S in Mumbai).

Managed control plane claims, including a stated 99.99% uptime SLA on the managed control plane page.

How teams wire AceCloud into multi-tenant scheduling

This is the boring path. It works.

Deploy a managed Kubernetes cluster plus GPU node pools (training vs inference).

Install NVIDIA GPU Operator once per cluster.

Enable MIG on inference pools; keep training pools full GPU.

Add Kueue for tenant quotas and job admission.

Use spot GPUs for interruptible inference/batch where it fits your SLOs.

Ops checklist for debugging “we can’t get GPUs”

This is what you run before you open a ticket.

1) Confirm the provider gate that blocked you

Quota errors and capacity errors look similar in dashboards. They aren’t.

AWS: instance type quotas by purchasing option; capacity reservations / capacity blocks.

GCP: reservations validate capacity at creation time.

Azure: vCPU quota tiers plus capacity reservation.

2) Confirm Kubernetes sees allocatable GPUs

If the node doesn’t advertise it, the scheduler can’t place it.

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
kubectl describe node <node> | grep -E "nvidia.com/gpu|nvidia.com/mig"

Device plugin plumbing is the core dependency here.

3) Confirm you didn’t fragment MIG profiles

MIG failures are often self-inflicted.

If your inference pool is carved into small profiles, large-profile jobs won’t place until you reconfigure the GPU. MIG profiles are fixed.

Conclusion

Cloud providers allocate GPUs through quota + capacity + placement + partitioning before Kubernetes schedules anything. Multi-tenant reliability comes from picking the right sharing primitive: full GPU for training, MIG for isolated inference, time-slicing only for best-effort. Add Kueue so jobs queue instead of stalling Pods. Use AceCloud when capacity gating is your bottleneck, while keeping the same Kubernetes + NVIDIA Operator model.

How to Set Up Edge Infrastructure for Low-Latency Production Apps in India

Daya Shankar — Thu, 12 Feb 2026 05:28:15 +0000

Edge infrastructure India work comes down to one thing: cut round trips.

Put Cloudflare CDN in front, run Cloudflare Workers for routing/auth short-circuits and cache control, keep your origin in AWS Mumbai, and use Redis for hot state.

Measure p95 by city/ISP, then tighten cache keys, warm critical paths, and cap retry storms.

Start with a latency budget you can defend

If you don’t set a budget per hop, you’ll “optimize” the wrong layer.

Define your target like an SRE, not a slide deck:

User SLO: p95 end-to-end latency (and p99 if you have real SLAs).
Breakdown: DNS + TLS + TTFB + payload.
Scope: split by metro and ISP. India is not one network.

Minimum measurement plan

RUM (Real User Monitoring) from browsers/apps. Tag requests with city, asn, isp if your RUM tool supports it.
Synthetics from at least: Delhi NCR, Mumbai, Bengaluru, Chennai, Hyderabad, Kolkata.
Server timing headers from origin so you can isolate backend time vs network time.

What you want to see on a single chart

Edge TTFB (Cloudflare)
Origin TTFB (Mumbai)
Redis time (if used)
App compute time

If you can’t see those separately, you’re flying blind.

Reference architecture: Cloudflare → Workers → AWS Mumbai → Redis

Lock the shape first so each knob has a clear home.

Here’s the stack you picked, with crisp ownership boundaries.

Client (India ISP)
|
| DNS + TLS + HTTP
v
Cloudflare Edge (CDN)
|
| Worker (routing, cache policy, auth short-circuit)
v
AWS Mumbai Origin (ALB/NLB -> app)
|
| hot state / rate limits / sessions
v
Redis (ElastiCache or self-managed)

What runs where (don’t mix this up)

Layer	Do	Don’t
Cloudflare CDN	cache static + cacheable API responses, terminate TLS, absorb spikes	run business logic that needs DB writes
Workers	route, normalize headers, enforce cache keys, cheap auth gates, redirects	call 5 downstream services from the edge
AWS Mumbai origin	serve uncached requests, durable logic, writes	depend on “edge will save us” if origin is slow
Redis	sessions, rate limits, feature flags, hot lookups	treat it like a source of truth

Configure Cloudflare CDN like you mean it

CDN defaults are generic; your production app needs explicit cache rules.

The #1 reason “edge didn’t help” is this: you didn’t make responses cacheable. The difference between a generic CDN setup and a secure CDN solution is intentional cache design and strict isolation.Step 1: Classify endpoints

You can’t cache what you haven’t categorized.

Make three buckets:

Static: JS/CSS/images/fonts. Cache hard.
Semi-static: config, feature flags, catalog, “home feed” variants. Cache with short TTL + SWR.
Dynamic: personalized, writes, payments. No cache.

Step 2: Control cache keys

Bad cache keys destroy hit ratio and spike origin load.

Rules of thumb:

Strip tracking params (utm_*, fbclid, gclid) from cache keys.
Don’t vary on cookies unless you must.
If you must vary, vary on a small whitelist (e.g., plan_tier, locale), not the full cookie blob.

Step 3: Turn on “stale while revalidate” behavior

SWR converts origin spikes into background refresh.

If Cloudflare features are available in your plan, configure:

short TTL for semi-static API responses (e.g., 30–120s)
allow stale serve during refresh

This is how you keep p95 stable during origin deploys and brief Mumbai hiccups.

Step 4: Avoid cache poisoning and auth leakage

One sloppy header can cache private data for strangers.

Hard rules:

Never cache responses that depend on Authorization unless you fully control the cache key and isolation model.
Set Cache-Control: private for truly user-specific payloads.
For “public but user-aware” endpoints, issue explicit cache keys based on a safe token (not raw auth headers).

Use Workers for short-circuits and cache policy, not heroics

Workers are the glue. Keep them small so you can reason for failure.

Workers shine for:

request normalization (headers, query params)
cheap routing decisions
edge auth gating (basic, not deep)
explicit cache behavior via caches.default

A Worker pattern that helps latency

Route and cache at the edge, then fall back cleanly to Mumbai.

export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);

// Normalize cache-busting junk.
["utm_source","utm_medium","utm_campaign","gclid","fbclid"].forEach(p => url.searchParams.delete(p));

// Cheap gate. Block obvious abuse before it hits Mumbai.
const apiKey = request.headers.get("x-api-key");
if (url.pathname.startsWith("/api/") && !apiKey) {
return new Response("missing api key", { status: 401 });
}

// Only cache safe GETs.
if (request.method !== "GET") {
return fetch(new Request(url.toString(), request));
}

// Cache semi-static API endpoints for short TTL.
const isCacheableApi = url.pathname.startsWith("/api/catalog") || url.pathname.startsWith("/api/config");
if (!isCacheableApi) {
return fetch(new Request(url.toString(), request));
}

const cache = caches.default;

// Build a safe cache key. Keep it small.
const locale = request.headers.get("accept-language")?.split(",")[0] ?? "en";
const cacheKey = new Request(url.toString(), {
method: "GET",
headers: { "x-locale": locale }
});

let resp = await cache.match(cacheKey);
if (resp) return resp;

// Fetch origin, then cache it.
resp = await fetch(new Request(url.toString(), request), {
cf: { cacheTtl: 60, cacheEverything: true }
});

// Don’t cache errors.
if (resp.status >= 200 && resp.status < 300) {
ctx.waitUntil(cache.put(cacheKey, resp.clone()));
}
return resp;
}
}

What this does

It routes only what’s safe.
It caches only what you explicitly allow.
It avoids caching auth-bound content.
It keeps origin clean for truly dynamic calls.

Canary Workers safely

Edge bugs are global bugs.

Canary patterns that work:

enable Worker only on a subset of paths
enable by header: x-edge-canary: 1
enable by % rollout based on a stable hash (cookie/session id)

Keep a kill switch. Script it. Don’t rely on “we can revert fast” during an incident.

Redis: keep hot state hot, and make it disposable

Redis saves RTT when used right; it becomes your bottleneck when used wrong.

Put the right data in Redis

Cache the things that are read-heavy and safe to lose.

Good Redis candidates:

session tokens / session metadata
rate limiting counters
feature flags/config snapshots
small lookup tables (tenant → plan, user → segment)

Bad Redis candidates:

“primary database but faster”
large blobs
unbounded key growth

TTL strategy decides p95

TTL is a latency control knob.

Rules:

Use short TTL for rapidly changing data (seconds to minutes).
Use long TTL only if invalidation is correct.
Never ship “no TTL” in a multi-tenant production system unless you enjoy OOM incidents.

Avoid hot keys and stampedes

One hot key can take down your whole cache layer.

Fixes:

shard counters (key:{user}:{bucket}) instead of one global counter
add jitter to TTL to avoid synchronized expiry
use a single-flight pattern (only one origin fetch per key)

Add circuit breakers

When Redis is slow, fail fast and move on.

Set client timeouts.
Cap retries.
If Redis is down, serve from edge cache or hit origin directly depending on endpoint criticality.

Don’t let Redis become a distributed queue by accident.

Harden the AWS Mumbai origin so edge doesn’t mask real slowness

Edge can cut distance; it can’t fix a slow backend.

Connection reuse matters

Most “Mumbai is slow” tickets are handshake overhead plus pool starvation.

Do the basics:

keep-alive between ALB and pods/instances
HTTP/2 where it makes sense
right-size connection pools to DB and Redis

Make origin cacheable too

CDN misses still happen. Origin should be fast on repeat reads.

Add in-process caches for ultra-hot config.
Cache DB reads where correctness allows.
Precompute expensive aggregates.

Scale the right layer

Scaling pods won’t fix a saturated DB pool.

Watch:

request queue depth at load balancer
DB connection wait time
Redis latency percentiles
CPU throttling if you set tight limits

Scale compute only when compute is the constraint. Everything else is noise.

Observability that catches edge failures fast

“Cache hit ratio” is not an SLO.

Track these as first-class metrics:

p95/p99 latency at the edge (client-facing)
origin TTFB for uncached routes
cache hit/miss per route group
Redis p95 latency and error rate
retry rate (gRPC/HTTP clients)
5xx rate at edge and origin

Correlation trick that saves hours

Add a request ID at the edge.
Pass it to origin as a header.
Log it in app + Redis calls.
Now you can grep one request across layers.

Rollout plan that won’t torch production

Edge changes ship fast; that’s good until it isn’t.

A safe rollout looks like this:

Deploy Worker behind a header flag.
Canary with internal traffic and one low-risk route group.
Measure p95 and origin offload.
Scale rollout by % in steps.
Rollback instantly if p95 moves the wrong way.

Also: test from India, not from your laptop in Europe. Pipe synthetic checks from real metros.

India-specific gotchas you should design for

India traffic punishes extra round trips and large payloads.

Common realities:

Mobile networks with variable loss and jitter.
ISPs with inconsistent peering.
DNS resolution variance.

Practical fixes:

keep payloads small (compress, trim JSON, avoid chatty endpoints)
avoid multi-call fanout on the critical path
cache aggressively where safe
fail fast on slow dependencies to protect p99

The build checklist

If you can’t tick these off, you don’t have an edge setup—you have a proxy.

RUM + synthetics split by metro/ISP
Explicit cache rules for static + semi-static endpoints
Worker only does routing/cache/auth short-circuit
Redis keys have TTL, no hot-key stampedes
Origin in AWS Mumbai is tuned for keep-alive and fast reads
Kill switch for Worker rollout
Dashboards show edge p95/p99 + origin TTFB + Redis p95

Managed Cloud Infrastructure: What’s Included, What’s Not, and Why It Matters

Daya Shankar — Thu, 12 Feb 2026 05:11:33 +0000

Managed cloud infrastructure means your provider runs day-2 ops for defined layers patching, monitoring, backups, and incident response while you still own identities, data, application config, and misconfig risk.

Read the responsibility matrix and SLA, then script runbooks and restore tests around the boundary. If the scope is vague, outages drag on.

“Managed” is a scope boundary

If you can’t point to the boundary in writing, you don’t have a managed service.

Cloud providers frame this as shared responsibility: the provider secures the underlying cloud platform; you secure what you deploy and configure on top of it.

What’s included in managed cloud infrastructure

These are the tasks you’re paying to stop doing by hand.

1) Uptime for provider-owned components

The provider should publish an SLA for the layer they run (control plane, storage service, DR service). Don’t assume “whole stack” uptime unless the SLA says so.

2) Patching for the managed layer

Providers typically patch what they own (platform, managed control planes, managed service runtimes). Your OS, node pools, and app dependencies may still be yours unless the contract states otherwise.

3) Monitoring + alerting for their layer

A real managed offering ships:

platform metrics and health checks
alert routing + escalation
a support boundary that says what they touch and what they don’t

4) Backup/DR primitives

You usually get replication and failover mechanics. You still sharpie in app consistency, restore validation, and recovery drills.

5) Change management on their layer

Expect documented maintenance windows, version policy, and an upgrade path for provider-owned components. Managed Kubernetes control planes are common cases.

What’s not included

This is where most “managed” expectations break.

1) Identity and access configuration

You own identities and access policy across cloud models. If IAM is wrong, “managed” won’t save you.

2) Your data and how it’s protected

Providers give encryption features. You choose classification, key custody, access patterns, and retention.

3) Your network intent

Providers run the physical network. You still configure routes, firewall rules, private connectivity, and segmentation. Misconfig here still drops prod.

4) Your application behavior

Providers keep the platform alive. They won’t fix your deployment config, bad queries, or memory leaks.

5) Restore testing (often missed)

Some DR offerings explicitly don’t include routine test drills as a managed feature. If you don’t test restores, you don’t have recovery just storage.

The responsibility matrix you should put in the contract

This is the table you paste into the SOW and grep during incidents.

Layer	Provider owns (typical)	You own (typical)
Datacenter + hardware	power, racks, physical security	nothing
Virtualization	host/hypervisor baseline	guest OS if you run VMs
Managed Kubernetes control plane	API server/etcd/control-plane upgrades	RBAC, admission, policies, namespaces, workloads
Worker nodes	varies by service tier	node OS patching, runtime, add-ons (unless explicitly managed)
Backups/DR engine	replication + orchestration	restore tests, app consistency, recovery validation
Security	“of the cloud” controls	“in the cloud” configuration, identities, data

Why it matters

This is incident math, not procurement fluff.

Faster incident routing

If the boundary is clear, you don’t spend 45 minutes arguing about whose problem it is. You open the right ticket and move on.

Cleaner RTO/RPO planning

Providers can offer targets like <15 min RTO and <5 min RPO for DR, but your app still needs to restore validation and cutover steps you can execute under stress.

Fewer “surprise” costs

Managed services reduce ops toil. They don’t remove engineering work caused by fragile app design or bad change control.

What this looks like with AceCloud.ai

This is how to map “managed” scope to actual service pages and enforceable statements.

Managed Kubernetes Control Plane: states HA operation and a 99.99% uptime SLA for production workloads. Treat that as “provider owns the control plane” in your RACI.
Cloud uptime SLA statement: AceCloud also publishes a 99.99% uptime claim with an explicit downtime math note (“52 minutes per year”). Use that language in your SOW if it matches your scope.
Disaster Recovery service: documents DR orchestration and publishes RTO/RPO claims (including the <15/<5 figures on a replication page). Also calls out limitations like DR test capabilities. Put both the capability and the limitation in your runbook.
Fully managed private cloud: if your requirement is isolation + “someone else runs the platform,” this is the managed-infra pattern without public multi-tenant tradeoffs.

Buying checklist

Ask these questions. Get the answers in writing.

What is the SLA, and which component? (control plane vs nodes vs storage vs network)
Who patches what? (control plane, node OS, CNI, ingress, runtime)
What is the support boundary? (what’s excluded, what voids support, what is “best effort”)
How do backups and restores work? (RTO/RPO, restore steps, restore testing cadence)
How do upgrades work? (maintenance windows, rollback, version policy)

Conclusion

Managed cloud infrastructure works when the boundary is explicit. You outsource day-2 ops for the layers the provider controls control plane upkeep, platform patching, monitoring, and DR mechanics. You still own identities, data protection choices, network intent, and workload config. Put the RACI in the contract, then script restores and incident runbooks around it.

Private connectivity vs VPN: when to upgrade your network architecture

Daya Shankar — Thu, 12 Feb 2026 05:04:22 +0000

A VPN gives you encrypted connectivity over the public internet. Private connectivity (Direct Connect / ExpressRoute / Interconnect) gives you a dedicated path with steadier latency and higher throughput but it usually doesn’t encrypt traffic by default, so you often layer IPsec or MACsec on top.

Upgrade when VPN jitter breaks SLOs, tunnel sprawl becomes ops debt, or you need predictable bandwidth for replication and AI data pipelines.

What we’re comparing

You can’t pick up the right tool if you’re mixing transport, encryption, and routing in the same sentence.

VPN (site-to-site IPsec): encrypted tunnels over the public internet. AWS calls each tunnel “an encrypted link,” and a connection ship with two tunnels for HA.
Private connectivity: dedicated transport from your edge to the provider edge (BGP, circuits, cross-connects). Great for consistency. Not automatically encrypted on most platforms.

How VPN behaves in production

VPN works fast. Then traffic shows up and starts acting like traffic.

What VPN is good at

This is why teams start here.

You can deploy it in days, not weeks.
You get encryption by design (IPsec).
It’s fine for bootstrap migrations, admin access, and “good enough” hybrids.

Where VPN starts hurting

These are the failure modes I see in tickets.

Jitter and random loss. Your tunnel is stable; the internet path isn’t.
Throughput ceilings. Encryption and tunnel count become the throttle.
Tunnel sprawl. One tunnel becomes 20, then nobody remembers why half exists.

If you’re “fixing the VPN” every month, you’re already paying the upgrade tax—just in engineer hours.

How private connectivity behaves in production

Private connectivity buys you a cleaner transport layer. It doesn’t buy you security unless you configure it.

What you get

This is what people actually pay for.

More consistent latency and throughput than internet paths (especially under load).
BGP-based routing. You can route traffic with explicit preferences and failover instead of praying.

What you don’t get by default: encryption

This is the part that security reviewers will flag, correctly.

Amazon Web Services: Direct Connect traffic is not encrypted by default. AWS says you must use transit encryption options (IPsec overlays, MACsec on supported links).
Microsoft Azure: ExpressRoute provides private connectivity but doesn’t encrypt data in transit by default; Microsoft tells you to add encryption and security measures.
Google Cloud: Google documents HA VPN over Cloud Interconnect specifically to encrypt traffic traversing Interconnect.

When to upgrade: the triggers that justify the work

If you’re not seeing these, stay on VPN and spend money elsewhere.

Upgrade to private connectivity when you hit two or more of these:

Replication misses windows. Database/DR sync falls behind because latency and loss swing.
Steady high-volume data movement. Large backups, model artifacts, embeddings, logs—every day.
You’ve built a tunnel zoo. Many sites, many VPCs/VNETs, many peers.
SLOs care about tail latency. p95/p99 is the product, not a vanity metric.
Compliance wants controlled transport. Then you add encryption on top (IPsec/MACsec), because “private” ≠ “encrypted.”

Decision matrix

This is the table I’d drop into a design review.

Requirement	VPN is enough	Private connectivity is the better tool
Setup speed	Need it this sprint	You can wait for circuit lead time
Latency consistency	App tolerates jitter	App breaks on jitter (DB/replication/real-time)
Bandwidth profile	Burst / low-to-moderate	Sustained heavy transfer
Network scale	Few sites / few clouds	Many sites, many VPCs, complex routing
Security requirement	“Encrypt over internet”	“Controlled transport” + you also encrypt (IPsec/MACsec)

The pattern that usually wins: private transport + encrypted overlay

You don’t have to choose “VPN or private.” You can stack them.

Private link for the path
IPsec on top for encryption
BGP underneath for routing control

All three major clouds document some version of this (Direct Connect + VPN/IPsec, ExpressRoute + additional encryption options, HA VPN over Interconnect).

Migration plan that doesn’t torch prod

Cutovers should be reversible. If they aren’t, you’re gambling.

Inventory and fix CIDRs first. Overlaps will ruin your day.
Stand up private connectivity in parallel. Don’t rip-and-replace the VPN on day one.
Bring up BGP with route filters. Only advertise what you mean to advertise.
Shift traffic by routing preference. Change route preference; don’t restart fleets.
Keep VPN as failover until you trust the link. Then decide if you keep it as “break glass.”

Where AceCloud fits

This matters if you’re building hybrid or multi-cloud and want consistent primitives.

AceCloud.ai documents IPsec VPN and private connectivity options as part of its cloud networking stack, plus isolated VPC networking you can use to segment environments.
If you need routing control without new hardware, they also position “virtual routers” with support for VPN types like IPsec and GRE.

For internet-facing applications, layering in secure CDN solutions helps extend that private, controlled architecture to the edge. A secure CDN can offload TLS termination, provide DDoS mitigation, enforce WAF policies, and cache static or dynamic content closer to users, reducing origin load while improving performance and resilience. In hybrid or multi-cloud setups, this also gives you a consistent security perimeter across providers.

(You still do the same engineering work: route design, failover design, encryption policy. The provider just gives you the building blocks.)

Conclusion

Stay on VPN when you need encrypted connectivity fast and your workloads tolerate internet behavior. Upgrade to private connectivity when network variance starts breaking SLOs, data transfer becomes steady and heavy, or tunnel sprawl becomes an ops problem. Then add encryption deliberately because private transport is usually not encrypted by default on AWS, Azure, or Google Cloud.

How PCIe, NVLink, and NUMA Topology Affect GPU Scheduling Outcomes

Daya Shankar — Mon, 09 Feb 2026 05:21:54 +0000

GPU topology changes what “8 GPUs” really means: NCCL step time, multi-node InfiniBand efficiency, and inference p99. NVLink can hide bad PCIe wiring.

NUMA never does. On PCIe-only cards like NVIDIA L40S (PCIe Gen4 x16, NVLink: No), the scheduler must respect PCIe and socket locality, or you’ll buy GPUs and get bus contention.

GPU topology is scheduling input, not trivia

Bridge: If the scheduler can’t see topology, it will place jobs that look valid and run slow.

Schedulers allocate counts (gpus=8). They don’t allocate paths (“these 4 GPUs share a PCIe switch and sit on the same socket as the NIC”). That gap creates two common outcomes:

Training: NCCL all-reduce stalls on the slowest hop.
Inference: p99 latency spikes when CPU threads and DMA traffic cross sockets.

If you run a gpu topology-sensitive fleet on a gpu cloud server, you either encode topology into placement rules or you accept variance as a “feature.”

The three wires that matter

Bridge: We’ll tie each wire to the exact failure mode you see in Kubernetes and Slurm.

PCIe fabric

Bridge: PCIe is fast until multiple devices converge on the same upstream link.

PCIe isn’t one big flat bus. It’s a tree: endpoints → switches → root complex → CPU socket. When traffic funnels through a shared upstream link, bandwidth becomes shared and latency jumps.

L40S-specific reality: L40S uses PCIe Gen4 x16 (64 GB/s bidirectional) and does not support NVLink. That means GPU↔GPU traffic stays on PCIe. No fast side-channel.

NVLink and NVSwitch

Bridge: NVLink changes the GPU↔GPU fast path, which changes how forgiving the node is.

NCCL supports PCIe and NVLink/NVSwitch, and it will route collectives differently based on what it detects.
If you’re on a node with NVLink, you can sometimes “get away with” weaker PCIe placement. On L40S, you can’t.

NUMA sockets

Bridge: NUMA decides whether “local” memory and PCIe devices are actually local.

Dual-socket servers have two NUMA domains. Each domain has its own memory controller and PCIe root complex resources. Cross-socket traffic uses the CPU interconnect (UPI/IF/QPI-class links). That’s where you pay the “SYS hop” penalty in many topology maps.

What topology looks like in real metrics

Bridge: This is how topology shows up when you’re staring at slow training jobs.

NCCL is topology-aware, but not topology-proof

Bridge: NCCL will pick a comm graph, but it can’t invent a faster hop.

NCCL provides collectives across GPUs within and across nodes and supports PCIe, NVLink, and InfiniBand.
It also exposes knobs that make the topology model explicit.

NCCL documents path cutoffs like PIX / PXB / PHB / SYS for peer-to-peer decisions (same PCI switch → across PCI switches → same NUMA node → across NUMA nodes).

That matters because NCCL all-reduce behaves like this:

One slow edge in the ring/tree drags the whole step.

Cross-socket edges are the usual culprit on PCIe-only nodes.

“InfiniBand is fine” but the job is still slow

Bridge: GPU↔NIC locality can bottleneck before you hit the fabric.

GPUDirect RDMA provides a direct data path between GPU memory and a third-party PCIe device such as a NIC.
If the NIC sits under the other socket, you can still get extra hops and host involvement depending on topology and configuration.

Result: you scale nodes and don’t scale throughput.

Inference p99 gets ugly under mixed load

Bridge: p99 spikes happen when you add jitter to the CPU↔GPU feeding path.

Inference often looks fine at p50 and fails at p99. On L40S nodes, the usual trigger is cross-socket CPU placement or PCIe contention from a neighboring workload.

Step 1: Print the topology map on every node class

Bridge: You can’t Script scheduling rules until you can prove the wiring.

Run this on each node SKU you plan to Deploy:

# GPU and link map 
nvidia-smi topo -m 

# NUMA layout 
lscpu | grep -E "Socket|NUMA" 
numactl --hardware 

# PCIe tree 
lspci -tv

If you rent capacity, ask your provider for nvidia-smi topo -m output before you commit. AceCloud can hand you those maps for specific gpu cloud server SKUs so you can design job shapes that fit the hardware.

Step 2: Turn topology into job shapes

Bridge: Asking for “8 GPUs” is vague. Asking for “a 4-GPU island” is schedulable.

On PCIe-only nodes, “8 GPUs” often means “two 4-GPU islands.” If your job spans islands, you introduce slow hops.

Define shapes up front:

Workload	Shape	Why it works
NCCL training, single node	4-GPU island or full 8-GPU node	avoids cross-island P2P penalties
NCCL training, multi-node IB	N nodes × (same shape)	keeps rank topology consistent
Inference	1 GPU per pod/task	reduces contention, easier NUMA pinning

Now you can Configure scheduling rules around shapes instead of raw counts.

Kubernetes: make topology visible or accept random placement

Bridge: Vanilla K8s schedules extended resources, not PCIe and NUMA reality.

Use Topology Manager for NUMA alignment

Bridge: This is how you keep CPUs, devices, and memory on the same NUMA node.

Kubernetes Topology Manager with single-numa-node can reject pods that can’t be placed with a single NUMA affinity, using hints from “hint providers.”
Device plugins can provide NUMA TopologyInfo so kubelet can make locality-aware assignments.

This is the difference between:

CPU threads on socket 0 feeding GPUs on socket 0, and

CPU threads on socket 1 starving GPUs on socket 0 through remote memory traffic.

Encode GPU islands as labels

Bridge: K8s can’t pick “the 4 GPUs under this PCIe switch,” so you model islands at the node-pool layer.

Practical pattern:

1. Split node pools by hardware topology class (consistent wiring).

2. Label nodes with the shape they support.

Example labels:

topo.gpu.shape=4island

topo.ib.local=true

Then select with affinity:

affinity: 
  nodeAffinity: 
    requiredDuringSchedulingIgnoredDuringExecution: 
      nodeSelectorTerms: 
      - matchExpressions: 
        - key: topo.gpu.shape 
          operator: In 
          values: ["4island","8full"]

This is boring. That’s why it works.

On managed Kubernetes, topology discipline is mostly a node-pool problem—mixing GPU SKUs or wiring revisions inside one pool guarantees inconsistent NCCL graphs.
If you depend on Kubernetes node autoscaling, make sure it scales the right topology-labeled pool; adding “more nodes” that don’t match your GPU island shape can make jobs slower, not faster.

Slurm: ask for socket-local GPUs and bind them

Bridge: Slurm exposes more of the machinery you need for topology-aware placement.

Slurm schedules GPUs via GRES and has GPU-specific allocation features.
srun supports --gpus-per-socket and --gpu-bind.
On some HPC systems, --gpu-bind=closest binds each task to the GPU in the same NUMA domain as the CPU core the rank runs on.

Example pattern for dual-socket nodes:

srun -N1 \ 
  --sockets-per-node=2 \ 
  --gpus-per-socket=4 \ 
  --cpus-per-gpu=8 \ 
  --cpu-bind=cores \ 
  --gpu-bind=closest \ 
  python train.py

Two wins:

You don’t spread GPUs across sockets unless you mean to.

You keep CPU threads close to the GPUs they feed.

Step 3: Verify placement inside the job

Bridge: If you don’t verify, you’ll blame code for a wiring problem.

Kubernetes

Bridge: Check what you got, not what you asked for.

kubectl exec -it <pod> -- bash -lc ' 
  echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES 
  nvidia-smi topo -m 
'

Slurm

Bridge: Same idea. Different launcher.

srun -N1 --gpus=4 bash -lc ' 
  echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES 
  nvidia-smi topo -m 
'

Then Grep for cross-socket indicators in the topo matrix and fix placement before you touch model code.

Step 4: Microbenchmarks that catch topology regressions

Bridge: Two short tests will tell you if the node can actually Scale.

NCCL collectives: nccl-tests

Bridge: If all-reduce is slow here, training will be slow everywhere.

nccl-tests is the standard harness for NCCL collective performance.

git clone https://github.com/NVIDIA/nccl-tests.git 
cd nccl-tests 
make MPI=1 

export NCCL_DEBUG=INFO 
export NCCL_DEBUG_SUBSYS=GRAPH 

mpirun -np 8 ./build/all_reduce_perf -b 8M -e 1G -f 2 -g 1 | tee nccl.log

Now compare nccl.log across nodes of the “same” SKU. If graphs differ, your fleet isn’t topology-consistent.

InfiniBand baseline: ib_write_bw

Bridge: Prove the fabric, then prove your NUMA binding assumptions.

ib_write_bw is part of the perftest utilities used for InfiniBand performance testing.

Server:

ib_write_bw --report_gbits

Client:

ib_write_bw <server-ip> --report_gbits

Then rerun with NUMA binding. Yandex’s guide shows the exact pattern: bind CPU and NIC by NUMA to isolate the path.

Decision checklist for gpu topology on a gpu cloud server

Bridge: Use this to pick nodes and placement rules that won’t surprise you.

1. Confirm the interconnect

If you’re on L40S, assume PCIe-only GPU↔GPU. NVLink isn’t there.

2. Pick a job shape

4-GPU island for most NCCL training on PCIe-only nodes.

Full 8-GPU only when the topology map shows a clean fabric.

3. Make NUMA a hard requirement

K8s: Configure Topology Manager + device plugin topology hints.

Slurm: use --gpus-per-socket and bind.

4. Verify inside the allocation

Run nvidia-smi topo -m.

Pipe logs and Grep for the patterns that correlate with slow steps.

5. Benchmark once per node class

nccl-tests for collectives.

ib_write_bw for fabric sanity.

Bottom line

Bridge: Scheduling outcomes depend on wiring, so treat wiring as an input.

PCIe, NVLink, and NUMA decide whether your scheduler produces fast placements or expensive slow ones. On L40S-based fleets, you don’t get NVLink to mask bad decisions. Encode gpu topology into node pools and constraints. Verify allocations with nvidia-smi topo -m. Then scale your training jobs with fewer surprises on-prem or on a gpu cloud server.

If you want, I can tailor this into a strict guest-post format for your target publication (their style guide, link density, and any required “how-to” sections) while keeping the same blunt engineering tone.

The Role of Edge & Distributed Data Centers in Reducing Compute Latency

Daya Shankar — Mon, 09 Feb 2026 05:06:58 +0000

Edge and distributed data centers reduce latency by cutting physical distance, hop count, and queueing on the network path. Fiber propagation is ~4.9 µs per km, so 1,000 km round trip costs ~10 ms before routers and congestion.

For LLM apps this mainly improves time-to-first-token; for video analytics it keeps frames local and ships events upstream.

Latency is a budget, not a feeling

Bridge: If you can’t break latency into parts, you’ll spend on edge and still miss p99.

When a CTO says “we need lower latency,” they usually mean one of these:

The app feels sluggish (human perception).

A control loop misses a deadline (systems behavior).

p99 is spiking and support tickets follow (business impact).

All three map to the same thing: end-to-end time from client → compute → client. Not “GPU speed.” Not “region choice.” End-to-end.

Here’s the part nobody can negotiate: physics. Light in fiber is slower than light in vacuum. A solid rule of thumb is ~4.9 microseconds per kilometer in single-mode fiber.

So if your users are ~1,000 km away from compute:

Propagation alone costs ~4.9 ms one way.

Round trip is ~9.8 ms before you pay for hops, TLS handshakes, congestion, retransmits, and any L7 proxy chain you’ve built.

Distance sets the floor. Queues decide whether you live near the floor or nowhere close.

A latency budget you can use in a meeting

Bridge: This is the “what edge can fix” list.

Component	What drives it	Edge helps?	What to measure
Propagation	geography	✅	RTT from real client networks
Hop tax	routers, NAT, proxies	✅ sometimes	traceroute + request timing
Queueing / jitter	congestion, last mile	✅ maybe	p95 vs p99 drift, loss
Compute	model/runtime contention	❌ unless compute moves	TTFT vs tokens/sec

Cloudflare’s performance write-ups are blunt about this: if you only watch averages, you miss what users actually experience, which is usually tail behavior.

“Edge” and “distributed DC” aren’t the same thing

Bridge: People say “edge” and mean three different architectures. That’s how projects derail.

In practice, you’re choosing between:

1. Distributed regions (more metros)
You run full stacks in multiple regions/metros. This gets you closer to users without operating hundreds of tiny sites.

2. Edge PoPs (compute near users)
Small footprints closer to users. Great for interactive workloads. Harder to operate at scale.

3. CDN edge (routing + caching + shielding)
Not full compute (usually), but it reduces hops, terminates TLS closer, and can hide origin slowness.

A common “regional + edge PoPs” design is:
edge PoP handles the interactive front door → regional DC handles heavy compute and durable data → edge returns results fast.

That’s the pattern you want for your two target workloads: LLM inference and video analytics.

LLM inference: edge mostly buys TTFT

Bridge: For LLMs, you need to separate “first token” from “full answer.”

LLM latency is not one number. It’s at least two:

TTFT (time-to-first-token): includes queuing, prompt prefill, and network latency.

Tokens/sec (generation rate): mostly compute + runtime efficiency (batching, KV cache behavior, kernel choice).

Here’s why edge matters: users don’t wait for the whole answer to finish. They wait for the first visible response. TTFT is your “first byte” metric for chat.

AWS shows this directly in a Local Zones example: moving inference closer can reduce TTFT versus a regional deployment.

What edge doesn’t fix for LLMs

Bridge: If tokens/sec is your problem, distance is not your bottleneck.

Edge won’t fix:

long prompts (prefill cost grows with prompt length)

bad batching strategy

GPU contention/noisy neighbors

slow decoding kernels

NVIDIA’s TTFT definition explicitly calls out that TTFT includes prompt prefill and queuing, not just “network.”

So the right question is:

Are we slow because the user is far away, or because the model is slow?

If it’s distance, edge helps. If it’s model/runtime, edge is a distraction.

A sane “regional + edge” LLM layout

Bridge: Put the interactive path close to users; keep the expensive stuff centralized until you prove you need edge GPUs.

A pattern that scales without turning into a fleet nightmare:

Edge PoP

terminate TLS

auth + rate limits

prompt guardrails

lightweight retrieval cache (if you can cache safely)

stream responses back immediately

Regional DC

main model inference (GPU)

vector DB + durable stores

full observability pipeline

batch jobs (re-embed, evaluation, fine-tunes)

Streaming matters here. TTFT is the “start talking” metric. AWS frames TTFT as the time until the first token/chunk arrives for streaming apps.

Video analytics: edge wins by not moving frames

Bridge: With video, the fastest packet is the one you never send.

For video analytics, the biggest latency lever is brutal and simple:

If you ship raw frames to a distant region, you pay network delay and bandwidth cost.

If you process near the camera, you ship events + metadata upstream.

That’s not marketing. That’s just moving less data over the WAN.

A review of edge analytics notes that filtering/processing at the edge reduces request latency and reduces required bandwidth.
ScienceDirect’s video analytics overview also describes edge video offloading as reducing computational latency by processing video at the edge.

The common deployment pattern

Bridge: This is the architecture you can actually operate.

Camera → local encoder → edge node (GPU or accelerator)

Edge node runs detection/tracking

Upstream sends:

object counts

bounding boxes

alerts

short clips only on incident

You can keep a regional hub for:

compliance storage

model registry

retraining data curation

centralized dashboards

But you don’t need to hairpin every frame through it.

The network layer: how traffic finds the “nearest” edge

Bridge: Edge compute doesn’t help if you can’t reliably route clients to a nearby healthy PoP.

Two tools show up over and over:

1) Anycast

Bridge: Anycast is the “announce the same IP from many places” trick.

Anycast advertises the same IP from multiple PoPs, and routing tends to steer clients to a nearby instance based on BGP path selection. Akamai describes Anycast as a mechanism announced from multiple points on the internet to reduce DNS RTT and improve performance.

Anycast isn’t magic. It’s “good enough” locality with fast failover properties when done right.

2) Latency-based routing + CDN front door

Bridge: When you can’t Anycast everything, you steer at DNS/CDN.

AWS documents latency-based routing patterns for active-active, using Route 53 and CloudFront to deliver low latency.

For CTO planning: you don’t need to pick one. Many stacks use:

Anycast for DNS / edge ingress

Latency-based routing for region selection

Health checks to avoid sending users to a dead PoP

Why private connectivity still matters in an “edge” story

Bridge: Edge reduces distance. Private connectivity reduces variance in the parts you control.

You can’t control the user’s last mile. You can control the middle mile between:

edge PoP ↔ regional hub

sites (factories/branches) ↔ your edge PoP

regional hub ↔ storage/DR location

That’s where private connectivity or private network segments pay off: fewer random internet detours, fewer surprise bottlenecks.

For AceCloud specifically, you’re not guessing whether networking primitives exist. They publish:

VPC constructs (subnets, routing, firewall rules)

Virtual Routers as a routing component in their networking suite

Private Network as a priced network service (you can budget it)

That’s the “plumbing” you need for controlled edge ↔ region backhaul.

What you actually pay for with edge

Bridge: Edge lowers latency. It raises operational surface area. Always.

Edge systems fail in boring ways:

expired certs

clock drift

disk full

one PoP stuck on an old model artifact

one PoP losing packets due to an upstream change

If you can’t do these reliably, don’t roll out 20 PoPs:

artifact pinning

canary deploy

rollback in minutes

centralized logs/metrics/traces per PoP

Cloudflare’s performance work repeatedly comes back to measurement discipline and focusing on the right percentiles, not feel-good averages.

At that point, managed kubernetes becomes the simplest way to standardize deployments across many PoPs, instead of hand-managing servers. A kubernetes managed control plane also reduces ops load (upgrades, HA, policy) while you focus on observability, canaries, and rollback discipline.

Mapping this to AceCloud: distributed footprint + controlled networking

Bridge: Tie the concept to real locations and primitives, not a hand-wavy “global edge.”

AceCloud’s public material gives you three concrete anchors:

They state they operate 10 data centers.

They announced a cloud region in Noida (with partners NetApp and Quantum).

Their IaaS FAQ names key locations including Mumbai and Atlanta.

So if you’re designing “regional + edge PoPs” on AceCloud.ai, the CTO-friendly path is:

Pick the closest regional hub (Noida/Mumbai/Atlanta depending on user base).

Put edge PoPs where user RTT is killing TTFT (LLM) or where frames originate (video).

Use private network segments and routing controls for edge ↔ region backhaul.

A decision checklist that prevents “edge for edge’s sake”

Bridge: This is the part that keeps the project from becoming a slide deck.

Edge is worth it when:

TTFT is the pain (LLM feels slow), and client RTT is a meaningful chunk of TTFT.

You can avoid moving big data (video frames) by processing locally and shipping events.

p99 is tied to distance/hops, not just runtime compute.

Edge is usually not worth it when:

Tokens/sec is your limiter (runtime/compute issue).

You can’t operate a fleet (no rollout discipline, no observability).

Your data path forces constant backhaul anyway (you didn’t actually reduce movement).

How to prove it with one week of work

Bridge: Measure first. Then spend.

Instrument client RTT by geography (real networks, not a lab).
Record TTFT and tokens/sec separately for LLM endpoints.
Measure p95 and p99 before and after.
For video, measure:
time from frame captured → event emitted
WAN bandwidth before/after edge filtering

If your p95 improves but p99 stays ugly, you probably moved compute closer but didn’t fix queueing, routing variance, or overload behavior. That’s not an edge problem. That’s a system behavior problem.

Conclusion

Edge and distributed data centers reduce latency when they reduce the right part of the budget: distance, hops, and unnecessary data movement. For LLM apps, edge usually improves TTFT; tokens/sec still depends on runtime and GPUs. For video analytics, edge is the default because it keeps frames local and ships events upstream. Build it as “regional + edge PoPs,” and use private networking to keep the backhaul predictable.