DEV Community: NTCTech

The "Lift-and-Shift to KVM" Fallacy

NTCTech — Mon, 04 May 2026 12:42:14 +0000

The VM conversion completed without errors. Every workload made it across. The migration dashboard showed green, the project lead closed the ticket, and the consultants left the building.

Three weeks later, backup verification jobs are silently failing. Monitoring dashboards are dark. The on-call team is operating without baselines. Nobody knows what normal looks like on the new platform.

The VM conversion worked. The migration did not.

This is the lift-and-shift KVM fallacy — and it isn't a KVM problem. It's a scoping problem. Most VMware-to-KVM migration plans capture the visible dependency — the hypervisor — and treat everything built around it as someone else's project. The Operating Model Gap is what that assumption leaves behind.

What Lift-and-Shift Actually Moves

Lift-and-shift KVM moves compute. Disk images transfer. Network definitions port. VM configurations recreate on the other side. From a data-plane perspective, the migration looks complete because the workloads are running.

What does not move:

Operational runbooks referencing vCenter constructs
Backup architecture built against VADP APIs
Monitoring thresholds calibrated to vSphere metrics
Provisioning workflows targeting vCenter endpoints
Snapshot behavior assumptions encoded in recovery procedures
Storage policy logic tied to vSAN semantics
Identity and access models mapped to vCenter RBAC
Operator muscle memory built over years of vCenter navigation None of this appears in the migration plan. All of it breaks after cutover.

The Operating Model Gap is the distance between what the migration plan captured and what the platform actually required to function. Every item in that list is a component of the operating model. The hypervisor conversion touches none of them.

VMware Was Never Just the Hypervisor

The framing that produces lift-and-shift KVM plans is this: VMware equals ESXi. Replace ESXi with KVM. Migration complete.

That framing is wrong. VMware was never ESXi. VMware was the control plane your entire operating model was built around.

| What the plan says | What actually changes |
|---|---|
| ESXi → KVM | vCenter (lifecycle and provisioning control) |
| | vMotion semantics (live migration behavior) |
| | vSAN (storage abstraction and policy model) |
| | NSX (network policy and microsegmentation) |
| | vROps / vRealize (observability and alerting logic) |
| | VADP (backup API framework) |
| | DRS (scheduling and placement policy) |
| | Snapshot behavior (application-consistent logic) |

A VMware environment is not a hypervisor with add-ons. It is an integrated control surface where compute scheduling, storage policy, network segmentation, observability, and recovery operations all converge. When you replace ESXi with KVM, every one of those layers needs a replacement or a rebuild — and unlike ESXi, KVM does not ship them included.

KVM is a kernel module. The management plane, storage architecture, network abstraction, and observability stack are your responsibility to assemble, integrate, and operate. That assembly is the migration work most lift-and-shift plans never scope.

The Operating Model Test: If vCenter disappeared tomorrow, what percentage of your operating model disappears with it?

For most VMware shops, the honest answer is somewhere between 60 and 90 percent. That percentage is the scope of what a lift-and-shift to KVM does not address.

The Three Failure Surfaces After Cutover

Lift-and-shift KVM migrations do not fail at cutover. They fail in operations. The failure surfaces are predictable, they appear in sequence, and they are almost never in the migration plan.

Failure Surface 1: Control Plane Replacement (Day 1–7)

You did not replace ESXi. You replaced vCenter.

vCenter was the operational control surface for provisioning new workloads, managing VM lifecycle, enforcing placement policy, controlling access, and targeting automation. When you move to KVM, vCenter is gone — and everything that pointed at it needs a new target.

The KVM ecosystem offers options: libvirt for direct management, Proxmox VE for a GUI-centric model, oVirt for a closer-to-vCenter experience, OpenStack for cloud-scale orchestration. Each is a different operating model. None is a drop-in replacement. The team that executed a lift-and-shift KVM migration and operated vCenter for a decade does not automatically know how to operate any of them under pressure at 2am.

This is the first stall point. Not because the management plane doesn't exist — it does — but because the operating model loses its control surface and the team has to rebuild operational confidence from scratch.

Failure Surface 2: Storage Semantics Collapse (Day 7–30)

You did not lose shared storage. You lost the storage abstraction your platform behavior depended on.

vSAN provided a distributed storage fabric with defined behavior around replication, failure domains, snapshot consistency, and policy-based placement. That abstraction encoded a set of assumptions your entire backup architecture, recovery procedures, and performance baselines were built against.

In a KVM environment, that abstraction is gone. You are now operating raw storage — whether Ceph, NFS, iSCSI, or local — and the behavior is different in ways that matter:

Snapshot behavior — application-consistent snapshot mechanics differ by storage backend; VADP is gone
Backup assumptions — protection jobs built against VADP APIs break immediately; rebuild is required
Performance characteristics — latency, IOPS, and throughput profiles differ between vSAN and Ceph under the same load pattern
Replication semantics — storage replication behavior and consistency guarantees are not equivalent
Failure domain logic — how the platform handles node loss differs from vSAN's policy model This is where migrations pass validation and fail under load. Workloads run. The environment looks healthy. The gaps appear during the first backup verification window, the first storage-intensive workload spike, or the first incident that requires a restore from a snapshot taken after cutover.

Failure Surface 3: Operational Signal Loss (Day 30+)

The workloads moved. The signals didn't.

VMware environments accumulate operational signal over years — dashboards calibrated to vROps metrics, alert thresholds tuned against vSphere counters, runbooks that reference specific vCenter constructs, capacity models built on historical data from the VMware telemetry stack. That signal is institutional knowledge encoded in tooling.

After a KVM migration, all of it is wrong. The old dashboards are meaningless because the metrics don't exist. The alert thresholds don't map because the counters are different. The runbooks reference objects that no longer exist. The on-call team is operating blind against a platform they don't have baselines for yet.

This is where Day 30 failure begins. Not a dramatic incident — a slow erosion of operational confidence, a growing number of "we're not sure what normal looks like" moments, and a steady accumulation of unresolved alerts the team has stopped trusting.

The observability rebuild is not a migration task. It is a post-migration operational project that takes weeks. It is almost never in the original migration scope.

When KVM Actually Fits

This is not a post about KVM being unsuitable for enterprise infrastructure. KVM is a legitimate hypervisor running production workloads at scale across some of the largest environments in the world. The question is not whether a lift-and-shift KVM approach works — it's whether your operating model is positioned for it.

KVM fits when the operating model already lives below VMware's abstraction layer. KVM is a Linux kernel module; operating it well means operating Linux well, at depth, under production pressure.

The signal that KVM is the right call:

Linux is already the operational center — the team thinks in hosts, not abstractions
Automation already targets infrastructure primitives directly, not vCenter APIs
The team has operated without VMware's abstraction layer under pressure — not in theory, in production
Sovereignty or cost physics make open-source the architectural requirement, not just the preference
Greenfield or container-adjacent workloads where VMware's abstraction was overhead, not operating leverage The distinction that matters is not "does the team know Linux." It is whether the team has operated infrastructure at the primitive layer under production pressure. A team with deep vCenter muscle memory that also has Linux skills is not the same as a team that has always operated below the abstraction. The former needs a longer runway and an explicit skills transition plan. The latter is ready.

Scope the Operating Model Before the Hypervisor

The correct sequencing for a lift-and-shift KVM migration is not: pick hypervisor, convert VMs, go live. It is: audit the operating model, scope the rebuild, then pick the hypervisor.

Four things to scope before the hypervisor decision is final:

01 — Management Plane Decision
Pick the management plane before the hypervisor. libvirt, Proxmox, oVirt, and OpenStack are not equivalent choices — each implies a different operational model, skill requirement, and automation target. The management plane decision determines the operating model. The hypervisor follows from it.

02 — Storage Semantics Audit
Map every storage dependency in the current environment — snapshot behavior, backup integration points, replication architecture, performance baselines. Document what the new storage backend provides and where the semantics differ. The delta is the rebuild scope. Treat it as a parallel workstream, not a migration task.

03 — Observability Rebuild Plan
Plan for zero operational signal on Day 1. The old dashboards are dead. The alert thresholds don't transfer. Build the observability stack against the new platform before workloads arrive — or accept that the first weeks post-cutover will be operationally blind.

04 — Skills Audit (Honest Version)
Not certifications. Not training course completions. Operational depth under pressure. Has the team operated storage at the Ceph or NFS primitive level during an incident? Have they managed KVM scheduling behavior under resource contention? Knowing how something works is not the same as having operated it when it breaks.

Architect's Verdict

KVM is not the problem. Treating the hypervisor as the platform is.

VMware was a control plane your entire operating model was built around. A lift-and-shift KVM project moves the compute layer and leaves the operating model — management plane, storage semantics, observability stack, backup architecture, and operational muscle memory — orphaned on the other side of the migration window.

The fallacy is not that KVM is harder than expected. The fallacy is scoping a lift-and-shift KVM project as a hypervisor migration when what you actually triggered is an operating model rewrite. Name it correctly before the project starts. Scope the rebuild explicitly. Run the Operating Model Test before you sign the migration plan.

If vCenter disappeared tomorrow and 70 percent of your operating model went with it, that 70 percent is the migration. The hypervisor swap is the easy part.

Originally published at rack2cloud.com

Google Just Moved the Control Plane Boundary

NTCTech — Fri, 01 May 2026 12:10:18 +0000

For a decade, the Kubernetes scaling playbook had one move: add another cluster.

Need more capacity? Add a cluster. Need workload isolation? Add a cluster. Need regional separation? Add a cluster. Need a dedicated GPU pool? Add a cluster. The cluster became the unit of scale because the control plane could not scale far enough to avoid making it one.

At Google Cloud Next '26, Google made the opposite bet. A single Kubernetes-conformant control plane spanning 256,000 nodes across multiple regions, managing a million accelerators as a unified capacity reserve. Not bigger Kubernetes. A different architectural claim entirely.

The claim is this: the control plane is now the unit of scale. The cluster is not.

Most platform architectures were not built around that assumption. They are still operating the old boundary — and that mismatch is what this post is actually about.

The Old Scaling Model Was Cluster Multiplication

The cluster-as-boundary model made sense when it emerged. Kubernetes control planes had real scale limits. Policy enforcement was cluster-scoped. Observability was cluster-local. Capacity pools were physically tied to the node groups a given control plane could manage.

So teams multiplied. A cluster per environment. A cluster per region. A cluster per team. A cluster per workload class. A cluster per GPU type. The operational pattern became: when you hit a boundary, add another cluster.

That solved the immediate problem. It also created a different class of problem that compounded silently:

Fragmented capacity. Idle capacity in one cluster could not be claimed by a workload running out of headroom in another.
Duplicated policy. Every cluster needed its own RBAC, network policy, and admission control. Changes had to propagate across every cluster. Drift was structural.
Disconnected observability. Metrics and logs were cluster-local. Understanding system-wide state required stitching together signals from dozens of independent sources.
Compounding operational overhead. Each cluster was a discrete object requiring lifecycle management, upgrades, and failure response. The industry normalized cluster multiplication because the alternative — scaling the control plane itself — was not a credible option. Until now.

Google Just Moved the Boundary

GKE Hypercluster is not a capacity announcement. It is an architectural boundary announcement.

A single, Kubernetes-conformant control plane managing 256,000 nodes across multiple Google Cloud regions, treating distributed infrastructure as a unified capacity reserve — that is a claim about where the boundary should sit. Not at the cluster. At the control plane.

The Control Plane Boundary is the logical boundary at which scheduling authority, policy enforcement, and capacity governance are unified. For a decade, that boundary was the cluster by necessity. Hypercluster is Google's signal that it does not have to be.

When the control plane boundary moves outward — from cluster-scope to fleet-scope:

Capacity planning becomes global
Policy becomes a control plane concern, not a cluster concern
Scheduling becomes capacity orchestration across a unified multi-region pool
Failure domains get redefined This is not a GKE-specific development. It is a signal about where the architectural center of gravity is moving.

Most Teams Still Operate the Old Boundary

Most platform architectures today are still built around four cluster-scoped assumptions:

Cluster as operational boundary. Runbooks, upgrade cycles, certificate rotation — all scoped to the cluster. This made sense when each cluster was the largest coherent unit. It becomes overhead when the control plane boundary moves outward.

Cluster as policy boundary. RBAC, network policy, admission webhooks — all applied at cluster scope, duplicated across every cluster in the fleet, drifting over time.

Cluster as capacity boundary. Cluster autoscaler, node pools, resource quotas — all defined within a cluster. Cross-cluster capacity awareness requires external tooling or manual coordination.

Cluster as failure boundary. Blast radius assumptions and availability zone mapping built around the cluster as the natural unit of failure.

These assumptions were correct architectural choices when the control plane could not scale past them. They become architectural debt when the control plane boundary moves.

What Breaks When the Boundary Moves

When the control plane boundary shifts, the old cluster-scoped assumptions do not just become inefficient — some of them break operationally.

Capacity planning stops being cluster-local. The question "how much headroom does this cluster have" becomes wrong. The right question is "what is the available capacity in this scheduling domain" — which may span regions and node types. GPU idle is already a capacity forecasting failure in cluster-local models. It compounds in fleet-scale models without the right abstraction.

Policy can no longer be cluster-scoped by default. Policy duplication that was an accepted operational cost becomes a design inconsistency across the unified scheduling domain.

Failure domains stop aligning cleanly to cluster boundaries. Blast radius design at control-plane-boundary scale is an explicit architectural decision, not a cluster-topology default.

Observability must model control-plane-wide state. Cluster-local metrics describe local state. Fleet-wide scheduling decisions require fleet-wide visibility. The gap between what dashboards show and what the system is actually doing does not shrink when the scheduling domain expands without deliberate instrumentation.

Scheduling becomes capacity orchestration, not node placement. Kubernetes scheduling at cluster scope is a bin-packing problem. At control-plane-boundary scope it is a capacity allocation problem. Different mental model, different tooling, different operational discipline.

This is where Kubernetes operations becomes distributed control plane design. That is the actual shift — not the chip count.

The Million-Chip Problem Is Not About Chips

The headline number from Hypercluster is a million chips. That is the wrong thing to pay attention to.

Google is not telling you that you need to manage a million chips. Google is telling you that the next infrastructure bottleneck is not compute — it is the control plane that governs compute.

The teams still scaling by multiplying clusters are solving yesterday's bottleneck. Every cluster added under the old model is a migration conversation waiting to happen under the new one. The cost of a cluster-multiplication architecture is not just operational overhead. It is the structural cost of a boundary assumption that the industry is moving past.

The control plane boundary is not a GKE feature. It is the next architectural forcing function in distributed infrastructure. The architectural question for everyone else is not whether to adopt Hypercluster. It is whether your platform design is built around a boundary assumption that is already changing.

Architect's Verdict

Kubernetes cluster multiplication was not a mistake. It was the correct architectural response to a real constraint: the control plane could not scale far enough to make it unnecessary.

That constraint has now been challenged directly. The Control Plane Boundary — the logical boundary at which scheduling authority, policy enforcement, and capacity governance are unified — belongs at fleet scope, not cluster scope. Google made that bet publicly at Next '26.

Most platform architectures are still designed around the cluster as that boundary. The four assumptions — cluster as operational boundary, policy boundary, capacity boundary, and failure boundary — were correct when the ceiling was low. They become architectural debt when the ceiling moves.

The million-chip number is not the story. The story is what it signals about where the bottleneck is moving. For a decade, teams added clusters to avoid hitting the control plane ceiling. The ceiling just moved. The question is whether your architecture was designed for the constraint, or for the problem the constraint was preventing you from solving.

The Control Plane Boundary has shifted. Most architectures have not.

Originally published at rack2cloud.com

GPU Scheduling in Kubernetes: Start Before the Scheduler

NTCTech — Thu, 30 Apr 2026 12:55:20 +0000

Most teams think GPU scheduling starts with the scheduler.

It starts with demand modeling.

By the time Volcano, Kueue, or KEDA enters the conversation, the expensive mistake has usually already been made. The cluster was provisioned against a theoretical peak that rarely materializes. The demand curve was never drawn. The concurrency profile was assumed rather than measured.

The core argument: GPU scheduling is not a capacity solution. It is a capacity enforcement layer. If you provisioned against the wrong demand curve, the scheduler cannot save you.

The Demand Model Preflight

Before you talk about schedulers, answer four questions:

1. What is your real concurrency floor? Not peak theoretical demand. The minimum sustained parallel work your cluster must support without queue collapse. If you cannot answer this from measurement, you don't have a demand model — you have an assumption.

2. What is burst, and what is noise? If demand spikes for ninety seconds, does that justify permanent GPU allocation — or should it queue? Burst shorter than your cold-start window is noise. Noise should not drive provisioning decisions.

3. How long does work stay resident? A model loaded in VRAM is not active work. If memory stays hot longer than compute stays busy, utilization is already overstated before the scheduler runs a single job.

4. What can wait, and for how long? Scheduling starts with tolerated latency. If every workload is marked urgent, none of them are schedulable efficiently.

If you cannot answer all four from data rather than assumption, the scheduler conversation is premature.

What Correct GPU Demand Modeling Looks Like

Seven inputs. Each one has a consequence if you get it wrong.

Request concurrency — If you modeled single-thread throughput, your cluster is sized for a workload that never actually runs.

Queue depth — How many jobs can wait before it becomes a latency problem? Most teams buy hardware when they should be designing queue behavior.

Burst profile — Short demand spikes get priced into permanent capacity. A correct burst profile separates the spike duration from the allocation decision.

Latency tolerance — Batch training tolerates queuing. Real-time inference does not. Sizing uniformly across both is a guaranteed waste pattern.

Batch vs inference mix — These are distinct provisioning decisions. A cluster optimized for training batch jobs has a different shape than one optimized for sustained inference throughput.

VRAM residency time — How long does a model stay loaded relative to how long it is actively processing requests? High residency-to-compute ratio means memory is doing the work of availability, not throughput.

Job duration variance — High variance creates scheduling fragmentation regardless of how well the scheduler is configured. Understanding variance at p50/p90/p99 determines whether gang scheduling or preemption policies are necessary.

Provision for Shape, Not Peak

The corrective action is a provisioning philosophy shift.

Wrong Target	Correct Target
Peak demand	Concurrency bands
Max model size	Queue tolerance
Future scale	Sustained demand windows
Worst-case headroom	Known burst ceilings

Concurrency bands come from request concurrency measurement. Queue tolerance comes from latency tolerance modeling. Burst ceilings come from burst profile analysis. The provisioning decision is downstream of the model — not upstream of it.

Where the Scheduler Actually Fits

The right evaluation criterion for a scheduler is not feature sets. It is whether the scheduler enforces the constraints your demand model defined.

Three tools, three enforcement roles:

Volcano → batch fairness / queue discipline. Implements fair-share scheduling and gang scheduling for distributed training. Enforces concurrency band design across workload classes.

Kueue → admission control / workload gating. Answers Preflight Question 4 directly — what can wait. Prevents jobs from entering the scheduling queue until capacity exists to run them.

KEDA → event-driven scale behavior. Answers Preflight Question 2 — burst vs noise. Scales to the burst ceiling the demand model defined, not to unbounded demand signals.

These are not alternatives. They are complementary enforcement layers at different points in the scheduling stack.

What Good GPU Scheduling Actually Looks Like

Not which scheduler. What the outcome looks like when the demand model is correct:

Jobs wait intentionally — queue latency exists by design, not by accident
Inference scales on bounded demand — KEDA scales to the burst ceiling, not beyond it
VRAM stays loaded for active work — residency-to-compute ratio is enforced operationally
Queue latency is tolerated by design — the latency tolerance input becomes an SLA
Expensive accelerators do not sit hot without work — loaded ≠ active, eliminated

Architect's Verdict

The scheduler is not where GPU efficiency begins. It is where good capacity decisions are enforced — or bad ones become permanent.

Build the demand model first. Provision to its shape. Then configure the enforcement layer. In that order, and no other.

Originally published at rack2cloud.com

Cost Visibility Is Not Cost Control

NTCTech — Wed, 29 Apr 2026 12:05:37 +0000

Cost visibility tells you what your architecture costs. Cost control determines whether that architecture should have existed in the first place.

These are not the same discipline. Most organizations treat them as if they are — and the FinOps data proves they have been doing so for years without fixing the underlying problem.

The State of FinOps 2026 report found that 98% of organizations are now actively managing AI spend. Tooling investment has increased. Executive ownership has expanded. Reporting has become more granular. And yet organizations without structured cost governance still waste 32–40% of their cloud budgets on idle resources, oversized instances, and structural inefficiencies that dashboards surface but cannot remove.

More visibility. Same waste. That is the signal worth paying attention to.

Visibility Is a Reporting Layer, Not a Control Layer

FinOps tools do several things well. They surface spend. They expose waste. They identify anomalies. They allocate costs across teams and workloads. These are genuinely useful capabilities — the problem is that none of them can prevent the architecture decision that created the bill.

This distinction matters because most cost governance programs are built around observation, not prevention:

Dashboards show you where money went
Alerts tell you spend has increased
Tagging lets you attribute cost to a team
Optimization recommendations identify inefficiency
Monthly reviews give you a structured moment to react Every one of those mechanisms operates after the decision. The commitment — the topology choice, the platform selection, the replication model, the egress dependency — was made upstream. By the time FinOps sees the number, the architecture has already answered the cost question.

Cloud cost is now an architectural constraint — but that constraint only bites when you treat cost as a design variable rather than a reporting output. Visibility is lagging telemetry. It tells you what happened. It does not determine what was allowed to happen.

That distinction is the entire argument.

The Spend Decision Horizon

There is a point in the architecture lifecycle where cost becomes structurally committed and no longer meaningfully adjustable through reporting. Call it the Spend Decision Horizon.

Before that horizon, cost is a design variable. Service topology, data movement paths, replication models, control plane placement, GPU sizing, retention architecture, egress dependencies, idle capacity policy — these decisions are live. The architect is in the room. The cost outcome is still shapeable.

After that horizon, cost is an observation. Dashboards appear. Tagging spreads. Allocation reports get generated. Anomaly alerts fire. Monthly optimization reviews happen. None of those activities change the architecture that produced the number.

The Spend Decision Horizon is not a concept. It is a handoff. Before it, the architect owns cost. After it, FinOps has the receipt.

The reason most cost governance programs underperform is that they are built entirely on the right side of that horizon. They are sophisticated receipt-reading operations with no authority over what gets ordered.

Where Cost Actually Gets Locked In

The Spend Decision Horizon is defined by five commitment points — the moments where spend transitions from negotiable to structural.

1. Data path design

How data moves through your architecture determines a significant portion of your recurring cost before a single workload runs. Cross-region reads, replication, egress, archive retrieval — these are not line items you optimize after deployment. They are the outcome of topology decisions made during design. Once the data path is established, the cost model follows it.

2. Control plane decisions

Always-on orchestration, management overhead, idle infrastructure, and operational tooling all carry a cost that compounds at scale. The control plane was placed before FinOps arrived.

3. Capacity forecasting

Peak-sized clusters, overprovisioned GPU infrastructure, and statically allocated compute are the loudest signals in any cost audit. But the overprovisioning was a forecast decision, not a utilization decision. GPU idle is a capacity forecasting failure, not a scheduler problem — and the same logic applies across all compute layers. You cannot optimize your way out of a demand model that was wrong at provisioning time.

4. Platform abstraction choices

Managed services, proprietary data layers, and convenience abstractions trade operational simplicity for structural spend commitment. Data gravity is the mechanism: once data accumulates around a managed platform, movement cost locks in. Vendor lock-in happens through the networking layer, not through APIs — and by the time the cost is visible in a dashboard, the dependency chain is already load-bearing.

5. Recovery architecture

Standby duplication, replication tax, and restore-path cost are a function of how recovery was designed. The replication model, the standby footprint, and the recovery tier placement all commit spend at design time. FinOps sees the storage and compute bill. It does not redesign the recovery architecture.

Why FinOps Can See Waste But Not Remove It

This is not a criticism of FinOps. It is a description of its structural position in the decision chain.

FinOps can identify unused resources, overprovisioned instances, bad commitment purchases, idle capacity, and untagged spend. That visibility is real and valuable. The problem is that identifying the consequence is not the same as owning the cause.

FinOps typically cannot change:

The service topology
The platform selection
The replication model
The dependency chain
The control plane footprint
The egress architecture

Those decisions were made by architects, platform teams, and engineering leads — usually without cost explicitly modeled as a design constraint. AI inference cost is the clearest current example: the decision to use a particular model, route to a particular endpoint, or replicate across a particular region commits spend that observability tooling can surface but not prevent.

There is a pattern that has emerged as FinOps has scaled into larger organizations: shared ownership becoming no ownership. When cost accountability is distributed across engineering, finance, and platform teams without clear authority over architectural decisions, the observation layer grows while the control layer stays frozen. More people watching the dashboard. Nobody with authority to change what the dashboard is measuring.

Cost Control Starts Before Deployment

The corrective framing is not a checklist. It is a single shift in where cost enters the architecture conversation.

Cost control starts at:

Architecture review, where topology and data path decisions are still live
Workload placement, where capacity forecasting is still a design input
Control plane design, where operational overhead is still negotiable
Dependency design, where platform abstraction tradeoffs are still explicit
Demand modeling, where GPU scheduling and capacity shape are still shapeable Not after the bill arrives.

The teams that consistently achieve meaningful cost efficiency are not the ones with the best dashboards. They are the ones that treat cost as a first-class architectural constraint — alongside reliability, security, and performance — before the first resource is provisioned.

Cost visibility is not the problem. Visibility is useful. The problem is treating it as the solution.

Architect's Verdict

The FinOps stack has never been more sophisticated. Spend is visible. Allocation is granular. Anomalies are caught faster. Optimization recommendations are automated. And organizations are still wasting a third of their cloud budgets on structural decisions that no amount of dashboard sophistication can undo.

Visibility is lagging telemetry. It describes the cost of decisions already made. It cannot reach back across the Spend Decision Horizon and change the topology, the platform choice, the replication model, or the capacity forecast that produced the number.

Cost control is not a reporting discipline. It is an architecture discipline. The five commitment points — data path, control plane, capacity forecasting, platform abstraction, and recovery architecture — are where spend is decided, not observed. Governance programs built entirely after those decisions are sophisticated receipt-reading operations with no authority over what gets ordered.

The Spend Decision Horizon is not a concept. It is a handoff. Before it, the architect owns cost. After it, FinOps has the receipt. The question is not whether your dashboards are good enough. The question is how much of your cost structure was already committed before FinOps was ever in the room.

Originally published at rack2cloud.com

Your AI Cluster Is Idle 95% of the Time

NTCTech — Tue, 28 Apr 2026 11:58:10 +0000

Your GPU utilization dashboard reads 40%. The cluster is healthy. The GPUs are loaded.

Except they're not working.

That 40% is a peak average across a monitoring window. It doesn't show the forty minutes after the spike when the inference queue drained and the cluster sat fully provisioned against a trickle of requests two nodes could have handled.

The cluster isn't underutilized. It's mispriced against actual demand.

That's a different problem with a different root cause — and the mistake that created it didn't happen in your scheduler. It happened at design time.

Why GPU Utilization Numbers Lie

Most monitoring platforms conflate two things with almost nothing in common: memory residency and compute activity.

A GPU can be fully loaded — model weights resident, tensors staged, inference engine warm — and simultaneously producing zero output. The Kubernetes GPU resource model treats GPU allocation as binary: assigned or not. There's no native distinction between memory-resident and compute-active states.

The hardware is occupied. No work is being done.

Loaded ≠ Active.

A model resident in VRAM is not a GPU doing work. It's a GPU holding a reservation. Most teams treat model-loaded status as GPU-in-use status and provision accordingly. That single assumption is responsible for more mispriced AI capacity than any scheduling inefficiency or orchestration gap.

The Three GPU Utilization Idle Modes

Not all idle compute is the same problem. Before you can fix the architecture, you need to name which mode you're in.

Batch Idle — The gap between training runs. The cluster stays hot between jobs because cold startup costs are high. That gap, multiplied across a training schedule, is pure idle compute priced at full cluster cost.

Inference Idle — The model is loaded. The inference engine is warm. Requests are arriving — just not at the rate the cluster was sized for. GPU utilization metrics show the GPUs as occupied. The memory utilization is real. The compute utilization is not.

Provisioning Idle — The earliest failure and the most expensive one over time. The cluster was sized for a workload that hasn't arrived yet. Peak inference demand for Q3. The large model run that's six weeks out. The hardware is live, the cost is running, and the demand it was priced against exists only in a planning document.

All three modes share one root cause: the demand curve was never modeled correctly.

This Was a Forecasting Failure

The framing that gets used for this problem is utilization. The fix must be better scheduling, better bin-packing, better autoscaling. That framing is wrong.

Low utilization is an output. The input was a provisioning decision made without adequate demand modeling.

Here's what the forecasting actually missed:

The demand curve was never modeled. Teams provisioned for theoretical peak without modeling actual request distribution across a typical operating window. Peak is real. It is also rare.
Concurrency was assumed, not measured. Most provisioning decisions are made against a single-request mental model — how fast can the cluster serve one request — rather than against a concurrent request distribution.
Residency was mistaken for throughput. A GPU holding a 70B parameter model in VRAM is not a GPU running at capacity. It's a GPU with a very expensive reservation.
Runtime limits were never set. Without execution budgets, the cluster expands to fill whatever headroom exists — and headroom was built in generously because the demand model was peak-anchored. Most teams never modeled the demand curve. They sized for theoretical peak, provisioned for future concurrency, and treated loaded memory as active work.

Did you model request concurrency before you provisioned — or did you just size for the busiest hour you could imagine?

What the Math Actually Looks Like

An 8× A100 cluster runs approximately $38,000/month in total cost of ownership. At 5% sustained utilization:

Monthly cluster cost:     $38,000
Sustained utilization:        5%
Productive compute/month:  $1,900
Idle compute/month:       $36,100

Annual forecasting error: $433,200

This is not a slightly inefficient cluster. It's a six-figure architecture constraint that compounds every month the provisioning assumption goes uncorrected.

This Is an Architecture Problem, Not a Scheduling Problem

The standard response to low GPU utilization is a scheduling intervention: deploy Volcano, tune KEDA, implement DCGM-based autoscaling.

These are real tools. They solve real problems. They do not fix this one.

Schedulers optimize execution of work that has been correctly provisioned for. What they cannot do is retroactively correct a demand model that was wrong at design time. If the cluster was provisioned for 10× the actual sustained request rate, a better scheduler produces a more efficiently idle cluster.

Schedulers can distribute work. They cannot fix demand you modeled incorrectly.

That fix happens before the cluster exists. It happens at design time, against a demand curve someone actually drew.

Architect's Verdict

The gpu utilization problem is not a utilization problem. It's a forecasting problem that manifests as gpu utilization data, gets diagnosed as a scheduling problem, and gets treated with tooling that addresses the symptom while the root cause compounds every billing cycle.

The central mistake is a category error: treating memory residency as compute activity. Every GPU idle mode — batch, inference, provisioning — traces back to a demand curve that was never drawn or was drawn incorrectly against theoretical maximums that rarely materialize in production.

The teams that solve this aren't running more sophisticated schedulers. They're provisioning against actual request distributions, modeling concurrency from measurement rather than assumption, and treating loaded memory as exactly what it is: an expensive placeholder.

Fix the demand model first. Everything else is optimization on top of a correctly sized foundation.

Originally published at rack2cloud.com

etcd Is Your Kubernetes Database: What Breaks and What to Watch

NTCTech — Sat, 25 Apr 2026 12:26:50 +0000

etcd is the only component in your Kubernetes control plane that holds state.

Not your API server. Not your scheduler. Not your controller manager. etcd.

If etcd is slow, your cluster is slow. If etcd is inconsistent, your cluster is inconsistent. If etcd fails, your control plane doesn't degrade — it stops.

Most teams don't think about this until the cluster starts behaving in ways they can't explain.

What etcd Actually Does

The API server is stateless. It validates your request, writes desired state to etcd, and returns. The scheduler watches etcd. The controller manager watches etcd. Every pod definition, secret, ConfigMap, lease, and node registration — written to etcd first, read from etcd later.

Kubernetes is a state machine. etcd is the state.

What Breaks (And Why It Doesn't Look Like etcd)

etcd failures don't surface as "database errors." They surface as:

kubectl get pods hanging for seconds
Pods stuck in Pending or Terminating indefinitely
Deployments not rolling, ReplicaSets not scaling
Leader election flapping and log storms across control plane components None of these point at etcd in your dashboard. They look like scheduler bugs, kubelet problems, or network weirdness. The actual cause is one layer below everything you're checking.

The 4 Failure Modes Nobody Monitors

1 — Disk Latency

etcd is disk-bound, not CPU-bound. Every write requires an fsync before it acknowledges. Slow IOPS = slow writes = slow API server = slow cluster. The entire call chain collapses to the speed of your disk.

This is why etcd requires SSD or NVMe. NFS and gp2 EBS will quietly degrade your control plane under load.

2 — Quorum Instability

3-node cluster: needs 2 to agree. 5-node: needs 3. Lose quorum and the cluster goes read-only — no writes, no scheduling, no reconciliation.

Common mistakes: 2-node clusters (zero quorum tolerance), 4-node clusters (same tolerance as 3, more cost), etcd members stretched across high-latency zones. Raft heartbeat timeouts are tuned for <10ms inter-member latency. Exceed that under normal load and you'll see leader elections fire.

3 — Large Object Writes

etcd has a 1.5MB per-value default limit and a 2GB total DB limit (8GB max). Both are reachable.

Usual offenders: CRDs storing runtime state, secrets used as blob storage, ConfigMaps holding multi-MB files. etcd is not an object store. Every oversized write slows the cluster and causes fragmentation.

4 — Compaction and Fragmentation

etcd keeps a history of every key revision. Without compaction, the DB grows unbounded. Without defrag after compaction, the on-disk footprint doesn't shrink.

The pattern: DB grows quietly to several hundred MB, performance softens, nobody connects it to etcd because nothing is explicitly broken. Then a large write event pushes toward the size limit and you have an incident.

The 5 Metrics That Actually Matter

If you're only watching CPU and memory on your control plane nodes, you are not monitoring etcd.

Metric	What It Tells You
`etcd_disk_wal_fsync_duration_seconds`	P99 >10ms = warning. P99 >25ms = problem. Most important etcd metric.
`etcd_server_leader_changes_seen_total`	Should be near zero. Frequent changes = instability.
`etcd_mvcc_db_total_size_in_bytes`	Track growth rate. Growing faster than your cluster = something over-writing.
`etcd_mvcc_db_total_size_in_use_in_bytes`	Large gap vs total size = fragmentation.
`etcd_server_slow_apply_total`	Nonzero and growing = investigate before it becomes an incident.

The Rules

DO:

✅ Dedicated local SSD/NVMe for etcd data directories
✅ 3 or 5 members — always odd, never 2 or 4
✅ Monitor fsync latency as your primary health signal
✅ Automate compaction and defragmentation
✅ Snapshot etcd — treat it like a production database backup DON'T:
❌ Co-locate etcd with noisy high-I/O workloads
❌ Store large payloads in ConfigMaps or Secrets
❌ Ignore fragmentation growth
❌ Assume managed etcd (EKS/GKE/AKS) needs no visibility
❌ Treat etcd as a transparent implementation detail

The Part Most Architectures Skip

Your pods can fail and reschedule. Your nodes can fail and drain. etcd loses quorum and your cluster stops accepting writes — full stop. No automatic recovery, no clever failover, no workload that routes around it.

Most Kubernetes architectures are designed assuming etcd works. Very few are designed for when it doesn't.

Treat etcd like the database it is — because it's the most important one in your cluster.

If etcd is slow, Kubernetes lies to you. If etcd is unavailable, Kubernetes stops. If etcd is corrupted, recovery becomes a rebuild problem — not a restart.

This post is part of the Modern Infrastructure & IaC series at rack2cloud.com. Full post with architecture diagrams and HTML signal cards at rack2cloud.com/etcd-kubernetes-database.

Operating Gateway API in Production: What the Migration Guides Don't Cover

NTCTech — Thu, 23 Apr 2026 13:14:15 +0000

You migrated. Traffic is flowing. ReferenceGrants are in place. The controller reconciliation loop is clean. And then — quietly, without a single alert firing — things start breaking in ways your observability stack was never built to see.

Most Gateway API migration guides end at cutover. That is the wrong place to stop. The real operational surface of gateway API production begins exactly where those guides close — and it is governed by a different set of failure physics than anything Ingress introduced.

The thesis is explicit: Gateway API doesn't just change how traffic is routed. It changes where routing failures live — and how invisible they become.

The Gap Nobody Talks About

Part 0 was the decision. Part 1 was the shift. Part 2 was the migration. Part 3 is the reality.

When you ran Ingress, failures were infrastructure-visible. A misconfigured annotation broke routing and your logs showed it. A missing backend returned a 502 and your alerting fired. The failure surface was shallow and legible.

Gateway API moves routing failures into the decision layer. HTTPRoutes can be accepted by the controller — syntactically valid, status condition green — while silently misrouting traffic. ReferenceGrants can be deleted during a routine namespace cleanup with no downstream alert. Header matching logic from the annotation era doesn't translate 1:1, and the mismatch produces no error. It just routes incorrectly.

This is not a tooling gap. It is an architectural one.

Observability: What Changes After Gateway API

Ingress failures were infrastructure-visible. Gateway API failures are decision-layer invisible.

Understanding what your monitoring stack actually covers requires mapping it against three distinct layers:

Layer 1 — Controller Metrics (What You Get)

Standard Prometheus scraping covers the controller layer. Reconciliation loop latency, controller health, memory and CPU. This is the layer most teams think of as "Gateway API observability" — and it is the least useful layer for diagnosing production routing failures. A healthy controller reconciliation loop tells you nothing about whether the routing decision it produced is correct.

Layer 2 — Spec State (What You Miss)

HTTPRoute status fields are not surfaced by default in most monitoring stacks. The conditions you need to be watching — Accepted, ResolvedRefs, Parents — exist in the Kubernetes API but require explicit instrumentation. A route in Accepted: True with a backend in ResolvedRefs: False will route requests to nothing — and your controller metrics will show green the entire time.

Layer 3 — Runtime Behavior (What Actually Matters)

Routing outcomes, backend selection, header and path matching decisions. 200 OK is the new 500: a request that returns a success status from the wrong backend is operationally identical to a silent outage. Runtime behavior requires traffic-level instrumentation — service mesh telemetry, eBPF-based flow data, or access log enrichment — to become visible.

Your monitoring stack sees the controller. It does not see the routing decision.

Policy Enforcement at the Gateway Layer

Gateway API introduces routing-level trust boundaries, not just network boundaries. The real shift is temporal:

NetworkPolicy → Packet-level, always-on
OPA / Gatekeeper / Kyverno → Admission-time, pre-deploy
Gateway API → Runtime routing authorization, request-time ReferenceGrant is not configuration. It is a security boundary.

A ReferenceGrant deletion — which can happen silently during namespace cleanup, RBAC rotation, or automated resource pruning — immediately collapses cross-namespace routing trust. There is no deprecation window. Traffic stops reaching its backend, and the only signal is a ResolvedRefs: False condition that most teams aren't alerting on yet.

The Day-2 Failure Patterns

These are not edge cases. These are the failures teams discover in the first 30–60 days of production.

Failure Mode 01 — Route Accepted, Traffic Misrouted
Accepted: True means valid configuration — not correct behavior. Backend weight misconfiguration, path prefix overlap, or header match ordering errors produce accepted routes that route to the wrong destination. No alerts fire. Traffic just goes somewhere wrong.

Failure Mode 02 — Cross-Namespace Trust Collapse
ReferenceGrant deleted during routine cleanup. Cross-namespace routing immediately fails. The backend is healthy, the controller is healthy, the HTTPRoute status goes ResolvedRefs: False and traffic stops. Recovery requires manual ReferenceGrant reconstruction.

Failure Mode 03 — Header Routing Regression
Annotation-era header logic doesn't translate 1:1 to HTTPRoute match semantics. The route is accepted, the match appears correct in the spec, and the wrong backend receives traffic silently.

Failure Mode 04 — Controller Version Skew
Gateway API evolves faster than most controller upgrade cycles. HTTPRoutes that reference unsupported features are accepted but silently not enforced — the spec says it should work, the controller says nothing, and behavior is undefined.

Failure Mode 05 — TLS Cert Rotation Gap
cert-manager and Gateway API have different mental models of certificate binding. Rotation timing mismatches produce TLS termination failures that appear as backend connectivity issues — not certificate errors — in most monitoring stacks.

Multi-Cluster and Multi-Tenant Considerations

Gateway API simplifies single-cluster routing. It complicates multi-cluster ownership.

The fundamental shift at multi-tenant scale: the problem is no longer routing. The problem is who is allowed to define routes.

Gateway-per-team is the operationally cleaner model for most enterprises — blast radius is contained, ReferenceGrant surface is minimal. The shared Gateway model reduces resource overhead but introduces a ReferenceGrant audit problem at scale that platform engineering needs to own, not application teams.

Cross-cluster route federation remains experimental. Model it as beta operationally, regardless of what the controller documentation claims.

The Real Problem

Teams think they migrated an ingress layer. What they actually introduced is a new control plane.

This is the thread that runs through the entire series. The control plane shift isn't a Gateway API phenomenon — it is the defining architectural pattern of this infrastructure era. Every layer that used to be configuration is now a control plane: service meshes, policy engines, GitOps operators, and now routing.

The teams that operate Gateway API well in production are not the ones with the best controllers. They are the ones that rebuilt their observability model before they needed it.

Gateway API doesn't fail loudly. It fails in decisions your tooling doesn't see.

Architect's Verdict

Part 0 was the decision. Part 1 was the shift. Part 2 was the migration. Part 3 is the reality — and the reality is that Gateway API production operations require a fundamentally different observability model, a new policy enforcement layer, and an audit discipline that didn't exist when you were running Ingress.

DO:

Treat Gateway API as a control plane layer — instrument routing decisions, not just traffic
Alert on HTTPRoute status conditions — ResolvedRefs: False is a production incident
Audit ReferenceGrants continuously — treat deletions as security boundary changes, not cleanup
Pin controller versions to the Gateway API channel they implement — track skew explicitly
Own the ReferenceGrant audit function at the platform engineering layer DON'T:
Assume Accepted: True means working — it means syntactically valid configuration
Treat migration as completion — cutover is the start of the operational surface, not the end
Let controller behavior drift from spec assumptions
Port Ingress annotation logic directly to HTTPRoute without verifying match semantics
Trust cross-cluster Gateway API federation claims without verifying your controller's implementation channel

Architecture diagrams and full failure mode breakdown at rack2cloud.com

Series:

Part 0: Ingress-NGINX Deprecation: What to Do Next
Part 1: Gateway API Is the Direction. Your Controller Choice Is the Risk.
Part 1.5: The Control Plane Shift
Part 2: Kubernetes Ingress to Gateway API Migration
Part 3: Operating Gateway API in Production ← You Are Here

Kubernetes Is Not an LLM Security Boundary

NTCTech — Wed, 22 Apr 2026 12:50:44 +0000

The CNCF flagged it three days ago. Most teams haven't processed what it actually means.

Kubernetes lacks built-in mechanisms to enforce application-level or semantic controls over AI systems. That's not a bug. It's not a misconfiguration. It's a category error in how we're thinking about AI workload security.

Kubernetes isolates containers. It does not isolate decisions.

What Kubernetes Actually Controls

To be clear about the problem, you need to be precise about the scope.

Kubernetes enforces pod isolation, RBAC, network policy, resource limits, and admission control. A well-configured cluster with Cilium, Kyverno, and Falco is genuinely hardened.

All of those controls operate at the infrastructure layer. None of them understand what an LLM is doing inside that boundary.

The Three-Layer Problem

Think of it as three distinct boundaries:

Infrastructure Boundary (Kubernetes): Controls compute, network, identity. Cannot see model behavior, prompts, or outputs.

Application Boundary: Controls API access and service logic. Cannot see model reasoning or semantic intent.

LLM Boundary — the actual risk layer: Controls prompts, outputs, tool usage. This is the layer your current tooling doesn't reach.

Most teams have the first two layers covered. The third is largely unaddressed.

The Failure Mode Kubernetes Will Never Catch

Here's the production scenario that matters:

User submits a prompt with a hidden injection instruction
Model retrieves internal context via RAG
Model outputs sensitive internal data in its response
Response returns HTTP 200
No alerts fire. No logs capture what the model decided. From Kubernetes' perspective: successful request. Pod healthy. RBAC respected. Latency within SLA.

From a security perspective: complete boundary failure.

This is the observability inversion. Traditional monitoring asks: did it run? was it fast? did it error?

LLM observability needs to ask: was it correct? was it safe? was it allowed?

Infrastructure observability measures execution. LLM observability measures outcomes.

What the Actual Boundary Requires

Four control layers need to exist above Kubernetes:

Ingress Control — prompt validation and injection filtering before the model sees the request.

Egress Control — output scanning and PII detection before the response leaves the system.

Action Control — for agentic systems with tool access, explicit allow-lists scoped per model and context. RBAC governs which service account can call which API. This governs which model, in which context, is permitted to trigger which action. Not the same constraint.

Audit Control — sovereign, immutable inference logging. If your inference logs live in a vendor's platform, you don't fully own the audit trail.

Emerging implementations like Kong AI Gateway and Portkey are building toward this pattern — but the pattern matters more than the product. These four components need to exist regardless of what implements them.

When Kubernetes Is Enough

To be honest: there are AI workloads where infrastructure controls are sufficient.

Stateless, isolated LLM — no persistent context
No tool access — text output only
No sensitive context in scope
No external system impact If your workload meets all four conditions, your infrastructure boundary largely holds.

The moment you add RAG retrieval, tool use, memory, or agentic orchestration — any one of them — you're operating at the LLM Boundary layer, and Kubernetes alone isn't sufficient.

Most enterprise AI workloads don't meet those conditions.

The Practical Takeaway

Your Kubernetes security posture is necessary. It is not sufficient for LLM workloads.

The cluster can be hardened. The model is still non-deterministic. Those are two different problems requiring two different control layers.

If you're running LLMs on Kubernetes with only infrastructure-layer controls, you have a boundary problem you haven't measured yet. The absence of alerts isn't evidence of safety — it's evidence that your observability doesn't reach the layer where LLM risk lives.

Full architecture breakdown including the LLM Security Boundary Model and LLM Control Plane Pattern framework at rack2cloud.com

AVS Is a Migration Strategy. Treating It as a Destination Is the Mistake.

NTCTech — Tue, 21 Apr 2026 12:20:25 +0000

Most teams evaluating Azure VMware Solution frame it as an architecture decision.

It isn't. AVS is a migration strategy — and the moment you start treating it as a destination, the financial and architectural consequences start compounding.

The Framing Problem

AVS looks like the safe path out of a Broadcom licensing conversation. Your team knows vSphere. Your tooling maps to VMware constructs. You move workloads without retraining anyone or rearchitecting anything.

What you're not choosing is where to run workloads. You're choosing how hard it will be to leave later.

AVS feels like staying on-prem — just relocated into Azure's billing model. That's the trap. Because you're not escaping VMware. You're relocating it into a metered, provider-controlled environment.

AVS doesn't remove lock-in. It changes where the lock-in lives.

What Changes When You Land on AVS

The familiar operational surface is real. vSphere, vSAN, NSX-T — your ops team recognizes everything they're looking at. Microsoft operates the hardware layer. You operate the guests.

What you lose is the exit path you had on-prem.

On-prem exit cost is physical and operational. AVS exit cost is financial, architectural, and contractual — simultaneously. When you eventually leave AVS, you're not executing a migration. You're executing a second transformation: translating VMware constructs to a target platform while simultaneously unwinding a managed service relationship and absorbing Azure egress costs at scale.

AVS exit is not a migration. It's a second transformation.

When AVS Is Correct

There are legitimate use cases — but they're narrower than the sales motion suggests.

AVS makes sense when:

Compliance requirements are written around vSphere-specific behaviors and can't be renegotiated
Your team has deep VMware expertise and no capacity to absorb an operational model shift during migration
You have a defined, dated exit plan to move off AVS onto native Azure within 3–5 years
You have specific application workloads with hard VMware dependencies that have no near-term abstraction path The key phrase is defined exit plan. If you don't have one, AVS becomes your destination by default.

The Hidden Cost Layer

The published price is for compute. The real cost is in everything around it.

Dedicated bare metal at a three-node minimum floor. vSAN storage overhead that materially reduces usable capacity. NSX-T licensing embedded in the bill whether you use the full capability stack or not. And the one most teams miss: traffic between AVS and native Azure services isn't always free. At scale, that adds up fast — and it almost never appears in the initial cost modeling.

The AVS Decision Test

Before finalizing the architecture decision, run one check.

Are you using AVS to:

Buy time for a defined migration? — Valid.
Avoid retraining your team? — Risky deferral.
Delay re-architecting legacy workloads? — Expensive later. Only one of these is a strategy.

The Verdict

AVS as a deliberate bridge with a committed exit timeline is a rational use of the platform. AVS without a defined exit path is deferred lock-in — you've traded Broadcom's licensing model for Microsoft's managed service model, paid for the familiar operational surface, and left yourself with an exit that's more expensive and more complex than what you started with.

Model the exit before you commit to the entry.

Full architectural breakdown — including the trade-off comparison table, exit cost analysis, and native Azure contrast — is on Rack2Cloud: Azure VMware Solution vs Native Azure: Architecture Trade-offs, Costs, and Exit Risk

The Restore Path Is the Most Neglected Part of Backup Design

NTCTech — Sun, 19 Apr 2026 13:37:47 +0000

The restore path is where backup architectures fail — not the backup job, not the retention policy, not the storage tier.

This is not an operations failure. It is a design omission.

Most architectures are designed to write data — not to get it back.

The Backup Job Is Not the Goal

Most backup architectures are designed around the protection plane — backup jobs complete, retention windows are enforced, replication targets are confirmed. Dashboards go green. SLA reports are generated. The architecture is declared healthy.

None of that measures whether recovery actually works.

A backup job confirms that data was written to a target at a point in time. It tells you nothing about whether that data can be read back under load, whether the application stack can be reconstructed in the correct sequence, whether identity dependencies survive the restore, or whether the recovered state is consistent at the application layer rather than just bootable at the VM layer.

The restore path is the sequence of operations, dependencies, and decision points between a backup completion event and a verified, production-usable recovered state. It is not a single operation. It is an architecture — and most teams have never designed it.

A successful backup proves nothing about your ability to recover.

What the Restore Path Actually Contains

Recovery doesn't fail in one place. It fails across layers that were never designed together.

A functional restore path has four layers that must be explicitly designed, not assumed:

Data retrieval. Where does the backup live, how long does retrieval take, and what are the network and hydration constraints at scale? Object storage restore speeds differ from on-premises targets by orders of magnitude. Cloud archive tiers introduce retrieval latency that can turn a four-hour RTO into a 48-hour one. The rehydration bottleneck is real — and it belongs in the design, not the postmortem.

Dependency sequencing. What order do workloads need to come back online? Databases before application tiers. Identity before anything that authenticates. DNS before anything that resolves. Most organizations have never documented this sequence. The engineers who know it are the ones who happen to be on call during an incident — and that is not an architecture. That is institutional knowledge waiting to walk out the door.

Identity bootstrap. If the production identity plane is compromised or unavailable, what does the recovery environment authenticate against? This is the question that stops most recoveries cold. Ransomware operators understand this — they target the identity plane specifically because a workload that cannot authenticate is not a recovered workload. It is a running VM with no access path.

Application-layer validation. A restored VM that boots is not a recovered application. Application-consistent recovery requires more than a successful backup job — it requires that the restored state is usable at the application layer, not just reachable over the network. Hash validation, restore pipelines, and application-layer health checks must be defined before an incident, not improvised during one.

Why Teams Skip It

The restore path is ignored because it doesn't produce visible success.

There is no dashboard for "can we actually recover."

Backup vendors measure protection-plane health because that is what they can instrument. Job completion rates, storage utilization, replication lag — these are real signals about a system that is working as designed. Recovery-plane health requires the organization to design and test it independently. No vendor ships a product that validates your dependency sequencing documentation or your identity bootstrap runbook. That work belongs to the architect.

The result is a discipline where the visible work gets done and the invisible work gets skipped. Recovery drills exist precisely to surface this gap — but most teams treat them as a compliance exercise rather than an architectural stress test. A drill that confirms the backup is readable is not a recovery test. A recovery test proves the entire restore path — retrieval, sequencing, identity, application validation — executes within the declared RTO under realistic conditions.

Backup success is easy to measure. Recovery success requires you to prove your assumptions wrong.

The Restore Path as a Design Constraint

Recovery is not a procedure problem. It is a constraint problem.

Your RTO is not a target. It is the output of constraints you probably haven't modeled.

Those constraints include retrieval throughput ceilings at your backup target tier, hydration time at scale, network path availability between the recovery environment and the backup source, identity availability in an isolated recovery context, and application dependency ordering that cannot be parallelized. Each constraint has a measurable impact on recovery time. Most organizations have modeled none of them.

The RTO in most DR documentation is not derived from constraint analysis. It is a number someone wrote down during a compliance exercise — unchallenged, untested, and disconnected from the actual physics of the restore path. When the incident arrives, the gap between the documented RTO and the real recovery time is not a surprise. It is the predictable output of skipping the constraint modeling.

The Three-Layer Resilience Model treats recovery as a distinct architectural layer — Layer 3, with its own design requirements and failure modes, separate from backup and DR. The restore path is the operational expression of that layer. If it has not been designed, Layer 3 does not exist regardless of how many backup jobs are completing successfully.

Architect's Verdict

If your organization has a documented backup architecture and no documented restore path, you have half a data protection design. The backup plane tells you that data exists somewhere. The restore path determines whether you can use it when it matters. Teams that invest in protection-plane completeness without modeling restore-path constraints are not protected — they are insured against a risk they have not actually priced.

Design the restore path with the same rigor you applied to the backup architecture. If you haven't tested your restore path against real constraints, your RTO isn't a commitment. It's a guess.

Originally published at rack2cloud.com

Agentic AI Has a Control Plane Problem — Because It Became the Control Plane

NTCTech — Fri, 17 Apr 2026 13:04:36 +0000

Agentic AI control plane governance is the architecture problem most teams are not modeling — and the one that will produce the most expensive failures in 2026.

The control plane became the most sensitive layer in modern infrastructure. So we locked it down.

Kubernetes gave us control plane isolation — the API server, etcd, and the scheduler separated from the workloads they govern. IAM gave us least-privilege scoping — execution authority bounded to the minimum required. Cloud architecture gave us blast radius containment — failure domains designed to limit the lateral spread of a single misconfiguration or breach.

We spent a decade building these constraints. They are not theoretical. They are the operational lessons of every infrastructure failure that taught us what happens when execution authority goes ungoverned.

Agentic AI reintroduces the same problem — without the controls.

We Rebuilt an Agentic AI Control Plane and Skipped Every Safeguard

The mapping is direct. Every infrastructure concept that governs how control planes operate has an agentic equivalent. None of them carry the governance model forward.

Infrastructure Concept	Agentic Equivalent	What's Missing
Control plane API	Tool / API invocation	Policy enforcement layer
IAM roles	Agent credentials	Scope boundaries, auditability
etcd / state store	Memory / vector store	Versioning, governance, access control
Orchestrator	Agent runtime	Isolation boundary

Every column on the right exists in agentic systems today. None of them carry the operational discipline that made the left column safe to run in production.

We spent a decade separating execution from control. Agentic AI collapses that boundary again.

The Agent Is No Longer an Application

This is where the architecture regression becomes a structural risk.

An application calls an API. An agent invokes tools, persists state, chains actions across systems, and makes decisions that trigger further actions — autonomously, at machine speed, across infrastructure it does not own.

That is not an application. That is a control plane with execution authority.

The distinction matters because the entire governance model for applications assumes bounded execution. An application has a defined scope. It calls what it is told to call. It does not decide. An agent decides — and those decisions have downstream effects across every system it can reach.

Most teams are treating agentic AI as a new class of application. They are deploying it inside the application layer, scoping its credentials like a service account, and monitoring it with the same observability stack they use for stateless workloads.

This is the architectural mistake. The agent is not operating at application scope. It is operating at control plane scope. And when a control plane runs without isolation, without enforced policy, and without bounded execution authority — you already know how that ends. You've seen it at the infrastructure layer.

This class of risk has a name: Unbounded Control Planes — a control plane that can create actions, without enforced policy, across systems it does not own.

Kubernetes failed closed. Agentic systems fail open.

The Four Failure Modes That Only Surface in Production

Failure Mode 01 — Credential Amplification

Agents aggregate permissions across every tool they can invoke. The effective access scope is broader than any single IAM role you reviewed at deployment. Blast radius is not the agent's scope — it is the union of every system it can reach.

Failure Mode 02 — Unbounded Execution Chains

One prompt becomes twelve API calls across three systems before a human sees any output. Each step can trigger the next. There is no circuit breaker, no step boundary, no re-evaluation gate. The execution chain is only visible after the damage is already distributed.

Failure Mode 03 — State Persistence Without Governance

Agent memory is not a cache. It is a state layer that shapes every future decision. It is not versioned, not scoped, not audited. When it influences a cross-system action six interactions later, the dependency is invisible — until a failure event forces the trace.

Failure Mode 04 — No Control Plane Isolation

The agent runtime lives inside the application layer. Its credential scope operates at infrastructure authority. There is no isolation boundary between where the agent executes and what it can modify. The application perimeter does not contain infra-level execution authority.

What Architects Need to Get Right (Before This Breaks in Production)

The answer is not a new security framework. It is the governance model you already built for infrastructure — applied deliberately to a layer that is behaving like infrastructure whether you designed it that way or not.

1. Treat Agent Credentials as Control Plane Credentials

If an agent can invoke APIs, it holds infrastructure authority — not application scope. No shared tokens. No implicit trust. Scoped, auditable, revocable — the same standard you apply to anything that can modify state at the infrastructure layer.

Agent identity is not app identity. It is control plane identity.

2. Isolate the Agent Runtime from the Systems It Controls

An agent should not operate inside the same blast radius as the resources it can modify. The execution boundary needs to be explicit — separate runtime, no direct lateral access, mediation layer between the agent and the systems it reaches.

If the agent lives inside your application layer, your control plane is already compromised.

3. Govern Memory as State — Not as a Feature

Persistent memory is not context. It is a state layer that influences future actions across systems. Version it. Scope it. Audit it. Apply the same governance you would apply to any state store that participates in cross-system decision-making.

Unbounded memory creates untraceable behavior.

4. Constrain Execution — Agents Should Not Chain Without Boundaries

The risk is not a single action. It is the accumulation of actions across systems without re-evaluation gates. Limit tool chaining. Enforce step boundaries. Require explicit re-evaluation before an agent proceeds across a system boundary.

Unbounded execution is how small decisions become systemic failures.

5. Reintroduce the Control Plane Boundary — Explicitly

Define where the agent's authority begins and ends before deployment, not after the first production incident. If you do not define the boundary, the agent will — and it will define it as broadly as its credentials allow.

We did not lose control of infrastructure because systems became complex. We lost control when we stopped enforcing boundaries. Agentic AI removes those boundaries by default. Architects need to put them back — deliberately.

Architect's Verdict

The agent is your agentic AI control plane.

If your agent can take action across systems, it is part of your control plane — whether you designed it that way or not. The governance model, the isolation requirements, the credential discipline — none of that is optional at control plane scope. You already know this. You built it once. The only question is whether you apply it again before production forces the lesson.

Architecture diagrams and full failure mode breakdown at rack2cloud.com.

Kubernetes Ingress to Gateway API Migration: How to Move Without Breaking Production

NTCTech — Wed, 15 Apr 2026 12:37:36 +0000

Most Gateway API migrations don't fail during the cutover.

They fail in the translation layer — quietly, before traffic ever moves. The annotation audit skipped. The ingress2gateway output treated as deployment-ready. The staging environment that shared none of the complexity of production. By the time the failure surfaces, it looks like a Gateway API problem. It isn't. It's a migration preparation problem.

Ingress-NGINX hit EOL on March 24 — the repository is read-only, no patches, no CVE fixes. Kubernetes 1.36 drops April 22 with Gateway API as the centerpiece. The window where this was a future consideration closed.

Before You Migrate — The Annotation Audit

The annotation count per Ingress resource is the number that determines which migration path is actually viable. Run this before anything else:

# Count annotations per ingress resource across all namespaces
kubectl get ingress -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name): \(.metadata.annotations | length) annotations"' | \
  sort -t: -k2 -rn

Three tiers, three different migration realities:

0–5 annotations — ingress2gateway 1.0 handles 80–90%. Most of what lands in your HTTPRoute manifests will be correct. Manual review still required.

6–20 annotations — partial translation. Common annotations (CORS, backend TLS, path rewrite, regex) are covered. Less common ones — configuration-snippet, auth-url, server-snippet — require architectural decisions.

20+ annotations — the tool cannot help you. What those annotations are collectively doing needs to be understood and redesigned before a single manifest is written.

Also find shared Ingress resources — single Ingress objects routing 40+ hostnames for multiple teams. These are coordination problems, not migration targets:

kubectl get ingress -A -o json | \
  jq -r '.items[] | select(.spec.rules | length > 5) |
  "\(.metadata.namespace)/\(.metadata.name): \(.spec.rules | length) host rules"'

ingress2gateway 1.0 — Syntax Translator, Not Architecture Translator

ingress2gateway 1.0 is a genuine improvement — supports 30+ common Ingress-NGINX annotations with behavioral equivalence tests that verify runtime behavior in live clusters, not just YAML structure.

ingress2gateway print \
  --providers=ingress-nginx \
  --namespace=production

Translates cleanly: host/path routing, TLS referencing existing Secrets, CORS headers, backend TLS, path rewrites, regex matching.

Does not translate:

nginx.ingress.kubernetes.io/configuration-snippet — custom Lua, no Gateway API equivalent
nginx.ingress.kubernetes.io/server-snippet — server-level config, no direct equivalent
nginx.ingress.kubernetes.io/auth-url / auth-signin — external auth, requires HTTPRoute filter or extension
ConfigMap global defaults — proxy buffer sizes, upstream keepalive, timeout values don't transfer automatically

Implicit defaults that disappear: Ingress-NGINX's ConfigMap applies defaults globally that aren't in your Ingress manifests. They don't transfer. Document your ConfigMap before migration.

What to Migrate First

Migration sequence matters more than migration speed.

Migrate first: New services with no Ingress config. Internal services with 2–3 host rules and no custom annotations. These establish the operational pattern.

Migrate second: Services with standard CORS, TLS, and path rewrite annotations ingress2gateway handles cleanly. Validate behavioral equivalence before decommissioning each Ingress resource.

Migrate last: configuration-snippet services, external auth integrations, shared Ingress resources, anything with a P1 incident in the last 90 days.

The Side-by-Side Pattern — The Only Safe Model

Cutover-first is an anti-pattern. Both controllers run simultaneously against the same cluster, sharing the same external load balancer IP.

# Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

# Deploy Gateway API controller alongside existing Ingress controller
kubectl apply -f https://github.com/nginx/nginx-gateway-fabric/releases/download/v1.5.0/nginx-gateway-fabric.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: nginx-gateway
spec:
  gatewayClassName: nginx-gateway
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: production-tls
        namespace: nginx-gateway

The one rule: Never configure both an Ingress resource and an HTTPRoute for the same hostname and path simultaneously. The two controllers compete for the same traffic.

HTTPRoute Translation — Before and After

# Before — Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080

# After — HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: nginx-gateway
  hostnames:
  - "app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: api-service
      port: 8080

Traffic splitting — native in HTTPRoute, no annotations needed:

rules:
- backendRefs:
  - name: api-stable
    port: 8080
    weight: 90
  - name: api-canary
    port: 8080
    weight: 10

Adjacent Dependencies — Address Before First HTTPRoute

cert-manager: Requires v1.14.0+ for Gateway API support. Configuration moves from Ingress annotations to Gateway resource annotations.

ExternalDNS: Requires v0.14.0+ for Gateway API support. DNS records for HTTPRoute hostnames won't be created automatically on older versions — DNS resolution fails silently.

Prometheus/alerting: Gateway API controllers expose different metric structures than Ingress-NGINX. Dashboards keyed to Ingress-NGINX metric names won't work without updates.

DNS Cutover Sequence

All services validated under load via HTTPRoutes in side-by-side state
Keep Ingress resources — rollback safety
Reduce DNS TTL to 60 seconds — 24 hours before cutover
Update external DNS record
Monitor error rates for 30 minutes
Remove decommissioned Ingress resources after 24 hours of clean traffic — not before

Production Failure Modes — Works in Staging, Breaks in Prod

Header routing mismatch — HTTPRoute header matching is exact by default. Ingress-NGINX treats some header matching case-insensitively. Verify your Gateway implementation's behavior explicitly.

ReferenceGrant missing — the most common failure in multi-team clusters. An HTTPRoute in namespace frontend referencing a Service in namespace api requires a ReferenceGrant. Without it: accepted status, 500 response.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-frontend-routes
  namespace: api
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: frontend
  to:
  - group: ""
    kind: Service

TLS handshake surprise — Ingress-NGINX's TLS defaults (cipher suites, protocol versions) live in the ConfigMap. Gateway API controllers start from their own defaults. Validate TLS behavior against legacy clients explicitly before cutover.

Implicit defaults disappearing — proxy timeouts, upstream keepalive, buffer sizes set in the Ingress-NGINX ConfigMap don't transfer. A service relying on a 600-second proxy timeout reverts to the controller's default silently. Audit the ConfigMap before any service migrates.

Architect's Verdict

ingress2gateway 1.0 handles straightforward migrations cleanly. The gap it cannot close is between syntax translation and architectural translation. Find the untranslatable annotations during the audit — not during the rollback.

The side-by-side pattern is the correct one. Both controllers running against the same load balancer IP costs nothing and eliminates the primary risk vector: the all-at-once cutover that discovers production failure modes under incident conditions.

The migration doesn't fail where you think it will. It fails in everything you assumed would just translate.

Part of the Rack2Cloud Kubernetes Ingress Architecture Series. Full post with interactive examples at rack2cloud.com.