DEV Community: Todea

HAMi Explained: Why Your GPUs Sit Idle While Your Bill Doesn't

Ivan Porta — Tue, 14 Jul 2026 16:32:43 +0000

Before the release of Dynamic Resource Allocation and its alpha partitionable-device and consumable-capacity extensions, Kubernetes scheduled GPUs only as whole cards, with no native option to allocate a fraction of a device. For example, if a pod requires only 4 GB of an 80 GB H100, it still reserves the entire GPU for its lifetime. This leads to significant underutilization of expensive hardware in shared clusters. Chatbots designed for peak traffic may remain idle on dedicated cards during off-peak hours; multiple inference services often run on separate GPUs despite low usage, and monitoring dashboards may show all GPUs allocated while tools like nvidia-smi often reveal minimal actual utilization. As a result, new budget requests for additional GPUs arise because the cluster appears to be at capacity. DRA narrows this gap by preventing over-committing a device's declared budget, but it does not natively enforce resource limits once a pod is running, which remains the device driver's responsibility. This is where HAMi provides value.

HAMi (Heterogeneous AI Computing Virtualization Middleware) closes the gap by allowing pods to request an exact portion of a GPU’s memory and compute, with an in-container enforcement layer that holds them to it, with no changes to applications or images. Companies like SNOW and NIO run HAMi in production, and the project reached CNCF Incubating in July 2026.

What is HAMi, actually?

HAMi is an open-source GPU virtualization layer for Kubernetes. It lets a pod request a precise amount of memory and an AI accelerator’s compute, then schedules the pod onto a device with that much free capacity, binding it to that slice at runtime. The same model spans many accelerators, including NVIDIA GPUs, Cambricon MLUs, Hygon DCUs, Ascend NPUs, Moore Threads, MetaX, and others, each via its own device plugin and isolation backend. On NVIDIA, for example, HAMi replaces the stock device plugin and enforces the memory cap by intercepting the CUDA driver API within the container; other vendors provide analogous library- or hardware-level isolation.

The user experience is deliberately simple: the standard device resource request is extended with one or more limits that the scheduler reads at scheduling time to find a card with sufficient free memory and compute resources. On NVIDIA, you keep requesting nvidia.com/gpu and add the two extended resources beside it: nvidia.com/gpumem and nvidia.com/gpucores.

resources:
  limits:
    nvidia.com/gpu: 1       
    nvidia.com/gpumem: 8000 
    nvidia.com/gpucores: 30

Two comparisons put that isolation model in context. MIG draws a hardware boundary: each instance gets its own SMs and a dedicated route through L2 cache slices, memory controllers, and DRAM channels. HAMi’s boundary is a fence, not a wall. Its limits are enforced in software inside the container, which reliably holds workloads to their budgets but cannot contain a device-level failure: a hung process, a driver reset, or an XID error on the shared card still hits every tenant on it. The two work at different layers, though, and they can be complementary rather than competing; we will come back to how HAMi can drive MIG directly.

Compared with time-slicing, however, HAMi offers significant improvements in stability. Time-slicing simply advertises multiple schedulable replicas of a single card and interleaves their workloads on the compute engine, without per-replica memory caps or isolation. That makes it dangerously easy for one workload to allocate past its notional share and take the others down with it. HAMi refuses the excess allocation instead. The following log shows PyTorch reporting a total capacity matching the 1000 MB limit set by nvidia.com/gpumem; when the pod tried to exceed it, the allocator stopped it with an OOM error, leaving the other vGPU tenants unaffected:

(EngineCore pid=95) ERROR 07-11 21:15:46 [core.py:1231] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 1000.00 MiB of which 12.00 MiB is free. Including non-PyTorch memory, this process has 1020.00 MiB memory in use. Of the allocated memory 864.68 MiB is allocated by PyTorch, and 41.32 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

How a shared-GPU request flows through HAMi

From manifest to enforced slice, a request passes through four hands, shown in the diagram below: the device plugin advertises each card’s capacity, the mutating webhook reroutes the pod to HAMi’s scheduler, the scheduler extender picks a node and a slice, and an in-container library enforces that slice at runtime.

On each node, the device-plugin (which runs as a DaemonSet) polls the local devices every 30 seconds via the vendor's management library (e.g., NVML for NVIDIA, which powers nvidia-smi). It then publishes this telemetry data by patching the node with a hami.io/node-<vendor>-register annotation.

kubectl get nodes gtrekter -o yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    hami.io/node-handshake: Requesting_2026-07-09 09:03:57
    hami.io/node-nvidia-register: '[{"id":"GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e","count":10,"devmem":6141,"devcore":100,"type":"NVIDIA GeForce RTX 4050 Laptop GPU","mode":"hami-core","health":true,"devicepairscore":{}}]'
    nvidia.com/gpu-driver-upgrade-enabled: "true"
  ...

When a pod is created, the API server calls HAMi's mutating webhook, which checks the pod's manifest against its supported device drivers, including NVIDIA, Ascend, AWS Neuron, and others. Each driver checks the container's resources.limits for recognized resource keys. For example, the NVIDIA driver looks for nvidia.com/gpu, nvidia.com/gpumem, and nvidia.com/gpucores, while the Cambricon MLU driver matches cambricon.com/vmlu. As soon as a compatible request is detected, HAMi updates the pod's schedulerName to hami-scheduler, ensuring the pod is handled by HAMi's device-aware scheduling logic rather than the default Kubernetes scheduler.

Note: Under HAMi, requesting nvidia.com/gpu: 1 no longer means "one whole card," but rather "one shared physical card," with the memory and core limits dictating the exact slice. However, a pod that omits nvidia.com/gpumem falls back to the configured default (100% of the card's memory).

kubectl logs -n hami-system deploy/hami-scheduler -c vgpu-scheduler-extender
I0710 13:51:38.552275  config.go:111] Initializing NVIDIA device
I0710 13:51:38.552355  config.go:248] All devices initialized successfully
...
I0710 14:02:14.161116  devices.go:647] "Resource requirements collected" pod="default/chat-qwen-64c4bd66f9-8qccs" requests=[{"NVIDIA":{"Nums":1,"Type":"NVIDIA","Memreq":4000,"MemPercentagereq":101,"Coresreq":60}}]

The vgpu-scheduler-extender builds a live usage map for every physical GPU in the cluster. It reads device metadata from each node's hami.io/node-nvidia-register annotation, then loops through the pods already assigned to that node to calculate the consumed GPU memory and compute capacity.

kubectl logs -n hami-system deploy/hami-scheduler -c vgpu-scheduler-extender
...
I0710 14:02:14.161182 node_policy.go:82] node node1 used 0, usedCore 0, usedMem 0,
I0710 14:02:14.161191 gpu_policy.go:104] device GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e user 0, userCore 0, userMem 0,
...
I0710 16:54:54.475656 node_policy.go:82] node node1 used 1, usedCore 60, usedMem 4000,
I0710 16:54:54.475667 gpu_policy.go:104] device GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e user 1, userCore 60, userMem 4000,

The scheduler then filters out any nodes lacking sufficient capacity and scores the remaining nodes based on the configured HAMi scheduling policy (nodeSchedulerPolicy and gpuSchedulerPolicy; the chart defaults are binpack at the node level and spread at the GPU level). The selected node and chosen GPU slice are recorded in the pod's annotations (hami.io/vgpu-devices-allocated) as a comma-separated list of UUID,Type,memMB,cores records.

$ kubectl describe pod -n default embed-bge-f84696cc9-7587j
Name:         embed-bge-f84696cc9-7587j
Labels:       hami.io/vgpu-node=node1
...
Annotations:  hami.io/vgpu-devices-allocated: GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e,NVIDIA,2000,30:;
              hami.io/vgpu-devices-to-allocate: ;;
              hami.io/bind-phase: success
              hami.io/bind-time: 1783702494
...

Once the pod is bound to a node, the Kubelet calls Allocate on that node's HAMi device-plugin. The plugin reads the pod's hami.io/vgpu-devices-to-allocate annotation (cleared once the plugin has consumed it, which is why the describe output above shows it empty) to decode the requested slice, identifying the target physical device and its exact compute and memory limits. It then returns a response detailing what the container needs to run against that specific slice: environment variables, host-path mounts (including the enforcement library), and device handles.

kubectl logs -n hami-system ds/hami-device-plugin -c device-plugin | grep -i 'Allocate Response'
...
I0710 16:54:54.499747  232069 server.go:719] Allocate Response [envs:{key:"CUDA_DEVICE_MEMORY_LIMIT_0" value:"2000m"} envs:{key:"CUDA_DEVICE_MEMORY_SHARED_CACHE" value:"/usr/local/vgpu/584ed76a-0856-40b3-8113-1ad696daa252.cache"} envs:{key:"CUDA_DEVICE_SM_LIMIT" value:"30"} envs:{key:"LIBCUDA_LOG_LEVEL" value:"1"} envs:{key:"NVIDIA_VISIBLE_DEVICES" value:"GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e"} mounts:{container_path:"/usr/local/vgpu/libvgpu.so" host_path:"/usr/local/vgpu/libvgpu.so.v2.9.0" read_only:true} mounts:{container_path:"/usr/local/vgpu" host_path:"/usr/local/vgpu/containers/1a5bac44-7888-4ea5-9382-3ef49ae1f1ba_tei"} mounts:{container_path:"/tmp/vgpulock" host_path:"/tmp/vgpulock"} mounts:{container_path:"/etc/ld.so.preload" host_path:"/usr/local/vgpu/ld.so.preload" read_only:true}]

The Kubelet applies this response directly to the container during creation. From that point, HAMi's preloaded control library sits between the application and the vendor driver, enforcing the slice in-process. It intercepts the driver's memory-allocation calls, checks each request against the slice's budget, and returns an OOM error if the workload attempts to exceed its cap. It also intercepts and rewrites driver management queries (like nvidia-smi), ensuring the reported total capacity reflects the assigned limit, and the free capacity reflects that limit minus current usage.

kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- env
CUDA_VERSION=13.0.2
NVIDIA_VISIBLE_DEVICES=GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e
CUDA_DEVICE_MEMORY_LIMIT_0=4000m
CUDA_DEVICE_SM_LIMIT=60
CUDA_DEVICE_MEMORY_SHARED_CACHE=/usr/local/vgpu/5e999bc3-2dbf-46b0-ba81-cc479a1c11dd.cache
...

kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- cat /etc/ld.so.preload
/usr/local/vgpu/libvgpu.so

kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- nvidia-smi --query-gpu=memory.total --format=csv,noheader
4000 MiB
[HAMI-core Msg(272:133498069993280:multiprocess_memory_limit.c:703)]: Cleanup on exit for PID 272
[HAMI-core Msg(272:133498069993280:multiprocess_memory_limit.c:739)]: Exit cleanup complete for PID 272

Scheduling policies that move the bill

As mentioned earlier, the scheduler first filters out nodes lacking sufficient capacity. For the remaining candidates, HAMi determines the optimal placement using two distinct scheduling tiers: the node level and the GPU level. Currently, HAMi supports two primary strategies for each: binpack (pack tasks as densely as possible) and spread (distribute tasks as widely as possible).

metadata:
  annotations:
    hami.io/node-scheduler-policy: "binpack"
    hami.io/gpu-scheduler-policy: "binpack"

Both policies come with distinct operational and financial trade-offs.

Node-Level Strategies

At the node level, the binpack policy packs workloads densely onto the fewest possible machines. This approach enables cluster autoscalers to quickly drain and spin down empty nodes, directly cutting cloud infrastructure bills. However, this density concentrates tenants, meaning a single node or driver failure has a much larger blast radius, and leads to increased contention with less bandwidth headroom.

Conversely, the spread policy distributes workloads evenly across all available nodes. While this successfully lowers per-node contention to provide headroom for unexpected traffic spikes and securely distributes fault domains, it keeps every node "warm." As a result, the autoscaler cannot scale down the cluster, leaving you paying for idle capacity.

GPU-Level Strategies

At the GPU level, the binpack policy completely fills one physical card before scheduling workloads on the next. This preserves entirely free GPUs for large models that require undivided resources and maximizes the number of small slices sharing a single card. The inherent trade-off is that it creates the densest possible cards, maximizing "noisy-neighbor" pressure and exposing a high number of tenants to single-card XID faults.

Alternatively, the spread policy balances workloads across all available cards. This minimizes compute and memory bandwidth contention, making it the superior choice for latency-sensitive inference workloads. The downside is that it fragments available memory across the cluster, meaning a subsequent large job requiring massive, continuous memory might fail to schedule even if the total free memory across the cluster is technically sufficient.

These are placement-time policies only: HAMi scores new pods as they arrive; it does not actively migrate running workloads. Financial savings therefore materialize as pods churn, meaning binpack pays off aggressively on highly elastic inference fleets, but much less so on long-lived, pinned pods.

HAMi and MIG: complementary layers

The fence and the wall operate at different layers, and you do not have to pick one per cluster: HAMi can manage both from a single pool. With dynamic MIG enabled, HAMi drives nvidia-mig-parted to carve and re-carve MIG geometry to fit incoming jobs, removing manual nvidia-smi mig work and pre-committed slice layouts. Pods keep requesting nvidia.com/gpu and nvidia.com/gpumem exactly as before; by default, the scheduler may satisfy that with either a hami-core slice or a MIG instance, and a workload that needs hard isolation can pin itself to the wall with the nvidia.com/vgpu-mode: "mig" annotation. The trade-off is granularity: MIG placements round up to the nearest allowed geometry (an 8 GB request on an A100-40GB lands on a 2g.10gb instance), where hami-core would have fenced off exactly what you asked for.

Operational reality

HAMi replaces your device plugin. The HAMi device plugin and the standard nvidia-device-plugin both want to own nvidia.com/gpu; running both on the same node causes them to fight over resource registration, leading to capacity gaps between real and virtualized counts, and GPU allocation is served by whichever plugin registered last, silently bypassing HAMi’s isolation. If you run the GPU Operator, keep it for the driver, container toolkit, DCGM, and disable its device plugin on HAMi-managed nodes. Scope HAMi with the gpu=on node label and roll it pool by pool, not cluster-wide on day one.
The scheduler image tracks your control plane. hami-scheduler wraps a stock kube-scheduler image, and current charts resolve its tag from the API server’s version at install time, which means a Kubernetes upgrade quietly leaves the old scheduler running until your next Helm upgrade. Add scheduler.kubeScheduler.image.tag to your Kubernetes upgrade runbook, or it will bite you six months in.
gpucores throttles; it does not reserve. The scheduler books your share at placement time, but at runtime that share is a ceiling, never a floor. The limiter is a token bucket checked at kernel launch, not a hardware partition: expect oscillation around the target, bursting above the share while the card is idle, and a few seconds of overlap before limits re-arm when a neighbor arrives mid-burst (set GPU_CORE_UTILIZATION_POLICY=force to pin the limiter on at all times). Benchmark p99 under a noisy neighbor before you commit to an SLO.

A practical recommendation

If your GPU fleet is A100/H100-class, shared by multiple teams, and your utilization dashboards would embarrass you in the finance review, pilot HAMi on one inference node pool this week. The install fits in an afternoon: label two nodes gpu=on, helm install, convert one over-provisioned inference Deployment to gpumem/gpucores slices, and let it run for a week while you watch allocation against DCGM utilization. The arithmetic you’re testing is blunt: an 8-GPU H100 node runs roughly $290,000 to $480,000 a year on-demand depending on cloud and region, so every node that bin-packing drains is real money back. On owned hardware, at roughly $25–30k per H100, every ten points of sustained utilization on a thousand-GPU fleet is $2.5–3M of capex doing work instead of nothing.

AWS Summit Seoul 2026: Korean Enterprises And Agentic AI

Ivan Porta — Thu, 21 May 2026 02:32:09 +0000

Over the last few years, people have been asking the same question about AI: with so much money going into models, GPUs, and data centers, when will it pay off? Earnings calls and analyst reports have often mentioned a possible bubble, and companies that tried generative-AI pilots in 2024 have been waiting to see real production results.

At AWS Summit Seoul 2026, Samsung Electronics, Yogiyo, and Yanolja presented production-oriented agentic-AI/AIOps cases, while KB Kookmin Bank presented a production-scale KBaaS API-infrastructure modernization case. Together, the sessions showed how Korean enterprises are applying AI-era cloud architecture to operations, development, and embedded finance. Here's a summary of how the event was organized, what the enterprise sessions showed, which AWS tools were most common, and the key patterns you can apply to your own plans.

The shift the day was framed around

AWS Korea CEO Ham Ki-ho opened the event by outlining three stages: Generative AI, Agentic AI, and Physical AI. He described Agentic AI as the active phase, where large language models act as reasoning engines that plan, use tools, and take action, instead of just answering prompts. Physical AI was presented as the next step, with a new Physical AI Frontier Program for Korean robotics, AI-chip, and manufacturing companies. This program will support everything from data collection to edge inference and help with global expansion.

AWS CFO Jon Felton announced a cumulative ₩12.6 trillion investment plan in Korea by 2031, described publicly as the largest-ever investment in Korea by a global cloud provider, with an estimated ₩15.06 trillion contribution to Korea's GDP between 2023 and 2027. The Industry Day talks that followed showed how this commitment will play out in practice.

The keynote also highlighted the main agentic tools AWS featured at the event, with Vice President Jason Bennett explaining each one:

Kiro: a new agentic IDE for software development, described on stage as a peer to Claude Code and Copilot-style coding assistants
Amazon Quick Suite: an agentic AI workspace for finding insights, conducting research, automating tasks, visualizing data, and taking actions across apps
AWS Transform and AWS Transform Custom: agentic modernization tools for mainframe, VMware, Windows, and legacy-code modernization, including version upgrades, runtime migrations, language translations, and architecture changes

What the enterprise talks actually showed

Across the agentic-AI sessions, a pattern emerged, but it was not universal. Samsung and Yanolja both used a supervisor or core-agent structure that delegates work to specialist agents. Yogiyo showed a Bedrock AgentCore-based AIOps workflow connected to operational data and tool functions. KBaaS was different: an EKS-based API platform and gateway modernization for embedded finance.

The numbers made it clear these were real deployments, not just experiments. Samsung Electronics used Samsung Account, which supports 2.1 billion users, over 50 services, more than 2.7 million requests per second, and 200,000 transactions per second across four regions on EKS with over 70 namespaces. Samsung set Day-1 AIOps goals including 90%+ MTTR reduction, 99% incident detection within 10 minutes, and reducing human-in-the-loop operational work below 20%, as part of a roadmap toward 'Toil 0% / Human First.' KB Kookmin Bank's KBaaS platform now handles about 1,800 APIs and 200 million calls per day after switching from a third-party API gateway to their own on AWS EKS. Yogiyo showed how they moved from checking multiple consoles and logs during incidents to using an AgentCore-based AIOps loop, which brings metric observation, correlation analysis, change-history comparison, and RCA grounding into one workflow. Yanolja shared the most technical details, describing how their domain-specialist agents for DevOps and SRE are managed by a Core Agent on Bedrock AgentCore Runtime, using Strands Agent as the SDK and end-to-end authentication.

Samsung, Yogiyo, and Yanolja all centered their agentic-AI cases on AIOps or infrastructure operations rather than customer-facing copilots. KB Kookmin Bank's case showed a different but related production pattern: modernizing a high-volume API platform on EKS for embedded finance.

On the development side, the counterpart is AI-DLC (AI-Driven Development Lifecycle), which is AWS's approach to AI-led development with humans checking at validation and final approval. Samsung saw a 70% reduction in lead time using this method, and LG Electronics' MS division reported double the productivity. The main idea is to separate intent from execution: humans decide what should happen and check the results, while agents handle the steps in between.

The AWS-native stack that recurred

A few key components showed up again and again in the enterprise sessions:

Amazon Bedrock AgentCore appeared across Samsung, Yogiyo, and Yanolja. Yanolja explicitly presented Bedrock AgentCore Runtime with Strands Agent and authentication. Yogiyo showed an AIOps architecture using Amazon Bedrock AgentCore with runtime, memory, gateway, and MCP tool functions. Samsung showed an AgentCore-based AIOps architecture with Supervisor, Domain, and Task agents.
Kiro was used as the agentic IDE, and Samsung included it in their toolset along with Amazon Quick Suite and Bedrock AgentCore.
Amazon EKS was the hosting platform for both Samsung Account and KBaaS, the two cases where the host was clearly specified.

The themes worth taking home

Several patterns stood out during the day. First, the supervisor pattern appeared twice independently. Samsung and Yanolja reaching the same structure without coordination is a strong signal. Second, in three of the four cases (Samsung, Yogiyo, Yanolja), the entry point was AIOps or infrastructure operations rather than customer-facing copilots. KB Kookmin Bank instead modernized its API platform on EKS for embedded finance. Third, humans are intentionally kept in the loop. Datadog's session made this clear, stating that sovereignty and judgment remain with people, while agents handle execution via the Datadog AI Agent Builder. Samsung's AI-DLC approach on the development side followed the same principle. Fourth, infrastructure needs to catch up. GS Neotek's session highlighted that the real GPU issue in agent workloads is not a shortage but poor utilization, excessive idle time, and over-allocation. They suggested using Dynamic Resource Allocation (DRA) on EKS as a next-generation operations model that considers workload needs, sharing policies, and topology.

Kubecost Explained: Kubernetes FinOps That Moves the Bill

Ivan Porta — Tue, 19 May 2026 01:38:37 +0000

Most platform teams are familiar with Kubernetes costs. The monthly cloud bill arrives, finance asks why it’s higher, and engineering can only respond with “more workloads.” This gap between what the bill shows and what platform teams can explain is exactly what FinOps aims to address to help optimize operational costs. The real question is whether you need an enterprise platform for this. For most teams, the answer is no. OpenCost and Kubecost can give platform teams the visibility they need, as long as the tool is paired with an operating cadence.

The pressure is real

Kubernetes accounting is no longer just about a single cluster or a single cloud. Most teams now manage fleets of clusters, often across multiple cloud providers, and sometimes combine on-premises control planes with managed services. Containers move between nodes, nodes move between zones, and the same workload might run in several regions to meet latency or compliance needs, making the cost attribution even harder.

Industry data has shown the same problem for years. Back in 2021, a CNCF FinOps survey found that most teams couldn’t reliably measure their Kubernetes spending, with over-provisioning and lack of accountability as the main issues. The same story happened in 2025, with a fleet-telemetry benchmark using real cluster data from over 2,100 organizations on AWS, GCP, and Azure, showing average CPU utilization at 10% and memory utilization at 23%. These numbers come from production telemetry, not just survey responses. The problem hasn’t changed in four years; if anything, it’s become clearer as fleets have grown.

The real issue isn’t that Kubernetes is too expensive. It’s that most teams don’t have enough visibility into what they’re spending.

What Kubecost actually is

Kubecost is a Kubernetes platform for cost allocation, optimization, and governance. After IBM acquired it in 2024, Kubecost now offers both open-source and enterprise versions. At its core, it sits on top of OpenCost, and runs as an in-cluster agent stack with several microservices: data collectors, a cloud-cost ingestor, a forecasting service, a per-node network-costs DaemonSet, and a fast ClickHouse-backed aggregator. By combining Kubernetes telemetry with cloud-provider billing data, Kubecost provides detailed cost allocation views, and accuracy improves when cloud billing integrations are configured and reconciled.

The platform is built around three primary pillars:

Cost allocation rolls spend up by namespace, label, service, workload, or Collection. Kubecost v3 adds a grouping concept that bundles Kubernetes and external cloud-side costs into a single deduplicated unit. Costs are tracked across CPU, memory, persistent volumes, GPUs, and network traffic.
Optimization recommendations suggest right-sized requests based on real usage, propose cheaper node types, and let teams configure quantile-based controls instead of accepting a one-size default. Recommendations can be archived for historical reference and exported as CSV or PDF.
Alerts and governance ship as configurable budget actions, scheduled reports, and Slack/email notifications. Alerts live next to the budget they belong to rather than as a separate alerting subsystem.

How a Kubecost install flows

Cost data collection begins inside your clusters. When it starts, the FinOps agent sets up a watch on the Kubernetes API for the pricing ConfigMap, so any custom pricing rules can be applied without a restart. It also resolves node pricing per node (falling back to a computed value when the cloud provider's price is unavailable). It then collects metrics from each Network Costs DaemonSet pod, creates a new binary snapshot, and, if configured, writes it to external storage such as Azure Blob Storage or AWS S3.

The Network Costs DaemonSet subscribes to the kernel's conntrack table via a netlink socket, parsing each flow per-direction byte and packet counters, and maintains an in-memory map of pod, node, service, and endpoint state via Kubernetes API watches. It uses this map to link observed connections to specific workloads.

While these agents send internal snapshots to shared storage, the Cloud Cost Ingestor manages external financial data. It runs on a schedule, connects to cloud provider billing exports, pulls daily CSVs, and backfills historical data. Because cloud providers release billing data with a several-hour delay, Kubecost reconciles cluster data with cloud billing after a short wait. This means the most recent day or two of cost data is only an estimate, while older data is fully reconciled (assuming a working cloud-billing integration).

The Aggregator is the main engine and uses an embedded ClickHouse database. In a multi-cluster deployment, it fans in snapshots from several agent clusters. It checks multiple ConfigMaps for configuration and falls back to defaults when none are present. It ingests agent snapshots and, when configured, external billing CSVs, then drives them through a multi-stage SQL pipeline that reconciles and de-duplicates overlapping costs and produces the final cost tables that other microservices consume. The Aggregator also manages data retention by setting per-table, per-resolution TTLs in ClickHouse, so fine-grained windows expire within days while rollups are kept for weeks or months.

Finally, the Forecasting Service serves as a predictive cost-monitoring tool, using this data to generate cost forecasts. At the same time, the Cluster Controller uses the Aggregator’s optimization insights to take actions, such as applying right-sizing recommendations directly in the cluster.

Allocate to a real owner first

Kubecost spreads node costs across the pods running on each node, typically weighted by resource requests, and rolls the result up by namespace, label, service, workload, or Collection, covering CPU, memory, PV, GPU, and network.

What makes this approach effective is good organization, not just technical setup. Each namespace should match a real owner, like a team, product, or department. When this mapping is in place, the allocation view shows which team is responsible for each cost, without needing a spreadsheet. Teams that already use labels like team, cost-center, or product can use these for the same purpose, and Collections help make label-based views easy to use.

Labels can change over time, namespaces can increase, and someone will eventually deploy into default. Reviewing the unlabeled bucket each week helps keep the data accurate and useful.

Right-size requests against actual usage

This is the lever that moves the bill the most. Often, developers set CPU and memory requests defensively, never revisit them, and the gap between request and use shows up directly on the invoice. The 10%-CPU / 23%-memory benchmark cited above is a useful authority anchor when finance asks for a number.

A pattern that reliably finds savings: plot requested vs. actual CPU and memory per workload over a few weeks, then walk each workload's request down to what it actually uses. One practical case from a service mesh deployment had proxy sidecars set to 100 millicores each. The node could host roughly 200 pods on paper, but the scheduler exhausted allocatable CPU at around 90 pods because every pod carried a 100 millicores sidecar request on top of its own. After the request-rightsizing pass, pod density per node tripled with no application change and no node fleet change.

Kubecost 3.0 makes this loop tighter. Container Request Sizing Insights show usage visualizations directly in the UI, recommendations can be archived, CSV/PDF exports include labels, and quantile-based controls let you set tighter recommendation percentiles for predictable services and looser ones for bursty workloads. The enterprise tier adds an Automated Container Request Sizing UI that operates across clusters with custom profiles, suspension controls, audit history, and a comparison between recommended and realized savings; the open-source tier gets a free allowance up to 250 cores on EKS primary clusters.

Using a percentile-based recommendation policy is usually the most effective approach in production.

Set the CPU request to the 90th percentile of actual CPU usage from the past week or month, and then add a safety margin. The Kubernetes VPA default is about 15% for CPU. Because CPU is time-shared, the kernel lets bursty workloads use extra capacity when it is available. Adding more padding for rare spikes usually just increases requests without much benefit.
Set the memory request to a high percentile of peak usage, and then add a safety margin. Memory is not time-shared, so going over the limit can cause OOM kills instead of graceful degradation. Aim for about the 90th percentile of peaks, then add a margin. The VPA default is about 20% for memory.

These settings decide the QoS class. A pod is considered Guaranteed only if every container has CPU and memory requests set equal to their limits. If this is not the case for any container, the whole pod is treated as Burstable, or as BestEffort if no container sets any requests or limits. This setup works well for most workloads. Reserve the fully Guaranteed class for critical workloads with strict latency SLAs, where CPU and memory requests match their limits. In those cases, you might waste some headroom, but you get the best eviction protection and, if needed, exclusive CPU pinning.

If you want an additional feedback before automating any of this, you can run VPA in recommendation-only mode for several weeks, or use KRR open-source. Comparing recommendations across KRR, VPA-recommendation-mode, and Kubecost is more reliable than trusting any single tool's number.

Capacity-versus-request as the North Star metric

The ratio that tells you most about cluster efficiency is total pod requests ÷ total node allocatable capacity, across CPU and memory: how much of what you pay for is even claimed by a pod. It is also what Kubecost's request right-sizing is built on. Kubecost ships with several targets against which recommendations are computed: Production 0.65, Development 0.80, High Availability 0.50 (Cluster Right-Sizing API). Below it you carry capacity nothing asks for; above it you've spent the headroom that cluster class should keep. Kubecost also picks the utilization it sizes against by context; development the trending 85th-percentile, production the 98th, HA the 99.9th; and only on a one-day window; longer windows use maximum usage. The often-quoted "85th percentile" is just the development one-day default, not a universal setting.

The ratio is what you watch; an autoscaler (Karpenter, Cluster Autoscaler) moves it — but only if requests are honest, which is why Kubecost's request right-sizing sits upstream of any autoscaling story. The autoscaler reacts to requests; Kubecost tells you whether they reflect reality.

Recent days are directional by design: reconciliation needs a full day of billing data, so for a roughly 48-hour window costs stay at public on-demand pricing unless a node is provably not on-demand; Spot is accurate sooner only through a separately configured AWS Spot data feed (Cloud Billing Integrations). Read efficiency — independent of reconciled pricing — separately from cost.
Node Group Sizing — formerly Cluster Right-Sizing, rebuilt in v3.0 — turns this into an action: it analyzes in-cluster CPU, RAM, and GPU utilization against node capacity over a configurable window and recommends, per node group, changing the node count or switching the instance type. It runs from a preset profile or a custom metric — usage.max/p95/p85/avg or request.max/avg — with a target-utilization threshold per resource, never below average requested resources. It detects node groups by each provider's standard label, so it works across EKS, AKS, and GKE without setup (v3.x docs).

Find the always-on workloads that don't need to be

Once you’ve handled allocation and right-sizing, look at workloads that run all day, every day, even when they don’t have to. In one platform team’s review, 31% of workloads used less than 25% CPU for almost the entire day, yet Kubernetes costs still went up by about 18% over the year. This happened because engineers spent a lot of time tuning capacity and dealing with alerts, and because each team set up its own autoscaling rules differently.

The triage falls into three buckets. Production services that are genuinely over-spec’d belong in the right-sizing loop above. Non-production environments — dev, integration, demo — rarely need to run on weekends or overnight; a scheduled scale-to-zero is the highest-ROI change in this category. Batch and stateless workloads with retry tolerance are candidates for Spot instances, which trade roughly a 90% discount for a two-minute interruption notice.

Kubecost helps you find underused workloads. With Kubecost 3.0’s Advanced Filters, you can quickly sort workloads by namespace, label, or service using AND/OR conditions right in the UI, instead of having to do it elsewhere.

Track commitment coverage and utilization separately

Reserved capacity and savings commitments are common sources of unnecessary cloud costs. Teams often either ignore them and pay full on-demand prices, or buy them and forget to check if they are being used, leaving discounts unused on resources that are no longer needed. There are two important metrics to watch, and they are easy to mix up:

Coverage means the portion of your regular usage that is protected by a commitment.
Utilization is how much of your commitment you actually use.

Each cloud provider has different tools, but they all fit into three main types, and the calculations work the same way everywhere:

Mechanism	Typical max discount vs on-demand	Commitment
Flexible spend commitment	~60–66%	1 or 3 yr; hourly $ commitment; applies across families/regions
Instance-specific reservation	~55–72%	1 or 3 yr; locked to a region + instance family/SKU
Spot / preemptible	up to ~90%	none; interruption notice from ~30 sec to ~2 min

A good approach is to aim for commitment utilization between 80% and 95%, instead of trying to reach 100%. Going for 100% leaves no room for normal changes, like removing unused instances, changing instance types, or handling a drop in traffic. It may look efficient in a quarterly review, but it can cause problems day-to-day. For coverage, aiming for 60% to 75% is reasonable. This range is high enough to get a good discount, but low enough to allow for changes each quarter. These ranges are based on practical experience, not rules set by the cloud provider.

With Kubecost, costs are first estimated using public on-demand cloud provider prices until the actual cloud bill is ready. When the bill becomes available, usually within about 48 hours, Kubecost updates its estimates with the real costs. This update includes Reserved Instances, Savings Plans, committed-use discounts, and Spot pricing, along with any special rates you might have, such as Enterprise Discount Programs.

When to use a commercial FinOps platform instead

Most teams should start with the open-source chart. You can look at the commercial tiers later, once your needs grow.

Capability / Feature	Open-source Kubecost	Commercial FinOps platform
Cost allocation (namespace, label, workload)	✓	✓
Optimization recommendations (right-sizing)	✓ (manual application)	✓ + automated application across clusters
Cloud-billing reconciliation	✓ (basic)	✓ + EDP / RI / custom-discount aware
Multi-cluster aggregation	Manual / federation	✓ (built-in)
SSO, RBAC, audit log	Limited	✓
History retention	Limited by your storage layer	Long-term, vendor-managed
Collections (cloud + K8s dedup)	✓ (3.x)	✓
Automated Container Request Sizing UI	✕ (free tier limited)	✓
Quantile-based recommendation controls	✓ (3.x)	✓
Advanced filters (AND/OR)	✓ (3.x)	✓
Support / SLA	Community	Vendor SLA

If you just need per-namespace allocation, basic recommendations, and Slack alerts for a few clusters, the open-source version is enough. But if you manage many clusters across different clouds, need automated fixes, want vendor-managed history, or need to give your finance team detailed, reconciled discount numbers, then a commercial platform is worth it. The decision should be based on these needs, not just on how the dashboard looks.

Operational reality

ClickHouse and a unified agent replace the old stack. In v3, the 2.x DuckDB store is replaced with a ClickHouse database. This change makes allocation and cloud-cost API queries much faster and more reliable at scale. It also removes the need for Prometheus, which cuts down on memory use and makes deployment easier, while still providing OpenCost-standard metrics.
History is a deliberate choice, not a default. Whatever the storage backend, the retention window is the upper bound on the period-over-period reporting you can produce. Monthly reporting requires at least 30 days; year-over-year requires a year. Tier cold data to object storage if on-cluster retention gets expensive faster than the engineering time to set up the tiered pipeline.
Reconciliation lag is structural. The 24–48 hour billing-reconciliation delay is a property of cloud-provider billing exports, not of Kubecost. Build the operating model around it: argue about last week, not yesterday.
Multi-cluster needs a story. Open-source Kubecost can federate across clusters, but the experience is rougher than the commercial multi-cluster aggregator. Beyond five or six clusters, decide early whether to run per-cluster Kubecost and aggregate externally — into your own warehouse, for example — or pay for the commercial multi-cluster path. Either is defensible; drifting between the two is not.
The EKS add-on offers a quick way to get started. The Kubecost v3 free tier has a $100k USD spend limit over 30 days, while the Amazon EKS optimized Kubecost bundle is listed by AWS as exempt from that spend limit.
The operating model is the deliverable. If a team installs Kubecost but does not set up regular reviews, they will drift just like a team without any FinOps tools. The standard approach is to have a small FinOps group, such as a platform engineer, a finance analyst, and an SRE on rotation, meet each week to review the capacity-versus-request ratio, identify the most over-provisioned workloads, and check any namespace with a significant change in monthly cost. For smaller teams, a 30-minute review every two weeks with the platform engineer and CTO can achieve the same results.

A practical recommendation

If you are considering a FinOps approach for a Kubernetes platform and do not have a contractual obligation to choose a commercial product, start by piloting open-source Kubecost 3.x. Installation can be completed in an afternoon. Assign at least one namespace to a designated owner, provide a request-versus-usage dashboard to one team for two weeks, and share the capacity-versus-request ratio in a channel visible to the platform team. If regular reviews of these metrics become routine, you have achieved FinOps. If not, adopting a commercial platform will not resolve the underlying issues.

What Platform Teams Can Expect From Crossplane v2.2

Ivan Porta — Tue, 05 May 2026 05:21:20 +0000

A developer submits a ticket to request a database. Three days later, the platform team responds, but the configuration isn’t quite right. The developer files another ticket and waits again. By the third week, the database might finally be ready. This cycle repeats across every team and environment, creating a daily reality that most platform teams recognize: developers lose significant time waiting for infrastructure, while platform teams struggle to keep up with the constant flow of requests.

There are plenty of good tools for provisioning. Terraform is platform-agnostic and widely used. CloudFormation is the go-to option on AWS, and every cloud provider offers its own console and CLI. Each tool does its job well. This article isn’t about choosing the best one. Instead, it looks at the problem from a different perspective.

Crossplane offers a Kubernetes-native approach to managing cloud resources. Instead of setting up infrastructure outside the cluster and then linking it back, Crossplane brings that infrastructure under the same control loop as your applications. This fits naturally with how many teams already work. Paired with GitOps, a pull request becomes the primary way to manage changes, and the cluster continuously reconciles toward the desired state.

The project has moved quickly over the past year. In August 2025, Crossplane v2 introduced big changes, like removing Claims and adding namespaced composite and managed resources. The latest release, v2.2, adds an alpha Pipeline Inspector for troubleshooting, broader CEL validation, and more improvements.

What Crossplane actually is

Crossplane is a control plane framework for platform engineering. You install it into a Kubernetes cluster, known as the management cluster, and that cluster becomes the control plane for everything outside it: cloud accounts, SaaS APIs, internal tools, and even other Kubernetes clusters. All of these are managed through the same Kubernetes API your applications already use. The management cluster itself must be set up separately; Crossplane does not create it for you. Once Crossplane is running, there is no state file and no separate workflow. Drift is fixed by the same reconciliation loop that keeps your Deployments healthy.

Crossplane has four major components. You can use all four or only the ones you need.

Managed resources (MRs) map directly to external cloud resources in Kubernetes. For example, an S3 from AWS or a ResourceGroup from Azure is considered an MR. Crossplane uses spec.forProvider as the main reference and keeps the actual cloud resource in sync with it. You create MRs using kubectl, and the provider handles provisioning and reconciliation.
Composition lets you create custom APIs using a function pipeline. There are three main parts to understand:
- A CompositeResourceDefinition (XRD) defines a schema. It tells Kubernetes, “here’s a new custom API kind I’m creating, and these are its fields.” You can think of it as a CRD with added features for Crossplane.
- A Composition acts as a recipe. It says, “when someone creates an XR of kind Foo, run this set of functions to create these MRs or other Kubernetes resources.” In version 2, this always uses a function pipeline.
- A Composite Resource (XR) is an instance of the API you defined with an XRD. When a user creates an XR, Crossplane uses the matching Composition’s pipeline to generate the needed resources. You can write functions in YAML, KCL, Python, or Go.
Operations run function pipelines to completion, similar to a Kubernetes Job. There are three modes: Operation (one-time), CronOperation (scheduled), and WatchOperation (event-driven). Operations are currently in alpha.
The package manager handles installing and updating providers, configurations, and functions.

How a Crossplane request flows

There are two entry points into this flow, depending on what you're applying.

When a developer or a pipeline creates an XR in a namespace, the composition engine watches it, runs the configured function pipeline, and creates the needed resources. These resources can be other Kubernetes resources, managed resources, or both.

When a user applies an MR directly, either by itself or as part of a Composition, the provider takes over. It monitors the MR through the Kubernetes API, calls the external system to create or update the real resource, and updates the status. After that, it keeps checking: if the real resource changes from spec.forProvider, the provider fixes it. All state is stored in etcd, so there is no separate state file.

When to use traditional IaC instead

Crossplane and tools like Terraform or CloudFormation overlap in scope (both can provision a cloud database) and differ in how. The right choice depends on where your platform already lives.

Capability / Feature	Terraform	CloudFormation	Crossplane
Control loop	Manual `apply` (or pipeline)	Manual stack create/update	Continuous reconciliation
Drift handling	Detect with `plan`; correct manually	Detect drift action; correct via stack update	Detected and corrected automatically
State	`tfstate` in a remote backend you secure (e.g., S3 with versioning, HCP Terraform)	AWS-managed (server-side)	Kubernetes API objects in the management cluster's etcd
Workflow	Separate from app deployment	Separate from app deployment	Same as `kubectl apply`
Composition	Modules	Nested stacks, Modules	XRDs + Compositions + functions
Languages	HCL, JSON	YAML, JSON	YAML, Go, Python, KCL, CUE, HCL (via composition functions)
Built-in policy	Variable validation and pre/postconditions (OSS); Sentinel and OPA integration in HCP Terraform / Enterprise	cfn-guard, Hooks	XRD CEL validations (incl. metadata in v2.2)
Multi-cloud	Provider per cloud, separate state	AWS-first (third-party types via the CloudFormation registry)	One control plane, one API surface
Footprint	A binary	AWS-managed service (CLI/SDK only)	A Kubernetes control plane (Crossplane core, providers, functions) backed by etcd
Operates outside Kubernetes	✓	✓	✕ — requires a management cluster

If your team does not use Kubernetes, Crossplane is not the best place to start. Terraform is simpler and does not need a control plane. But if you are on Kubernetes, especially if you already use Argo CD or Flux, it is easy to manage your infrastructure in the same way. Crossplane is the closest option for writing infrastructure as code and handling it like the rest of your declarative cluster state.

What new with v2.2

v2.2 adds five things you'll notice in practice and one that quietly improves reliability. Each one closes a specific gap that platform teams have been hitting in production.

Pipeline inspector (alpha): Composition functions are powerful, but they have always been hard to debug. If a pipeline acts up on a running control plane, you could only see what each function got and returned by writing tests, running crossplane render locally, or adding your own instrumentation. v2.2 adds the pipeline inspector. When you turn on the feature flag, the Crossplane controller intercepts every RunFunctionRequest and RunFunctionResponse and forwards them over gRPC to a Unix socket you set up. A sidecar read from this socket and handle the data however you need: stream it to stdout during development or send it to an audit pipeline in production. To use it, add --enable-pipeline-inspector to Crossplane. The default socket path is /var/run/pipeline-inspector/socket, but you can change it with --pipeline-inspector-socket.

  # Enable the pipeline inspector feature flag
  args:
    - --enable-pipeline-inspector
    - --pipeline-inspector-socket=/var/run/pipeline-inspector/socket

  # Inject the pipeline inspector sidecar
  sidecarsCrossplane:
    - name: pipeline-inspector
      image: xpkg.crossplane.io/crossplane/inspector-sidecar:v0.0.3
      args:
        - --socket-path=/var/run/pipeline-inspector/socket
        - --max-recv-msg-size=8388608  # 8MB
      volumeMounts:
        - name: pipeline-inspector-socket
          mountPath: /var/run/pipeline-inspector
      resources:
        requests: { cpu: 10m, memory: 64Mi }
        limits:   { cpu: 100m, memory: 128Mi }

  # Add the shared volume for Unix socket communication
  extraVolumesCrossplane:
    - name: pipeline-inspector-socket
      emptyDir: {}

  extraVolumeMountsCrossplane:
    - name: pipeline-inspector-socket
      mountPath: /var/run/pipeline-inspector

XRD validation outside spec: XRD validation outside x-kubernetes-validations, (which are Kubernetes' CEL-based validation rules) used to only work on fields under an XR's spec. If you wanted to enforce rules like "all Database names must start with db-", you had to use an external admission controller such as Kyverno, OPA/Gatekeeper, or a custom webhook. With v2.2, that restriction is gone. Now, you can write CEL rules outside of spec, and the API server enforces them at admission time.

  apiVersion: apiextensions.crossplane.io/v1
  kind: CompositeResourceDefinition
  metadata:
    name: databases.platform.example.org
  spec:
    group: platform.example.org
    names:
      kind: Database
      plural: databases
    versions:
      - name: v1alpha1
        served: true
        referenceable: true
        schema:
          openAPIV3Schema:
            type: object
            x-kubernetes-validations:
              - rule: "self.metadata.name.startsWith('db-')"
                message: "Database names must start with 'db-'"
            properties:
              spec:
                type: object
                properties:
                  region:
                    type: string

ImageConfig runtime for dependencies: A Crossplane package, including Providers, runs as a Deployment. To customize the Deployment, such as by adding service account annotations, pod labels, or container arguments, use a DeploymentRuntimeConfig and reference it from the package.

kind: Provider
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-azure-network:v1.0.0
  runtimeConfigRef:
    name: azure-workload-identity

This approach works well when you install the package directly. However, Crossplane can also install packages as dependencies. In that case, you could not get Workload Identity or any other runtime customization onto providers installed as dependencies.

ImageConfig is a cluster-scoped resource that matches packages based on their image prefix, not on which Provider or Configuration object created them. In v2.2, a new field was added: spec.runtime.configRef. With this change, Crossplane applies the DeploymentRuntimeConfig to any package whose image is matched, regardless of how it was installed.

  apiVersion: pkg.crossplane.io/v1beta1
  kind: ImageConfig
  metadata:
    name: azure-workload-identity
  spec:
    matchImages:
      - prefix: xpkg.crossplane.io/crossplane-contrib/provider-azure-
      - prefix: xpkg.crossplane.io/crossplane-contrib/provider-family-azure
    runtime:
      configRef:
        name: azure-workload-identity

Every Azure family provider, whether installed directly or added as a dependency, receives the runtime config.

RequiredSchemas for functions: Composition and functions sometimes need the OpenAPI schema of a resource to validate inputs, make schema-aware decisions, or generate resources dynamically. Before v2.2 you could ask Crossplane for the corresponding CRD as a RequiredResource, parse it, and extract the schema yourself, but only for custom resources, since built-in kinds like Deployment don't have a CRD. v2.2 introduces RequiredSchemas on the RunFunctionResponse thich returns the schema for any kind, built-in or custom.
crossplane beta trace improvements: You can now pass a kind (and optionally a namespace) instead of a single resource and get the dependency tree for every instance. And --watch (alias -w) keeps the output live, the way kubectl get -w does.
Function packages no longer install bundled CRDs: CRDs included in a function package are not applied to the cluster anymore. Also, packages with unknown or disallowed kinds now install successfully and simply skip those objects. Previously, the install would fail in these cases.
The package cache layout has changed: Cache filenames now come from the package’s OCI source and digest instead of the PackageRevision’s Kubernetes name. This change affects side-loading used by some provider e2e suites.

Operational reality

The management cluster is your state. Crossplane does not use an external state file. All XRDs, Compositions, XRs, and managed resources are stored in the management cluster’s etcd. If you lose that cluster without backups, your cloud resources keep running, but Crossplane loses track of them and stops reconciling. Silent drift can build up. Treat the management cluster like any production-critical Kubernetes cluster: use a highly available control plane, back up etcd, and avoid running it on your laptop. Local k3s or kind clusters are fine for learning, demos, or the Get Started guide, but not for important state. This is the trade-off for not having a Terraform state file: you solve one operational problem but gain another that is easier to overlook.
Upgrade through v2.1, not directly. Crossplane performs CRD migrations with each minor version upgrade, so skipping versions can cause you to miss important migrations. If you are on v1.x, use the Crossplane v2 upgrade guide. If you are on v2.1, upgrade directly to v2.2.
v1.20 is not yet end-of-life. v1.20 is still supported and has not reached end-of-life yet. However, since you are on a maintenance-only branch, it’s a good time to start planning your upgrade to v2.x.
Pipeline inspector is alpha. The flag is off by default and the contract may still change. Sidecar image versioning is also not stable yet. Try it in development, since function pipelines are much easier to understand when you can see them, but do not add it to your incident-response runbook yet.
Namespaced MRs are not yet universal. AWS managed resources are fully namespaced. The Upbound Azure provider, which is widely used, and GCP are currently rolling out this feature.
v2 removed several things. Native patch-and-transform composition, the ControllerConfig type, external secret stores, composite resource connection details, and the default registry for packages are no longer available. Most users can upgrade without breaking changes, but if you use these features, you will need to do some cleanup. Before upgrading, run kubectl get pkg and make sure every package uses a fully qualified image, such as registry.example.com/repo/package:tag.

A practical recommendation

If you are considering a control plane for your Kubernetes platform and do not have a strong reason to stick with Terraform, try Crossplane v2.2 first. The Get Started guide can be completed in an afternoon on any Kubernetes cluster. If Crossplane meets your needs, you can manage both application and infrastructure workflows with one declarative model. If not, you will have a clear, documented reason to keep your current tools.

If you are already using Crossplane v2.1, upgrade to v2.2. Features like server-side apply on the MRD controller, dependency-aware runtime config, schema access for functions, and better trace output are valuable even if you do not use the pipeline inspector. If you are still on v1.x, pin to v1.20, migrate any deprecated features, then upgrade to v2.x and continue from there. v2 offers good backward compatibility, but the deprecations are real.