DEV Community: Matthias Bruns

Kubernetes 1.36 Workload-Aware Scheduling: Gang Scheduling and Resource Optimization for AI/ML Workloads

Matthias Bruns — Thu, 04 Jun 2026 08:38:02 +0000

Kubernetes 1.36 marks a significant leap forward in workload-aware scheduling, building on the foundation laid in version 1.35. The new capabilities address critical gaps in how Kubernetes handles complex AI/ML workloads, distributed training jobs, and batch processing scenarios that require coordinated resource allocation. This isn't just another incremental update—it's a fundamental shift toward treating groups of related pods as first-class scheduling entities.

The traditional Kubernetes scheduler operates on individual pods, which creates problems for workloads that need multiple pods to start together or not at all. Gang scheduling solves this by ensuring that either all pods in a group get scheduled simultaneously, or none do. Combined with workload-aware preemption and opportunistic batching, these features transform how Kubernetes handles resource-intensive workloads.

Understanding Workload-Aware Scheduling

Kubernetes v1.35 introduced the foundational Workload API alongside basic gang scheduling support, but v1.36 takes this much further. The core concept revolves around treating related pods as a single scheduling unit rather than independent entities.

Traditional scheduling fails spectacularly with distributed workloads. Consider a 4-node distributed training job where each node needs 8GB of memory. Without workload-aware scheduling, the scheduler may place 3-of-4 ranks of training job A and leave the 4th pending forever because no node has capacity. The entire job becomes deadlocked, consuming resources but producing no useful work.

Workload-aware scheduling addresses this through three key mechanisms:

Gang scheduling ensures all-or-nothing pod admission
Workload-aware preemption treats pod groups as single entities for eviction decisions
Opportunistic batching efficiently processes identical pods together

Gang Scheduling: All-or-Nothing Resource Allocation

Gang scheduling implements the fundamental principle that certain workloads only make sense when all components can run simultaneously. The minCount field defines the quorum: at least that many pods must be schedulable together for the group to be admitted.

This is particularly crucial for AI/ML workloads where distributed training requires all worker nodes to be available. A partially scheduled job not only wastes resources but can also prevent other workloads from being scheduled due to resource fragmentation.

The gang scheduling implementation in Kubernetes 1.36 goes beyond simple all-or-nothing logic. It lets controllers, status reporting, future preemption behavior, and future workload-aware features reason about related pods even if those pods do not need strict all-or-nothing admission. This flexibility allows for more nuanced scheduling policies where some pods in a group might be optional while others are mandatory.

Workload-Aware Preemption

Traditional Kubernetes preemption operates at the pod level, which can create chaos for multi-pod workloads. KEP-5710 brings in "workload-aware preemption," meaning that groups of related Pods (PodGroups) are now treated as a single entity for both scheduling and preemption.

This change prevents scenarios where the scheduler evicts some but not all pods from a distributed workload, leaving the remaining pods in a useless state. Instead of removing pods one by one, the scheduler now understands the relationships between pods and makes preemption decisions at the workload level.

For AI/ML workloads, this is transformative. When cluster resources become scarce, the scheduler can now intelligently choose between evicting an entire lower-priority distributed training job versus partially disrupting multiple jobs. This leads to better resource utilization and fewer failed training runs.

Resource Optimization Patterns

Kubernetes 1.36 introduces several patterns for optimizing resource allocation in complex workloads. The most significant is the ability to express resource requirements at the pod level rather than just the container level.

Pod.spec.resources field accepts cpu, memory, and hugepages-* only — extended resources stay container-scope. This allows you to define the resource envelope once for multi-container pods instead of repeating specifications across containers. Container-level fields override pod-level resources when set, providing flexibility for mixed workload patterns.

This is particularly useful for AI/ML workloads that often combine multiple containers: a training container, a data preprocessing sidecar, and monitoring agents. Instead of calculating and specifying resources for each container individually, you can define the total pod requirements and let Kubernetes handle the distribution.

Opportunistic Batching for Identical Workloads

Opportunistic batching efficiently processes identical Pods by recognizing when multiple pods have identical resource requirements and scheduling characteristics. This feature is particularly valuable for batch processing workloads where you might have hundreds of identical data processing jobs.

The scheduler can now group these identical pods and make scheduling decisions for the entire batch rather than evaluating each pod separately. This dramatically reduces scheduling latency for large-scale batch workloads and improves cluster efficiency by considering resource allocation patterns across similar workloads.

For machine learning inference workloads, this means faster deployment of model serving pods and more efficient resource packing. The scheduler understands that these pods are functionally identical and can optimize their placement accordingly.

Practical Implementation Strategies

When implementing workload-aware scheduling for AI/ML workloads, start by identifying which of your workloads truly require coordinated scheduling. Not every multi-pod application needs gang scheduling—web applications with multiple replicas, for example, typically benefit from gradual rollouts rather than all-or-nothing deployment.

Distributed training jobs are the obvious candidates, but consider other scenarios:

Multi-node inference pipelines where all stages must be available
Data processing workflows with strict dependency requirements
Batch jobs that require specific node configurations across multiple pods

For resource optimization, leverage the new pod-level resource specifications when you have multi-container pods with shared resource pools. This is common in AI/ML workloads where containers share GPU memory or large datasets mounted as volumes.

Configuration Best Practices

Configure workload-aware scheduling features gradually. Start with non-critical workloads to understand the behavior and impact on your cluster. The scheduling changes can affect cluster resource utilization patterns, so monitor carefully during initial deployment.

For gang scheduling, set realistic minCount values that reflect the actual requirements of your workloads. Setting the value too high prevents scheduling when some nodes are temporarily unavailable. Setting it too low defeats the purpose of coordinated scheduling.

When using workload-aware preemption, establish clear priority classes for different types of workloads. Interactive workloads might have higher priority than batch processing, but long-running training jobs might need protection from frequent preemption once they've started.

Monitoring and Troubleshooting

Workload-aware scheduling introduces new failure modes that require updated monitoring strategies. Traditional pod scheduling metrics don't capture the complexity of group scheduling decisions. Monitor for scenarios where pod groups are partially scheduled but waiting for additional resources.

Pay attention to resource fragmentation patterns. Gang scheduling can sometimes lead to less efficient resource packing if not properly configured. Monitor cluster utilization to ensure that the coordination benefits outweigh any packing inefficiencies.

For AI/ML workloads specifically, track metrics around training job success rates and time-to-start. These workloads often have strict SLA requirements, and the new scheduling features should improve both metrics significantly.

Future Considerations

The workload-aware scheduling features in Kubernetes 1.36 represent the beginning of a larger transformation in how Kubernetes handles complex workloads. Future versions will likely expand these capabilities with more sophisticated resource coordination and cross-cluster scheduling awareness.

Consider how these features fit into your broader AI/ML infrastructure strategy. The improved scheduling capabilities enable more efficient use of expensive GPU resources and can reduce the need for dedicated training clusters in some scenarios.

As these features mature, expect to see ecosystem tools that leverage the new APIs for more intelligent workload placement and resource optimization. The foundation laid in 1.36 opens up possibilities for application-aware scheduling that goes far beyond what's possible with traditional pod-level scheduling.

The workload-aware scheduling improvements in Kubernetes 1.36 address real pain points in running complex, resource-intensive workloads. For organizations running AI/ML workloads at scale, these features can significantly improve resource utilization and reduce the operational complexity of managing distributed training and inference workloads.

Building Production-Ready AI Gateways: Custom Transformations with Rust

Matthias Bruns — Mon, 01 Jun 2026 09:03:49 +0000

AI workloads have exposed the fundamental limitations of traditional API gateways. While REST APIs follow predictable patterns, AI applications deal with streaming responses, variable latency, complex authentication flows, and business logic that changes faster than infrastructure teams can keep up. The result? Most organizations end up with a patchwork of direct integrations, each with its own security model, rate limiting, and monitoring—exactly the kind of sprawl that gateways were supposed to prevent.

The answer isn't another configuration-heavy proxy. It's building AI gateways that can be extended with custom business logic while maintaining the performance and security requirements of production systems. Rust has emerged as the language of choice for this challenge, offering the performance characteristics needed for low-latency proxying and the safety guarantees required for production infrastructure.

Why Traditional Gateways Fall Short for AI Workloads

Traditional API gateways were designed for synchronous request-response patterns with predictable payloads. AI workloads break these assumptions in several ways:

Streaming responses that can last minutes, not milliseconds. Your gateway needs to handle WebSocket connections, server-sent events, and chunked transfer encoding without buffering entire responses in memory.

Variable latency that makes traditional timeout configurations meaningless. A code generation request might take 30 seconds during peak hours but complete in 2 seconds at night.

Complex authentication flows that go beyond simple API keys. AI agents need to authenticate on behalf of users, maintain session state, and handle provider-specific auth patterns.

Business logic that changes weekly, not quarterly. You need to transform requests based on user context, implement custom rate limiting per model, or route traffic based on real-time cost optimization—logic that's impossible to express in YAML configurations.

As Solo.io points out, this is why they built Agent Gateway as an AI-native solution that combines "deep MCP and A2A protocol awareness, robust traffic policy controls, inference gateway support" rather than trying to retrofit existing gateway technology.

The Rust Advantage for AI Gateways

Rust's combination of performance and safety makes it ideal for building extensible AI gateways. Unlike interpreted languages that add latency overhead, or systems languages that trade safety for speed, Rust delivers both.

The numbers speak for themselves. AISIX, built with Rust, achieves "sub-millisecond proxy overhead" while maintaining memory safety. When you're proxying millions of AI requests per day, every millisecond matters—both for user experience and infrastructure costs.

But performance is only part of the story. The real advantage is Rust's approach to extensibility. Instead of plugin architectures that require separate processes or runtime sandboxing, Rust lets you compile custom business logic directly into the gateway binary. This eliminates the overhead of inter-process communication while maintaining memory safety guarantees.

Building Custom Transformations

The key to production-ready AI gateways is the ability to implement custom transformations that handle your specific business logic. This goes far beyond simple request routing—you need to transform payloads, implement complex authentication, and apply business rules that change based on user context.

Here's where Rust's type system becomes crucial. Unlike dynamic languages where transformation logic can fail at runtime with cryptic errors, Rust's compiler ensures your transformations are correct before they ever see production traffic.

The CNCF blog post on extending AI gateways explains this approach: "What if you need to transform a request body in a way no existing filter supports? What if your business has unique logic that no off-the-shelf gateway can anticipate? You build your own extension."

The architecture typically involves implementing transformation traits that the gateway runtime can invoke. Your custom logic gets compiled into the same binary as the core gateway, eliminating the performance overhead of external plugins while maintaining clear separation of concerns.

Implementing Security Policies

Security in AI gateways requires more than traditional API authentication. You're dealing with sensitive prompts, potentially regulated outputs, and the need to audit every interaction for compliance purposes.

Rust's ownership model makes it particularly well-suited for implementing security policies. Memory safety prevents entire classes of vulnerabilities, while the type system ensures that sensitive data can't accidentally leak between requests or tenants.

A typical security implementation might include:

Request sanitization that removes or masks sensitive information before it reaches upstream providers. This needs to happen at line speed without introducing latency.

Response filtering that ensures outputs comply with your organization's content policies. Unlike simple keyword filtering, this often requires semantic analysis that can't be implemented in gateway configuration files.

Audit logging that captures the complete request-response cycle while respecting privacy requirements. The challenge is doing this efficiently enough to handle high-throughput workloads.

Multi-tenant isolation that ensures one customer's requests can't interfere with another's. This includes not just authentication, but resource isolation and rate limiting.

The key insight is that these policies need to be implemented as code, not configuration. Business requirements change too quickly for static rule engines to keep up.

Rate Limiting for AI Workloads

Traditional rate limiting based on requests per minute breaks down for AI workloads. A single request might consume thousands of tokens and cost dollars, while another uses a few tokens and costs pennies. You need rate limiting that understands the actual resource consumption of AI requests.

This requires custom logic that can:

Parse request payloads to estimate token consumption before sending requests upstream. This prevents expensive requests from consuming your entire quota.

Track actual usage based on response headers from providers. Most AI providers return token counts in their responses, but the format varies between providers.

Implement sliding windows that account for the variable duration of AI requests. A simple token bucket algorithm doesn't work when individual requests can take minutes to complete.

Handle provider-specific limits that might be based on tokens per minute, requests per day, or concurrent connections. Each provider has different limits that need to be tracked independently.

The implementation complexity here is why off-the-shelf gateways struggle with AI workloads. You need custom logic that understands your specific usage patterns and business requirements.

Integration Patterns for Enterprise Environments

Enterprise AI deployments require integration patterns that go beyond simple proxying. You need to integrate with existing identity providers, cost management systems, and observability platforms.

Identity integration typically involves mapping enterprise user identities to provider-specific authentication. This might mean exchanging OIDC tokens for API keys, or implementing custom authentication flows that work with your existing SSO infrastructure.

Cost attribution requires tracking usage per user, project, or cost center. This data needs to flow into existing financial systems, often requiring custom export formats or API integrations.

Observability integration means more than just logging requests. You need distributed tracing that follows requests across multiple AI providers, metrics that understand AI-specific performance characteristics, and alerting that accounts for the variable latency of AI workloads.

Provider abstraction that lets you swap AI providers without changing client code. This requires implementing translation layers that convert between different provider APIs while maintaining semantic compatibility.

As one practitioner noted in their experience building an AI gateway from scratch, the goal is "an OpenAI-compatible proxy with semantic caching, multi-tenant billing, provider fallback, and an admin console." The key word here is "compatible"—your gateway needs to speak the same wire protocol as existing providers while adding the enterprise features you need.

Performance Considerations

AI gateways operate in a unique performance environment. Unlike traditional APIs where you optimize for throughput, AI workloads require optimizing for latency while handling long-running connections efficiently.

Memory management becomes critical when handling streaming responses that might run for minutes. You can't buffer entire responses in memory, but you also can't let memory usage grow unbounded during long-running requests.

Connection pooling needs to account for the fact that AI provider connections might be held open for extended periods. Traditional connection pool implementations that assume short request durations can lead to connection exhaustion.

Backpressure handling is essential when upstream providers are slower than your clients expect. You need to implement flow control that prevents memory exhaustion while maintaining responsiveness.

Resource isolation ensures that one tenant's expensive requests don't impact another tenant's performance. This requires more than just CPU limits—you need to consider memory usage, connection counts, and downstream provider quotas.

The advantage of implementing these optimizations in Rust is that you get predictable performance characteristics. Unlike garbage-collected languages where performance can degrade unpredictably under load, Rust's deterministic memory management ensures consistent latency even during peak traffic.

Deployment and Operations

Production AI gateways need operational characteristics that match enterprise requirements. This means not just high availability, but also the ability to deploy updates without disrupting long-running AI requests.

Rolling deployments become complex when some requests might run for minutes. You need graceful shutdown procedures that allow existing requests to complete while preventing new requests from being routed to instances that are being replaced.

Configuration management needs to support hot reloading of policies without restarting the gateway. Business rules change frequently, and you can't afford downtime every time someone updates a rate limit or security policy.

Health checking must account for the fact that healthy instances might have high latency due to upstream provider performance. Traditional health checks that measure response time can incorrectly mark healthy instances as unhealthy.

Monitoring and alerting require AI-specific metrics. Traditional gateway metrics like requests per second and error rates don't tell you much about AI workload health. You need metrics that track token consumption, model performance, and cost attribution.

Looking Forward

The future of AI gateways lies in their ability to evolve with rapidly changing AI workloads. This means building systems that can be extended and modified without requiring complete rewrites.

Rust's combination of performance, safety, and expressiveness makes it the ideal choice for this challenge. As LangDB's AI Gateway demonstrates, you can build production-ready gateways that provide "unified interface to all LLMs using OpenAI API format" while maintaining the performance characteristics needed for enterprise deployments.

The key is recognizing that AI gateways aren't just proxies—they're platforms for implementing the custom business logic that makes AI workloads production-ready. By building these platforms in Rust, you get the performance of systems languages with the safety guarantees needed for production infrastructure.

The organizations that succeed with AI will be those that can adapt their infrastructure as quickly as the AI landscape evolves. Custom Rust transformations provide the flexibility to implement whatever business logic your AI workloads require, while maintaining the performance and security characteristics that production systems demand.

Kubernetes 1.36 Workload-Aware Scheduling: Gang Scheduling and Resource Optimization for AI/ML Workloads

Matthias Bruns — Thu, 28 May 2026 08:29:57 +0000

Kubernetes 1.36 introduces significant improvements to workload-aware scheduling that fundamentally change how AI/ML and batch processing workloads run in production clusters. The new architecture separates concerns between static templates and runtime state management, enabling true gang scheduling and coordinated resource allocation for the first time as a native Kubernetes feature.

If you're running distributed training jobs, batch processing pipelines, or any workload that requires multiple pods to start together, these changes will eliminate the resource waste and scheduling inefficiencies you've been battling with custom schedulers and workarounds.

Understanding the Architecture Evolution

Kubernetes 1.36 builds on the foundation laid in 1.35 with a clean architectural separation. According to the official Kubernetes blog, the system now separates API concerns where "the Workload API acts as a static template, while the new PodGroup API handles the runtime state."

This separation matters because it allows controllers, status reporting, and future workload-aware features to reason about related pods even when those pods don't require strict all-or-nothing admission. As Ryota Sawada explains, "The workload aware scheduling breaks that template part into workload and the actual runtime object into PodGroup, and that clear separation gives us even further clear connection point for the DRA."

The practical impact is that you can now define workload templates once and reuse them across multiple runtime instances, each with their own PodGroup managing the actual pod lifecycle and coordination.

Gang Scheduling: All-or-Nothing Pod Admission

Gang scheduling solves the fundamental problem of partial workload admission. In traditional Kubernetes scheduling, pods from a distributed training job might be scheduled individually, leading to scenarios where some pods start while others remain pending due to resource constraints. This creates resource waste and training delays.

The new gang scheduling implementation uses the all-or-nothing policy through the minCount field. As documented in the Medium article by Heba Elayoty, "The minCount field defines the quorum: at least that many pods must be schedulable together for the group to be admitted."

This means your distributed training job with 8 worker pods will only start when all 8 pods can be scheduled simultaneously, preventing partial deployments that consume resources without producing useful work.

Workload-Aware Preemption

Kubernetes 1.36 introduces workload-aware preemption through KEP-5710, which treats groups of related pods as single entities for both scheduling and preemption decisions. According to Palark's analysis, "groups of related Pods (PodGroups) are now treated as a single entity for both scheduling and preemption. Rather than removing Pods one by one, the scheduler will figure out" how to handle entire workload groups.

This prevents the scenario where a high-priority workload preempts only some pods from a lower-priority distributed job, leaving the remaining pods running but unable to make progress. Instead, the scheduler considers the entire workload group when making preemption decisions.

Implementing Gang Scheduling for AI/ML Workloads

To implement gang scheduling for your AI workloads, you'll work with the Workload and PodGroup APIs. The Workload API defines the static template for your distributed job, while the PodGroup manages the runtime coordination.

Here's how to structure a distributed training workload that requires all pods to start together:

apiVersion: workload.k8s.io/v1alpha1
kind: Workload
metadata:
  name: distributed-training-template
  namespace: ml-workloads
spec:
  podTemplate:
    spec:
      containers:
      - name: trainer
        image: tensorflow/tensorflow:latest-gpu
        resources:
          requests:
            nvidia.com/gpu: 1
            memory: 8Gi
            cpu: 4
          limits:
            nvidia.com/gpu: 1
            memory: 8Gi
            cpu: 4
        env:
        - name: WORLD_SIZE
          value: "8"
        - name: RANK
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['workload.k8s.io/pod-index']
---
apiVersion: workload.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: training-job-001
  namespace: ml-workloads
spec:
  workloadRef:
    name: distributed-training-template
  minCount: 8
  replicas: 8
  schedulingPolicy: AllOrNothing

The minCount: 8 ensures that all 8 training pods must be schedulable before any of them start. This prevents resource waste from partial deployments and ensures your distributed training job has the full complement of workers before beginning.

Resource Optimization Strategies

Gang scheduling enables several resource optimization strategies that weren't possible with individual pod scheduling:

Coordinated Resource Allocation: Since all pods in a workload group are scheduled together, you can optimize resource requests knowing the entire workload's requirements. This prevents over-provisioning individual pods to account for uncertainty about whether the full workload will be admitted.

Improved Cluster Utilization: Gang scheduling reduces resource fragmentation by ensuring workloads only consume resources when they can run effectively. This is particularly important for GPU clusters where partial workloads tie up expensive resources without producing results.

Predictable Scheduling Behavior: With all-or-nothing admission, you can predict when workloads will start based on available cluster capacity, making it easier to plan batch processing windows and manage SLAs.

Integration with Dynamic Resource Allocation

The workload-aware scheduling improvements in Kubernetes 1.36 integrate closely with Dynamic Resource Allocation (DRA) for GPU scheduling. This integration provides native support for coordinated GPU allocation across pod groups, eliminating the need for custom schedulers or external resource managers.

The clear separation between Workload and PodGroup APIs creates what Ryota Sawada calls "an even further clear connection point for the DRA," enabling sophisticated resource allocation policies that consider the entire workload's GPU requirements when making scheduling decisions.

Production Implementation Guidelines

When implementing workload-aware scheduling in production, consider these key practices:

Start with Non-Critical Workloads: Begin by implementing gang scheduling for development and testing workloads before moving to production training jobs. This allows you to validate the behavior and tune resource requirements without impacting critical workflows.

Monitor Resource Utilization: Track how gang scheduling affects overall cluster utilization. While it may temporarily reduce utilization as workloads wait for full resource availability, it should improve effective utilization by reducing wasted partial deployments.

Set Appropriate Timeouts: Configure reasonable timeouts for workload admission to prevent jobs from waiting indefinitely for resources. This is particularly important in shared clusters where resource availability fluctuates.

Plan for Preemption Scenarios: Design your workload priorities and resource requests with workload-aware preemption in mind. Higher-priority workloads will preempt entire lower-priority workload groups, not individual pods.

Cost Optimization Benefits

Gang scheduling delivers measurable cost benefits for AI/ML workloads:

Reduced GPU Waste: By preventing partial training jobs from consuming GPU resources without making progress, gang scheduling can significantly improve GPU utilization rates. This is critical given GPU costs in cloud environments.

Lower Networking Costs: Distributed training jobs that start all pods simultaneously reduce the time spent in initialization and synchronization phases, minimizing cross-zone networking costs for multi-zone deployments.

Improved Throughput: Coordinated scheduling reduces the time between job submission and completion, allowing you to process more workloads with the same infrastructure investment.

Monitoring and Observability

Effective monitoring of workload-aware scheduling requires tracking metrics at both the individual pod and workload group levels. Key metrics include:

Workload admission latency (time from submission to all pods scheduled)
Resource utilization efficiency (productive vs. idle resource time)
Preemption frequency and impact on workload groups
Queue depth for pending workloads waiting for gang admission

The PodGroup API provides status information about workload coordination that wasn't available with individual pod monitoring, enabling better visibility into distributed workload behavior.

Future Roadmap and Considerations

Kubernetes 1.36 represents the second major iteration of workload-aware scheduling, building on the foundation introduced in 1.35. The Kubernetes blog notes that "The recent 1.35 release of Kubernetes delivered the first tranche of workload aware scheduling improvements," indicating this is an evolving area with more enhancements planned.

Future developments will likely focus on more sophisticated scheduling policies, better integration with cluster autoscaling, and enhanced support for heterogeneous workloads that mix different resource types and scheduling requirements.

The clean API separation introduced in 1.36 provides a solid foundation for these future enhancements while maintaining backward compatibility with existing workloads.

Getting Started

To begin using workload-aware scheduling in Kubernetes 1.36, ensure your cluster has the feature gates enabled and start with simple gang scheduling use cases. The native support eliminates the need for third-party schedulers and custom controllers that many teams have been using as workarounds.

Focus on workloads where coordination provides clear benefits—distributed training, batch processing pipelines, and any application where partial deployment creates resource waste or operational complexity. The investment in migrating to workload-aware scheduling pays dividends through improved resource efficiency and more predictable application behavior.

The architectural improvements in Kubernetes 1.36 make workload-aware scheduling a production-ready solution for coordinated workloads, finally bringing native support for patterns that AI/ML and batch processing teams have needed for years.

Kubernetes 1.36 Workload-Aware Scheduling: Gang Scheduling and Resource Optimization for AI/ML Workloads

Matthias Bruns — Mon, 25 May 2026 08:41:08 +0000

Kubernetes 1.36 introduces significant improvements to workload-aware scheduling that fundamentally change how AI/ML and batch workloads run in production clusters. The new architecture cleanly separates concerns between the Workload API and the PodGroup API, enabling true gang scheduling and sophisticated resource optimization for distributed training jobs.

After working with distributed ML workloads on Kubernetes for years, we've seen too many training jobs fail because pods get scheduled across resource-constrained nodes, or worse, partially scheduled and left hanging. Kubernetes 1.36's workload-aware scheduling finally addresses these pain points with native support for gang scheduling and topology-aware algorithms designed specifically for high-performance distributed workloads.

The Evolution from Kubernetes 1.35 to 1.36

Kubernetes v1.35 introduced the first tranche of workload-aware scheduling improvements, making workloads a first-class citizen for kube-scheduler instead of relying on custom schedulers. However, v1.35 had architectural limitations that v1.36 addresses head-on.

The key breakthrough in v1.36 is the significant architectural evolution that cleanly separates API concerns: the Workload API acts as a static template, while the new PodGroup API handles the runtime state. This separation enables more sophisticated scheduling decisions and better integration with existing Kubernetes controllers.

Kubernetes 1.36 successfully introduces a topology-aware and DRA-aware scheduling algorithm for the Kubernetes kube-scheduler, specifically designed for high-performance distributed workloads like AI/ML training. The DRA (Dynamic Resource Allocation) integration is particularly important for GPU-intensive workloads that need specific hardware configurations.

Understanding Gang Scheduling in Kubernetes 1.36

Gang scheduling solves a critical problem in distributed workloads: ensuring that all pods in a workload group are scheduled together or not at all. Without gang scheduling, you might end up with partial deployments where some pods are running while others are stuck pending, effectively wasting resources and preventing the workload from making progress.

The all-or-nothing policy is at the core of gang scheduling. The minCount field defines the quorum: at least that many pods must be schedulable together for the group to be admitted. This prevents the common scenario where distributed training jobs get partially scheduled and hang indefinitely.

The benefits extend beyond just admission control. Gang scheduling lets controllers, status reporting, future preemption behavior, and future workload-aware features reason about related pods even if those pods do not need strict all-or-nothing admission.

Configuring Workload-Aware Scheduling for AI/ML Workloads

Note: The workload-aware scheduling features in Kubernetes 1.36 are in alpha status. You'll need to enable feature gates and understand that APIs may change in future releases.

The new architecture introduces two key APIs that work together:

Workload API (Static Template)

The Workload API defines the static configuration for your workload group. This includes resource requirements, topology constraints, and scheduling policies that don't change during the workload's lifecycle.

PodGroup API (Runtime State)

The PodGroup API handles the runtime state with native Job controller integration. This separation allows the scheduler to make more informed decisions about pod placement while maintaining clean separation of concerns.

Resource Optimization Strategies

For AI/ML workloads, resource optimization goes beyond simple CPU and memory allocation. You need to consider:

Topology-Aware Scheduling

The new topology-aware scheduling algorithm understands the physical layout of your cluster and can make intelligent decisions about pod placement. This is crucial for distributed training where network topology directly impacts performance.

For GPU-intensive workloads, the scheduler can now consider:

NUMA topology for optimal memory access patterns
GPU interconnect topology (NVLink, InfiniBand)
Network bandwidth between nodes
Storage locality for large datasets

DRA Integration for GPU Workloads

The DRA-aware scheduling algorithm represents a major step forward for GPU resource management. Instead of treating GPUs as simple countable resources, the scheduler can now understand GPU capabilities, memory requirements, and interconnect requirements.

This enables more sophisticated scheduling decisions like:

Ensuring all pods in a training job get GPUs from the same generation
Placing pods to maximize GPU interconnect bandwidth
Avoiding GPU memory fragmentation across training steps

Production Deployment Considerations

Cluster Configuration

Before deploying workload-aware scheduling in production, ensure your cluster is properly configured:

Feature Gates: Enable the necessary alpha feature gates for workload-aware scheduling
Scheduler Configuration: Configure the kube-scheduler to use the new scheduling algorithms
Resource Discovery: Ensure proper resource discovery for GPUs and other specialized hardware

Monitoring and Observability

Workload-aware scheduling introduces new metrics and events that you should monitor:

PodGroup Status: Track the state of pod groups and admission decisions
Scheduling Latency: Monitor how long it takes to schedule workload groups
Resource Utilization: Track resource efficiency improvements from better scheduling

Failure Handling

Gang scheduling changes how you need to think about failure handling:

Partial Failures: With gang scheduling, partial failures result in the entire workload group being rescheduled
Resource Contention: Understand how the scheduler handles resource contention when multiple workload groups compete for the same resources
Preemption Behavior: The new preemption logic considers workload groups as units, not individual pods

Best Practices for AI/ML Workloads

Right-Sizing Workload Groups

Don't make workload groups too large. While gang scheduling ensures all-or-nothing admission, larger groups are harder to schedule and more likely to fail admission. Find the right balance between coordination requirements and schedulability.

Resource Request Accuracy

With workload-aware scheduling, accurate resource requests become even more critical. The scheduler makes admission decisions based on the total resource requirements of the workload group, so underestimating resources can lead to poor performance, while overestimating reduces schedulability.

Topology Constraints

Use topology constraints judiciously. While they can significantly improve performance for distributed workloads, overly restrictive constraints can make workloads unschedulable in smaller clusters.

Migration from Custom Schedulers

Many organizations currently use custom schedulers like Volcano or YuniKorn for gang scheduling. Kubernetes 1.36's native support provides a migration path, but consider:

Feature Parity

Evaluate whether the native workload-aware scheduling provides all the features your current custom scheduler offers. Some advanced features may still require custom schedulers.

Gradual Migration

Plan a gradual migration strategy. You can run both scheduling systems in parallel during the transition period, scheduling different workload types with different schedulers.

Monitoring and Validation

Implement comprehensive monitoring to validate that the native scheduler performs as well as your custom solution for your specific workloads.

Future Outlook

The workload-aware scheduling improvements in Kubernetes 1.36 represent just the beginning. The clean API separation between Workload and PodGroup opens possibilities for future enhancements like:

More sophisticated preemption policies
Advanced resource sharing strategies
Better integration with cluster autoscaling
Enhanced support for multi-tenant workload scheduling

Conclusion

Kubernetes 1.36's workload-aware scheduling represents a significant step forward for AI/ML workloads in production environments. The combination of gang scheduling, topology-aware algorithms, and DRA integration addresses long-standing pain points in distributed workload management.

While these features are still in alpha, they provide a clear path toward native support for complex workload scheduling requirements. Organizations running AI/ML workloads should start evaluating these capabilities and planning migration strategies from custom schedulers.

The architectural improvements in v1.36 create a solid foundation for future enhancements, making this release a turning point for workload-aware scheduling in Kubernetes. For production AI/ML workloads, the investment in understanding and adopting these new capabilities will pay dividends in improved resource utilization, reduced job failures, and simplified cluster management.

Kubernetes 1.36 Pod-Level Resource Managers: Advanced Resource Optimization in Production

Matthias Bruns — Fri, 22 May 2026 10:31:03 +0000

Kubernetes 1.36 fundamentally changes how we think about resource management with the introduction of pod-level resource managers. This alpha feature shifts resource allocation from rigid per-container boundaries to flexible pod-centric specifications, enabling better resource utilization and cost optimization for complex workloads.

Traditional container-level resource management forces you to over-provision resources because containers can't dynamically share CPU and memory within a pod. With multi-container applications—especially those with sidecars—this leads to significant waste. Pod-level resource managers solve this by allowing containers within a pod to share allocated resources intelligently.

Understanding Pod-Level Resource Management

The new pod-level resource management system in Kubernetes 1.36 extends the kubelet's Topology, CPU, and Memory Managers to support pod-centric resource specifications. Instead of specifying requests and limits for each container individually, you can now define resource pools at the pod level that containers share dynamically.

According to the Kubernetes documentation, Kubernetes 1.36 only supports resource requests or limits for specific resource types: cpu and/or memory and/or hugepages at the pod level.

This approach is particularly valuable for:

Applications with variable resource consumption patterns
Multi-container pods where workload distribution changes over time
Performance-sensitive applications requiring NUMA-aware resource allocation
Cost optimization scenarios where precise resource sharing reduces over-provisioning

Current Limitations and Considerations

Before implementing pod-level resource managers, understand the current constraints. As noted in the pod-level resource assignment documentation, Kubernetes 1.36 has specific limitations:

Resource Types: Only CPU, memory, and hugepages resources can be specified at pod-level. You cannot use pod-level management for GPU, storage, or custom resources yet.

Operating System: Pod-level resources are not supported for Windows pods, limiting this feature to Linux-based workloads.

Alpha Status: Since this is an alpha feature, expect potential API changes and stability issues in production environments.

These limitations mean you'll need a hybrid approach—using pod-level management for supported resources while maintaining container-level specifications for others.

Implementing Pod-Level Resource Specifications

Pod-level resource management requires careful planning of your resource allocation strategy. The key is identifying which workloads benefit most from shared resource pools versus those that need strict container isolation.

For applications with predictable resource patterns, traditional container-level management remains appropriate. However, for workloads with:

Bursty CPU usage across containers
Memory sharing between application and sidecar containers
NUMA-sensitive performance requirements

Pod-level management provides significant advantages.

When designing your resource specifications, consider the total resource envelope your pod needs rather than trying to predict individual container requirements. This shift in thinking—from container-centric to pod-centric resource planning—is fundamental to leveraging these new capabilities effectively.

Integration with Vertical Pod Autoscaling

The pod-level resource managers work alongside Kubernetes 1.36's enhanced vertical scaling capabilities. While the blog post mentions that in-place vertical scaling for pod-level resources graduates to beta, this integration enables dynamic resource adjustment without pod restarts.

This combination is powerful for workloads with changing resource needs. Instead of static over-provisioning, you can start with conservative pod-level allocations and let the vertical pod autoscaler adjust resources based on actual usage patterns.

The in-place scaling capability means resource adjustments happen without disrupting running containers, maintaining application availability while optimizing resource utilization.

Performance Optimization Strategies

Pod-level resource managers excel in scenarios requiring fine-grained performance tuning. The integration with kubelet's Topology Manager enables NUMA-aware resource allocation, critical for high-performance computing workloads and memory-intensive applications.

For CPU-intensive workloads, pod-level management allows containers to burst beyond their individual allocations when other containers in the pod are idle. This dynamic sharing improves overall resource utilization without sacrificing performance guarantees.

Memory management becomes more sophisticated with pod-level allocation. Instead of each container holding reserved memory that might go unused, the pod maintains a shared memory pool that containers access as needed. This is particularly beneficial for applications with complementary memory usage patterns.

Hugepages support at the pod level enables better performance for applications requiring large memory pages, such as databases and high-frequency trading systems. Pod-level hugepage allocation simplifies configuration while maintaining performance benefits.

Cost Optimization Through Resource Sharing

The primary cost benefit comes from eliminating resource waste caused by container-level over-provisioning. Traditional approaches require estimating peak resource needs for each container, leading to significant unused capacity.

Pod-level resource management allows you to provision based on aggregate pod requirements rather than individual container peaks. Since containers rarely hit peak usage simultaneously, this approach typically reduces total resource allocation by 20-40% for multi-container applications.

For batch processing workloads, pod-level management enables better resource packing. Instead of reserving resources for each processing stage, you can allocate a resource pool that different containers use as the workload progresses through its lifecycle.

Monitoring and cost attribution become more straightforward with pod-level allocation. Instead of tracking resource usage across multiple containers and trying to understand their interdependencies, you get a single view of pod-level resource consumption.

Production Implementation Guidelines

Rolling out pod-level resource managers requires a phased approach due to the alpha status of this feature. Start with non-critical workloads to gain experience with the new resource management model.

Begin by identifying candidate workloads:

Applications with multiple containers that have complementary resource usage
Workloads currently experiencing resource waste due to over-provisioning
Performance-sensitive applications that could benefit from NUMA awareness

Test thoroughly in staging environments, paying particular attention to resource contention scenarios. While pod-level sharing improves utilization, it can also create new failure modes if not properly configured.

Establish monitoring for pod-level resource utilization to understand actual usage patterns. This data is crucial for tuning resource allocations and identifying opportunities for further optimization.

Plan for gradual migration from container-level to pod-level resource management. You'll likely run hybrid configurations during the transition, with some pods using the new model while others remain on traditional container-level allocation.

Monitoring and Observability

Effective monitoring becomes even more critical with pod-level resource management. Traditional per-container metrics don't provide complete visibility into resource sharing dynamics within pods.

Focus on pod-level resource utilization metrics to understand how containers are actually using shared resources. Look for patterns in resource contention and identify containers that might be starving others of resources.

Implement alerting for resource exhaustion at the pod level, not just individual containers. A container might appear to have sufficient resources allocated while the pod as a whole is resource-constrained.

Track the effectiveness of your resource sharing by comparing pod-level allocation to actual usage over time. This data helps refine your resource specifications and identify workloads that benefit most from pod-level management.

Future Considerations

As pod-level resource managers mature from alpha to stable, expect expanded resource type support and improved Windows compatibility. The current limitations around resource types and operating systems will likely be addressed in future releases.

Integration with other Kubernetes resource management features will continue evolving. Watch for improvements in how pod-level managers interact with resource quotas, limit ranges, and cluster autoscaling.

The performance benefits of NUMA-aware allocation will become more significant as hardware continues evolving toward higher core counts and more complex memory hierarchies. Pod-level resource management positions your infrastructure to take advantage of these hardware improvements.

Getting Started

Kubernetes 1.36's pod-level resource managers represent a significant evolution in container resource management. While the alpha status requires careful evaluation for production use, the potential benefits for resource optimization and cost reduction are substantial.

Start experimenting with pod-level resource management in development environments to understand the operational changes required. Focus on workloads where resource sharing provides clear benefits, and build expertise gradually before broader production deployment.

The combination of pod-level resource managers with enhanced vertical scaling creates new possibilities for dynamic, efficient resource utilization that weren't possible with traditional container-centric approaches. For platform engineering teams managing large-scale Kubernetes deployments, these features offer a path toward significantly improved resource efficiency and reduced infrastructure costs.

Kubernetes 1.36 Pod-Level Resource Managers: Advanced Resource Optimization in Production

Matthias Bruns — Wed, 20 May 2026 13:04:53 +0000

Kubernetes 1.36 brings significant improvements to resource management with pod-level resource managers and enhanced vertical scaling capabilities. These features address long-standing challenges in optimizing infrastructure costs while maintaining application performance, particularly for resource-intensive workloads that require fine-grained control over CPU, memory, and hugepages allocation.

Understanding Pod-Level Resource Management

Traditional Kubernetes resource management operates at the container level, requiring you to specify requests and limits for each container individually. This approach works well for simple applications but becomes cumbersome when managing complex workloads with multiple containers that need to share resources dynamically.

Kubernetes 1.36 only supports resource requests or limits for specific resource types: cpu and/or memory and/or hugepages at the pod level. This represents a fundamental shift from the container-centric model to a more flexible pod-centric approach.

The enhancement proposal seeks to support pod-level resource management, enabling Kubernetes to control the total resource consumption of the pod, relieving users from the burden of meticulously configuring resources for each container.

Pod-Level Resource Managers: Alpha Feature Overview

Kubernetes v1.36 introduces Pod-Level Resource Managers as an alpha feature, bringing a more flexible and powerful resource management model to performance-sensitive workloads. This enhancement extends the kubelet's Topology, CPU, and Memory Managers to support pod-level resource specifications (.spec.resources), evolving them from a strictly per-container allocation model to a pod-centric one.

Since this is an alpha feature, expect potential API changes and use it only in non-production environments for testing and validation. The alpha status means the feature may have bugs and could be removed in future releases without notice.

Key Benefits of Pod-Level Resource Managers

Simplified Resource Configuration: Instead of calculating and distributing resources across multiple containers, you define total pod requirements once. This is particularly valuable for microservices architectures where sidecar containers need to share resources with main application containers.

Better Resource Utilization: Pod-level managers can make more intelligent decisions about resource allocation within the pod boundary, potentially reducing waste from over-provisioning individual containers.

Enhanced Performance for NUMA-aware Workloads: The topology manager integration allows for better NUMA node affinity when resources are managed at the pod level rather than scattered across containers.

Implementing Pod-Level Resource Specifications

As a beta feature, Kubernetes allows you to specify the CPU, memory and hugepages resources at the Pod-level. This means you can now define resource requests and limits for an entire Pod, enabling easier resource sharing without requiring granular, per-container management of these resources.

Here's how to configure pod-level resources:

apiVersion: v1
kind: Pod
metadata:
  name: resource-intensive-app
spec:
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
      hugepages-1Gi: "2Gi"
    limits:
      cpu: "8"
      memory: "16Gi"
      hugepages-1Gi: "4Gi"
  containers:
  - name: main-app
    image: my-app:latest
    # No container-level resource specs needed
  - name: sidecar
    image: monitoring-agent:latest
    # Resources shared from pod-level allocation

A pod's resource usage is restricted by limits, which can also be set at the pod-level or individually for containers within the pod. Again, pod-level limits are prioritized when both are present. This allows for flexible resource management, enabling you to control resource allocation at both the pod and container levels.

In-Place Vertical Scaling for Production Optimization

One of the most significant improvements in Kubernetes 1.36 is the graduation of in-place vertical scaling for pod-level resources to beta status. This feature addresses a critical gap in production resource management by allowing you to adjust resource allocations without recreating pods.

Why In-Place Scaling Matters

Traditional vertical scaling in Kubernetes requires pod recreation, which means:

Application downtime during scaling operations
Loss of local state and cached data
Potential service disruption for stateful applications
Complex coordination for rolling updates

In-place scaling eliminates these issues by modifying resource allocations on running pods, making it ideal for:

Database workloads that benefit from memory adjustments
Machine learning training jobs that need CPU scaling
Batch processing applications with varying resource requirements

Production Implementation Strategy

For production workloads, implement a gradual rollout strategy:

Start with Non-Critical Workloads: Test in-place scaling on development and staging environments first
Monitor Resource Metrics: Use tools like Prometheus and Grafana to track resource utilization before and after scaling
Implement Automation: Create controllers that automatically adjust resources based on metrics
Set Up Alerts: Monitor for scaling failures or resource contention issues

Cost Optimization Through Intelligent Resource Management

Pod-level resource managers enable several cost optimization strategies that weren't practical with container-level management:

Right-Sizing at Scale

Instead of over-provisioning each container to handle peak loads, you can:

Set pod-level limits based on actual aggregate usage patterns
Allow containers to burst and share resources dynamically
Reduce the total resource footprint by eliminating per-container safety margins

Dynamic Resource Allocation

With in-place scaling, implement time-based resource adjustments:

Scale down resources during off-peak hours
Increase allocation for batch processing windows
Adjust based on seasonal traffic patterns

Improved Bin Packing

Pod-level resource specifications provide the scheduler with better information for node placement decisions, leading to:

Higher node utilization rates
Reduced cluster size requirements
Better cost per workload ratios

Performance Considerations and Best Practices

NUMA Topology Optimization

For high-performance computing workloads, pod-level resource managers work with the kubelet's topology manager to:

Ensure CPU and memory allocation on the same NUMA node
Optimize for cache locality and memory bandwidth
Reduce cross-NUMA traffic for better performance

Memory Management for Large Pages

When using hugepages with pod-level resources:

Pre-allocate hugepages on nodes before scheduling pods
Monitor hugepage usage to prevent fragmentation
Consider the impact on other workloads sharing the node

CPU Affinity and Isolation

Pod-level CPU management allows for:

Better CPU core allocation strategies
Reduced context switching overhead
Improved performance for CPU-intensive applications

Monitoring and Observability

Implement comprehensive monitoring for pod-level resource usage:

Key Metrics to Track

Pod-level CPU and memory utilization
Resource request vs. actual usage ratios
Scaling operation success rates
Node-level resource fragmentation

Alerting Strategies

Set up alerts for:

Pods approaching resource limits
Failed in-place scaling operations
Node resource pressure conditions
Unusual resource usage patterns

Migration Path from Container-Level Resources

When migrating existing workloads to pod-level resource management:

Audit Current Resource Configurations: Document existing container resource specifications
Calculate Aggregate Requirements: Sum up total pod resource needs
Test in Staging: Validate behavior with pod-level specifications
Gradual Migration: Move workloads incrementally to minimize risk
Monitor Performance: Compare performance metrics before and after migration

Security and Isolation Considerations

While pod-level resource management offers flexibility, maintain security boundaries:

Use resource quotas at the namespace level to prevent resource exhaustion
Implement network policies to isolate sensitive workloads
Consider using separate node pools for workloads with different security requirements
Monitor for potential resource-based side-channel attacks

Looking Forward: Production Readiness

As pod-level resource managers mature from alpha to beta and eventually to stable, expect:

Enhanced integration with cluster autoscaling
Better support for GPU and custom resources
Improved observability and debugging tools
More sophisticated resource allocation algorithms

Resource management is a cornerstone of running applications in Kubernetes. Properly managing resources ensures that your applications perform optimally, that your cluster remains stable, and that resources are efficiently utilized.

Kubernetes 1.36's pod-level resource managers represent a significant step forward in making resource management more intuitive and cost-effective. By adopting these features thoughtfully and monitoring their impact, you can achieve better resource utilization, reduced infrastructure costs, and improved application performance.

Start experimenting with these features in non-production environments, develop your operational procedures, and prepare for broader adoption as the features graduate to stable status. The investment in understanding and implementing pod-level resource management will pay dividends in both cost savings and operational efficiency.

Open Component Model in Production: Building Software Bills of Delivery for Cloud-Native Supply Chains

Matthias Bruns — Sun, 10 May 2026 07:52:31 +0000

The Open Component Model (OCM) represents a fundamental shift in how we approach software supply chain security. While most organizations struggle with visibility into their distributed systems' dependencies, OCM provides an open standard for creating comprehensive Software Bills of Delivery (SBOD) that capture everything from container images to configuration files, signatures, and version constraints across your entire delivery pipeline.

Unlike traditional software bills of materials that focus on source dependencies, OCM tracks the actual artifacts you deliver to production. This distinction matters when you're managing complex cloud-native applications where the gap between what you build and what you deploy can introduce significant security risks.

What Makes OCM Different

The Open Component Model specification defines OCM as "an open standard to describe software-bill-of-deliveries (SBOD)" that is "technology-agnostic and machine-readable format focused on the software artifacts that must be delivered for software products."

This focus on delivery artifacts rather than source code dependencies addresses a critical gap in most supply chain security approaches. When you deploy a microservices application, you're not just shipping your application code – you're delivering container images, Helm charts, configuration files, certificates, and often third-party components. OCM captures all of these elements with their relationships and provenance.

The model organizes everything around components and component versions. A component represents a logical unit of software (think: a microservice, a library, or an entire application), while component versions represent specific, immutable releases of that component. Each component version contains:

Resources: The actual artifacts you deliver (container images, binaries, charts)
Sources: References to source code and build information
References: Dependencies on other components
Signatures: Cryptographic proof of authenticity and integrity

Building Software Bills of Delivery

Creating effective software bills of delivery with OCM starts with the OCM CLI, which provides the primary interface for interacting with OCM elements. The CLI helps you "create component versions and embed them in CI and CD processes."

To get started with the CLI, you can install it using the official installer:

curl -sfL https://ocm.software/install-cli.sh | bash

The CLI also supports multiple installation methods including Nix. According to the repository documentation, you can use Nix for ad-hoc execution or permanent installation:

# ad-hoc cmd execution
nix run github:open-component-model/ocm -- --help

# install development version
nix profile install github:open-component-model/ocm

Note: The main OCM project is currently marked as "Work In Progress" with the warning that "expect heavy changes, especially in the Library API." The team is working on a stable API, so consider this when planning production deployments.

The CLI operates on component descriptors – JSON or YAML files that define your component versions. These descriptors capture not just what you're delivering, but how it relates to other components and where it came from.

Repository Mappings and Storage

OCM supports multiple storage backends through its repository mapping system. The current implementation supports:

OCI repositories: Using the repository prefix path of an OCI repository to implement an OCM repository
CTF (Common Transport Format): File-based binding for representing component versions as filesystem content (directory, tar, tgz)

This flexibility means you can store your software bills of delivery alongside your container images in existing OCI registries, or package them as portable files for air-gapped environments. The OCI mapping is particularly powerful because it leverages existing registry infrastructure while adding OCM's metadata layer.

Cryptographic Signing and Verification

Supply chain security requires more than just tracking – you need cryptographic proof that artifacts haven't been tampered with. OCM provides built-in signing and verification capabilities that work across all supported repository implementations.

The signing process captures not just individual artifacts, but the entire component version including its relationships. This means you can verify not only that a container image is authentic, but that its configuration, dependencies, and metadata are also untampered.

OCM's approach to signing addresses a common problem in cloud-native environments: how do you verify the integrity of complex, multi-artifact deployments? Traditional approaches might sign individual container images, but OCM signs the complete delivery package.

Automated Deployment with OCM Controllers

For production deployments, manual CLI operations don't scale. The OCM Controllers are "designed to enable the automated deployment of software using the Open Component Model and Flux."

The OCM K8s Toolkit provides a Kubernetes operator that deploys OCM resources into your cluster. You can install it using the provided Helm chart:

helm install ocm-k8s-toolkit oci://ghcr.io/open-component-model/kubernetes/controller/chart \
    --namespace ocm-k8s-toolkit-system \
    --create-namespace

The controllers integrate with GitOps workflows, particularly when combined with FluxCD. This integration enables deploying Helm charts or Kustomizations from OCM resources while maintaining full traceability from source to deployment.

Cross-Environment Transport

One of OCM's most powerful features is its ability to transport component versions across different environments while maintaining integrity and traceability. This capability is essential for organizations that need to move software between development, staging, and production environments, or across different cloud providers.

The transport mechanism works at the component version level, meaning you can move entire applications with all their dependencies and metadata intact. This includes scenarios like:

Promoting applications from staging to production
Deploying to air-gapped environments
Moving workloads between cloud providers
Disaster recovery scenarios

OCM's transport preserves signatures and verification chains, so you can prove that what you're deploying in production is exactly what was tested and approved in your staging environment.

Integration Patterns

OCM's design philosophy emphasizes integration with existing toolchains rather than replacement. The model provides a common language that different tools can use to exchange information about software artifacts.

For example, your CI pipeline might use OCM to package build artifacts with their metadata, your security scanning tools might add vulnerability information as OCM labels, and your deployment tools might consume OCM component versions to understand what they're deploying.

This approach allows you to build comprehensive supply chain tracking without replacing your entire toolchain. OCM acts as the integration layer that connects your existing tools with consistent metadata and provenance tracking.

Production Considerations

When implementing OCM in production, several factors require careful consideration:

Repository Strategy: Choose between OCI-based storage for integration with existing registries, or CTF for scenarios requiring file-based transport. Many organizations use OCI repositories for active development and CTF for archival or air-gapped deployments.

Signing Infrastructure: Establish clear policies for who can sign component versions and how signing keys are managed. OCM supports both public key and certificate-based verification, allowing integration with existing PKI infrastructure.

Automation Integration: Plan how OCM fits into your existing CI/CD pipelines. The CLI can be embedded in build processes, while the controllers handle deployment automation.

Governance and Compliance: OCM's comprehensive metadata capture supports compliance requirements, but you need policies defining what information to capture and how to use it for audit purposes.

The Future of Software Supply Chain Security

OCM represents a maturing approach to supply chain security that goes beyond simple dependency scanning. By focusing on delivery artifacts and their relationships, it provides visibility into what actually gets deployed to production environments.

The project's commitment to open standards and integration with existing tools makes it a practical choice for organizations serious about supply chain security. As cloud-native environments become more complex, having a standardized way to track, sign, and verify entire application deployments becomes essential.

For teams building distributed systems, OCM offers a path to comprehensive supply chain visibility without requiring wholesale changes to existing development and deployment processes. The key is starting with clear goals for what you want to track and verify, then building OCM integration incrementally into your existing workflows.

The combination of standardized metadata, cryptographic verification, and cross-environment transport capabilities positions OCM as a foundational technology for secure software delivery in cloud-native environments.

Ingress Migration Strategy: From Deprecated Controllers to Gateway API

Matthias Bruns — Mon, 04 May 2026 08:07:24 +0000

The Kubernetes community's announcement of Ingress NGINX's retirement in March 2026 has created an urgent need for migration planning across thousands of production clusters. With no security patches, bug fixes, or updates coming after the final v1.15.1 release, organizations must act now to avoid running unmaintained software with escalating security risks.

This isn't just about swapping one ingress controller for another. It's an opportunity to modernize your Kubernetes networking stack with better abstractions, improved security models, and future-proof architectures. This guide provides a practical framework for evaluating your options and executing a zero-downtime migration.

Understanding Your Current State

Before choosing a migration path, you need to audit what you're actually using. The migration assessment tool helps identify feature dependencies, but start with this basic check:

kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx

If this returns pods, you're affected. Now catalog your ingress resources and their complexity:

kubectl get ingress --all-namespaces -o yaml > current-ingress-config.yaml
kubectl get configmap -n ingress-nginx ingress-nginx-controller -o yaml > current-configmap.yaml

Pay special attention to:

Custom annotations beyond basic path routing
ConfigMap customizations for global behavior
TCP/UDP services configuration
SSL/TLS termination patterns
Rate limiting and authentication rules

The complexity of these configurations will determine your migration strategy and timeline.

Migration Path Options

You have three primary migration targets, each with distinct trade-offs:

F5 NGINX Ingress Controller

The most direct path for teams heavily invested in NGINX-specific features. NGINX Ingress Controller v5.4.0 introduces native validation and CORS support specifically to ease ingress-nginx migrations.

Best for: Organizations with complex NGINX configurations, custom snippets, or NGINX Plus licensing.

Migration complexity: Low to medium, depending on annotation usage.

Gateway API with Envoy Gateway

The forward-looking choice that embraces Kubernetes' networking future. Gateway API provides role-based configuration, better traffic management primitives, and vendor neutrality.

Best for: Teams building new applications or willing to invest in modern Kubernetes networking patterns.

Migration complexity: Medium to high, requires rethinking traffic management concepts.

Cloud Provider Solutions (ALB, GKE Ingress, etc.)

Platform-specific controllers that integrate deeply with cloud load balancers. AWS recommends migrating to AWS Load Balancer Controller first, then to Gateway API when ready.

Best for: Teams already committed to a specific cloud platform.

Migration complexity: Medium, with vendor lock-in considerations.

Risk Assessment Framework

Evaluate each migration option against these critical factors:

Feature Parity Analysis

Not every ingress-nginx feature has direct equivalents in alternative controllers. Create a feature mapping document:

# Example feature assessment
current_features:
  - ssl_redirect: "nginx.ingress.kubernetes.io/ssl-redirect"
  - rate_limiting: "nginx.ingress.kubernetes.io/rate-limit"
  - custom_headers: "nginx.ingress.kubernetes.io/configuration-snippet"

migration_gaps:
  nginx_controller:
    - ssl_redirect: "Direct annotation support"
    - rate_limiting: "VirtualServer CRD required"
    - custom_headers: "Policy CRD recommended"

  gateway_api:
    - ssl_redirect: "HTTPRoute redirect filter"
    - rate_limiting: "Implementation-specific policy"
    - custom_headers: "ExtensionRef to controller policy"

Operational Impact

Consider the blast radius of your migration:

Traffic patterns: Peak load times when you can't afford disruption
Team expertise: Learning curve for new APIs and troubleshooting
Tooling integration: CI/CD pipelines, monitoring, and GitOps workflows
Compliance requirements: Change management and approval processes

Security Implications

Each migration path has different security characteristics:

NGINX Ingress Controller: Familiar security model, but requires ongoing license management for Plus features
Gateway API: Role-based access control (RBAC) with separation of concerns between platform and application teams
Cloud controllers: Integration with cloud IAM and security services

Zero-Downtime Migration Patterns

Pattern 1: Parallel Controller Migration

Run both controllers simultaneously during the transition period. This approach minimizes risk but requires careful traffic management.

# Install new controller in separate namespace
kubectl create namespace nginx-ingress-new

# Deploy with different ingress class
helm install nginx-ingress-new nginx-stable/nginx-ingress \
  --namespace nginx-ingress-new \
  --set controller.ingressClass=nginx-new

Gradually migrate services by updating the ingressClassName field:

# Before
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-app
spec:
  ingressClassName: nginx  # Old controller
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

# After
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-app
spec:
  ingressClassName: nginx-new  # New controller
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

Pattern 2: Blue-Green Cluster Migration

For organizations with sophisticated deployment pipelines, migrating entire clusters provides the cleanest separation.

# Create new cluster with target ingress controller
# Deploy applications to new cluster
# Switch DNS/load balancer traffic
# Decommission old cluster

This pattern works well with GitOps workflows where infrastructure and applications are versioned together.

Pattern 3: Gateway API Progressive Migration

When migrating to Gateway API, start with basic HTTP routing and gradually adopt advanced features:

# Phase 1: Basic routing
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-app
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - app.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: example-service
      port: 80

# Phase 2: Add traffic policies
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-app
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - app.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301
    backendRefs:
    - name: example-service
      port: 80

Migration Execution Strategy

Phase 1: Preparation (Weeks 1-2)

Audit current configuration using the NGINX migration tool
Set up staging environment that mirrors production traffic patterns
Train team members on new APIs and troubleshooting approaches
Update monitoring and alerting for new controller metrics and logs

Phase 2: Pilot Migration (Weeks 3-4)

Select low-risk applications with simple routing requirements
Deploy target controller in parallel with existing setup
Migrate pilot applications and validate functionality
Document lessons learned and refine migration procedures

Phase 3: Production Migration (Weeks 5-8)

Create migration timeline prioritizing applications by criticality
Implement automated testing to validate each migration step
Execute migrations during maintenance windows with rollback procedures
Monitor application performance and user experience metrics

Phase 4: Cleanup (Weeks 9-10)

Remove old ingress controller after all applications are migrated
Update documentation and runbooks for new architecture
Conduct post-migration review to capture improvements for future migrations

Automation and Tooling

The NGINX Ingress Migration Tool provides automated assessment and conversion suggestions. For Gateway API migrations, consider tools like:

Gateway API conformance tests to validate controller behavior
Helm chart templating to generate both Ingress and Gateway API resources during transition
GitOps automation to manage configuration drift between environments

Common Pitfalls and Solutions

Annotation Hell

Many teams discover they're using dozens of controller-specific annotations. The NGINX documentation recommends a "CRD-first" approach, migrating to policy-based configuration instead of annotation sprawl.

Load Balancer IP Changes

Cloud controller migrations often result in new load balancer IPs. Plan DNS TTL reductions and consider using CNAME records pointing to load balancer hostnames instead of A records with IPs.

Feature Gaps

Not every ingress-nginx feature has direct equivalents. Document these gaps early and plan workarounds or application changes. Sometimes the "missing" feature represents an opportunity to simplify your architecture.

Looking Forward

The ingress-nginx retirement forces a decision that many teams have been postponing. While disruptive in the short term, this migration presents an opportunity to modernize your Kubernetes networking stack with better security models, improved observability, and future-proof APIs.

Gateway API represents the future of Kubernetes networking, with growing ecosystem support and vendor adoption. Even if you choose an intermediate migration target like NGINX Ingress Controller or a cloud provider solution, plan for eventual Gateway API adoption as the standard matures.

The key to success is starting early, testing thoroughly, and migrating incrementally. The March 2026 deadline may seem distant, but complex production environments require months of planning and validation to ensure zero-downtime transitions.

Your future self will thank you for investing in this migration properly rather than rushing through it under deadline pressure.

Software Bills of Delivery: Beyond SBOMs with Component Models

Matthias Bruns — Fri, 01 May 2026 07:55:36 +0000

Traditional Software Bills of Materials (SBOMs) have served as the foundation for software supply chain transparency, but they're showing their limitations in today's complex, distributed environments. While SBOMs catalog components and dependencies, they fall short of tracking the complete delivery lifecycle across cloud-native architectures. Enter Software Bills of Delivery and component models—a more comprehensive approach that tracks not just what's in your software, but how it moves through your entire delivery pipeline.

The SBOM Foundation and Its Limits

A Software Bill of Materials is a formal record of all software component parts and software dependencies used in application development and delivery. The National Telecommunications and Information Administration (NTIA) has established baseline requirements for SBOM creation and delivery, making them increasingly mandatory for government contracts and enterprise software.

SBOMs excel at providing a snapshot of components at build time. They answer critical questions like "What open source libraries are we using?" and "Do we have any known vulnerabilities?" But they struggle with the dynamic nature of modern software delivery:

Static snapshots vs. dynamic environments: SBOMs capture a point-in-time view, but container images, serverless functions, and microservices change constantly
Limited artifact tracking: They focus on source dependencies but miss runtime artifacts, configuration files, and deployment metadata
Deployment context gaps: SBOMs don't track where components are deployed, how they're configured, or their runtime relationships

These limitations become critical when you're managing hundreds of microservices across multiple clusters, each with their own dependency chains and deployment patterns.

Component Models: The Next Evolution

Component models extend beyond traditional SBOMs by treating software artifacts as first-class entities with rich metadata throughout their lifecycle. The Open Component Model (OCM) offers a practical solution for the delivery of software artifacts, providing a standardized way to describe, package, and transport software components with their complete context.

Unlike SBOMs, component models track:

Artifact provenance: Complete build and deployment history
Runtime dependencies: Not just compile-time dependencies, but actual runtime relationships
Configuration metadata: Environment-specific settings and their sources
Delivery pipelines: How components move through CI/CD systems
Deployment topology: Where components run and how they communicate

This comprehensive tracking enables what we call "Software Bills of Delivery"—living documents that evolve with your software as it moves through environments.

Software Bills of Delivery in Practice

A Software Bill of Delivery goes beyond listing components to track the complete delivery journey. Here's what this looks like in a real cloud-native environment:

# Example component descriptor with delivery metadata
apiVersion: delivery.appetizer.io/v1
kind: ComponentDelivery
metadata:
  name: user-service-v2.1.3
  namespace: production
spec:
  component:
    name: user-service
    version: 2.1.3
    source:
      repository: github.com/company/user-service
      commit: abc123def456
      buildTime: "2024-01-15T10:30:00Z"

  artifacts:
    - type: container
      name: user-service
      digest: sha256:8f8a8b8c8d8e8f...
      registry: registry.company.com

    - type: helm-chart
      name: user-service-chart
      version: 2.1.3
      repository: oci://registry.company.com/charts

    - type: config
      name: database-config
      source: vault://secrets/user-service/db

  dependencies:
    runtime:
      - name: postgres
        version: "14.9"
        location: cluster-db.production.svc.cluster.local

      - name: redis
        version: "7.2"
        location: elasticache.us-west-2.amazonaws.com

    buildtime:
      - name: golang
        version: "1.21.5"

      - name: gorilla/mux
        version: "v1.8.0"

  deployment:
    environments:
      - name: staging
        deployedAt: "2024-01-15T14:20:00Z"
        status: successful
        replicas: 2

      - name: production
        deployedAt: "2024-01-15T16:45:00Z"
        status: successful
        replicas: 5

    pipeline:
      buildId: "build-456789"
      approvals:
        - approver: security-team
          timestamp: "2024-01-15T12:00:00Z"
        - approver: platform-team
          timestamp: "2024-01-15T14:00:00Z"

This delivery manifest captures not just what components exist, but their complete journey through your delivery pipeline.

Implementing Component Models for Supply Chain Security

Modern software supply chain security requires tracking components across their entire lifecycle. Here's how component models address key security concerns:

Provenance Tracking

Component models maintain cryptographic proof of artifact origins:

type ComponentProvenance struct {
    BuildSystem   string            `json:"buildSystem"`
    SourceRepo    string            `json:"sourceRepo"`
    CommitHash    string            `json:"commitHash"`
    Builder       string            `json:"builder"`
    BuildTime     time.Time         `json:"buildTime"`
    Attestations  []Attestation     `json:"attestations"`
    Signatures    []Signature       `json:"signatures"`
}

type Attestation struct {
    Type      string    `json:"type"`      // "build", "test", "security-scan"
    Result    string    `json:"result"`    // "pass", "fail", "warning"
    Details   string    `json:"details"`
    Timestamp time.Time `json:"timestamp"`
    Verifier  string    `json:"verifier"`
}

Runtime Dependency Resolution

Unlike static SBOMs, component models track actual runtime dependencies:

# Runtime dependency discovery
runtimeDependencies:
  discovered:
    - service: payment-api
      version: "1.4.2"
      protocol: https
      endpoint: payment-api.internal:443
      discoveredAt: "2024-01-15T16:50:00Z"

  declared:
    - service: user-db
      version: "14.9"
      protocol: postgresql
      endpoint: user-db.production:5432

  external:
    - service: stripe-api
      endpoint: api.stripe.com
      version: "2023-10-16"
      sla: "99.9%"

Vulnerability Correlation

Component models enable precise vulnerability tracking across deployments:

{
  "vulnerabilityReport": {
    "component": "user-service:2.1.3",
    "scanTime": "2024-01-15T17:00:00Z",
    "findings": [
      {
        "cve": "CVE-2024-0001",
        "severity": "high",
        "affectedArtifacts": [
          "container:user-service@sha256:8f8a8b8c8d8e8f"
        ],
        "deploymentImpact": [
          {
            "environment": "production",
            "instances": 5,
            "exposure": "internal-only",
            "mitigations": ["network-policy", "pod-security-policy"]
          }
        ]
      }
    ]
  }
}

Distributed Systems and Component Relationships

In distributed architectures, understanding component relationships becomes crucial. Component models map these relationships explicitly:

Service Mesh Integration

componentRelationships:
  upstreamServices:
    - name: api-gateway
      traffic: "100%"
      protocol: http/2

  downstreamServices:
    - name: user-db
      queries: ["SELECT", "UPDATE", "INSERT"]
      connectionPool: 10

    - name: notification-service
      protocol: grpc
      async: true

  sidecarComponents:
    - name: envoy-proxy
      version: "1.28.0"
      config: istio-proxy-config

    - name: datadog-agent
      version: "7.49.0"
      config: monitoring-config

Cross-Cluster Dependencies

Modern applications span multiple clusters and cloud providers. Component models track these distributed relationships:

crossClusterDependencies:
  - component: shared-cache
    cluster: us-west-2-cache
    endpoint: redis.shared.company.internal
    latency: "< 5ms"

  - component: analytics-pipeline
    cluster: data-processing
    protocol: kafka
    topics: ["user-events", "audit-logs"]

  - component: cdn
    provider: cloudflare
    endpoints: ["cdn.company.com"]
    cachePolicies: ["static-assets", "api-responses"]

Automation and Tooling

Component models shine when integrated into automated workflows. Here's how to implement automated delivery tracking:

CI/CD Integration

#!/bin/bash
# Build pipeline integration
set -e

# Build the component
docker build -t user-service:${BUILD_ID} .

# Generate component descriptor
cat > component-descriptor.yaml <<EOF
apiVersion: ocm.software/v3alpha1
kind: ComponentVersion
metadata:
  name: user-service
  version: ${BUILD_ID}
spec:
  provider: company.com
  sources:
    - name: source
      type: git
      access:
        type: github
        repoUrl: github.com/company/user-service
        ref: ${GIT_COMMIT}
  resources:
    - name: image
      type: ociImage
      access:
        type: ociRegistry
        imageReference: registry.company.com/user-service:${BUILD_ID}
EOF

# Sign and upload
ocm add resources component-descriptor.yaml
ocm sign component-descriptor.yaml --private-key=${SIGNING_KEY}
ocm transfer component component-descriptor.yaml ${OCI_REGISTRY}

Runtime Discovery

Automated discovery keeps component models current:

func discoverRuntimeDependencies(ctx context.Context, podName string) (*RuntimeDependencies, error) {
    // Network traffic analysis
    connections, err := analyzeNetworkConnections(podName)
    if err != nil {
        return nil, err
    }

    // Service mesh integration
    envoyConfig, err := getEnvoyConfiguration(podName)
    if err != nil {
        return nil, err
    }

    // DNS resolution tracking
    dnsQueries, err := analyzeDNSQueries(podName)
    if err != nil {
        return nil, err
    }

    return &RuntimeDependencies{
        NetworkConnections: connections,
        ServiceMeshRoutes:  envoyConfig.Routes,
        DNSResolutions:     dnsQueries,
        DiscoveredAt:       time.Now(),
    }, nil
}

Security Benefits and Compliance

Component models provide significant security advantages over traditional SBOMs:

Compliance Automation

CISA's SBOM requirements focus on transparency, but component models enable automated compliance checking:

complianceChecks:
  - policy: "no-high-severity-vulnerabilities"
    status: "enforced"
    lastCheck: "2024-01-15T18:00:00Z"

  - policy: "signed-artifacts-only"
    status: "enforced"
    exceptions: []

  - policy: "approved-base-images"
    status: "warning"
    violations:
      - component: "legacy-service"
        reason: "using deprecated base image"

Supply Chain Attack Detection

Component models enable sophisticated attack detection:

type SupplyChainAnomaly struct {
    Type        string    `json:"type"`
    Component   string    `json:"component"`
    Description string    `json:"description"`
    Severity    string    `json:"severity"`
    DetectedAt  time.Time `json:"detectedAt"`
}

func detectAnomalies(current, previous *ComponentModel) []SupplyChainAnomaly {
    var anomalies []SupplyChainAnomaly

    // Unexpected dependency changes
    for _, dep := range current.Dependencies {
        if !containsDependency(previous.Dependencies, dep) {
            anomalies = append(anomalies, SupplyChainAnomaly{
                Type:        "unexpected-dependency",
                Component:   current.Name,
                Description: fmt.Sprintf("New dependency added: %s", dep.Name),
                Severity:    "medium",
                DetectedAt:  time.Now(),
            })
        }
    }

    // Build environment changes
    if current.BuildEnvironment.Builder != previous.BuildEnvironment.Builder {
        anomalies = append(anomalies, SupplyChainAnomaly{
            Type:        "builder-change",
            Component:   current.Name,
            Description: "Build environment changed",
            Severity:    "high",
            DetectedAt:  time.Now(),
        })
    }

    return anomalies
}

Implementation Roadmap

Moving from SBOMs to component models requires a phased approach:

Phase 1: SBOM Enhancement

Start by enriching existing SBOMs with delivery metadata. Integrate SBOM generation directly into CI/CD pipelines and add deployment context.

Phase 2: Component Model Adoption

Implement component descriptors alongside SBOMs. Begin tracking artifact relationships and deployment topology.

Phase 3: Runtime Integration

Deploy discovery agents to automatically update component models with runtime dependencies and actual service relationships.

Phase 4: Advanced Analytics

Implement anomaly detection, compliance automation, and predictive security analysis based on component model data.

The Future of Software Supply Chain Transparency

Software Bills of Delivery represent the evolution from static documentation to dynamic, living records of software components and their relationships. By implementing component models, organizations gain unprecedented visibility into their software supply chains, enabling proactive security, automated compliance, and confident deployment practices.

The shift from SBOMs to comprehensive component models isn't just about better documentation—it's about building software supply chains that are transparent, secure, and resilient by design. As distributed systems become more complex, this level of visibility becomes essential for maintaining security and operational excellence.

Component models provide the foundation for the next generation of software supply chain security tools, enabling organizations to move from reactive vulnerability management to proactive risk prevention. The question isn't whether to adopt this approach, but how quickly you can implement it to stay ahead of emerging threats and compliance requirements.

Open Component Model in Production: Building Software Bills of Delivery for Cloud-Native Supply Chains

Matthias Bruns — Tue, 28 Apr 2026 07:58:41 +0000

The software supply chain has become a critical attack vector, with incidents like SolarWinds and Log4Shell exposing how vulnerable our interconnected systems really are. Traditional Software Bills of Materials (SBOMs) tell us what components exist, but they don't capture the full picture of how software actually gets delivered and deployed. Enter the Open Component Model (OCM) – an open standard that goes beyond simple dependency tracking to create comprehensive Software Bills of Delivery (SBOD) for cloud-native environments.

Unlike SBOMs that focus on what's in your code, OCM describes the complete delivery pipeline – from source repositories to runtime configurations. This matters because modern applications aren't just code; they're complex assemblies of container images, Helm charts, configuration files, and deployment manifests spread across multiple repositories and registries.

What Makes OCM Different from Traditional Supply Chain Tools

The Open Component Model specification defines a technology-agnostic format for describing software delivery artifacts. Where SBOMs answer "what libraries am I using?", OCM answers "what exactly needs to be delivered for this software to run?"

This distinction is crucial in cloud-native environments where your application might consist of:

Multiple microservices from different repositories
Container images with specific tags and digests
Kubernetes manifests with environment-specific configurations
Helm charts with values files
External dependencies like databases or message queues

Traditional tools struggle to connect these dots across repository boundaries. OCM creates a unified model that captures not just the artifacts, but their relationships and deployment context.

Core OCM Concepts for Production Implementation

Component Descriptors

At the heart of OCM is the component descriptor – a machine-readable manifest that describes a deliverable software component. Think of it as a shipping manifest that lists everything needed to successfully deploy and run your software.

apiVersion: ocm.software/v3alpha1
kind: ComponentVersion
metadata:
  name: my-web-app
  version: 1.2.3
  provider: appetizer-labs
spec:
  resources:
  - name: app-image
    type: ociImage
    version: 1.2.3
    access:
      type: ociRegistry
      imageReference: registry.example.com/my-web-app:1.2.3
  - name: helm-chart
    type: helmChart
    version: 1.2.3
    access:
      type: ociRegistry
      imageReference: registry.example.com/charts/my-web-app:1.2.3
  - name: config
    type: yaml
    access:
      type: github
      repository: github.com/appetizer-labs/my-web-app-config
      ref: v1.2.3
  sources:
  - name: app-source
    type: git
    access:
      type: github
      repository: github.com/appetizer-labs/my-web-app
      ref: v1.2.3

This descriptor captures not just what artifacts exist, but where they're stored and how to access them. The sources section maintains traceability back to source code, while resources describes the deliverable artifacts.

Resource Types and Access Methods

OCM supports multiple resource types out of the box:

ociImage for container images
helmChart for Helm packages
yaml for configuration files
executable for binaries
blob for arbitrary data

Access methods define how to retrieve these resources:

ociRegistry for OCI-compliant registries
github for Git repositories
s3 for object storage
localBlob for local files

This flexibility lets you model complex delivery scenarios where artifacts live in different systems.

Setting Up OCM in Multi-Repository Environments

Installation and Basic Configuration

Start by installing the OCM CLI tool:

# Install OCM CLI
curl -L https://github.com/open-component-model/ocm/releases/latest/download/ocm-linux-amd64.tar.gz | tar xz
sudo mv ocm /usr/local/bin/

# Verify installation
ocm version

For multi-repository setups, you'll typically have one repository per microservice plus a central repository for component descriptors. Here's a practical structure:

my-platform/
├── services/
│   ├── user-service/
│   ├── payment-service/
│   └── notification-service/
├── charts/
│   └── platform-chart/
└── components/
    └── platform-component/
        └── component-descriptor.yaml

Creating Component Descriptors for Microservices

Each microservice should generate its own component descriptor during the CI/CD process. Here's how to automate this with GitHub Actions:

name: Build and Package Component
on:
  push:
    tags: ['v*']

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Build container image
      run: |
        docker build -t ${{ github.repository }}:${{ github.ref_name }} .
        docker push ${{ github.repository }}:${{ github.ref_name }}

    - name: Generate component descriptor
      run: |
        ocm create componentversion \
          --provider appetizer-labs \
          --name ${{ github.repository }} \
          --version ${{ github.ref_name }} \
          --resource name=app-image,type=ociImage,version=${{ github.ref_name }},access='{"type":"ociRegistry","imageReference":"${{ github.repository }}:${{ github.ref_name }}"}' \
          --source name=source,type=git,access='{"type":"github","repository":"${{ github.repository }}","ref":"${{ github.ref_name }}"}'

    - name: Push component descriptor
      run: |
        ocm transfer componentversion component-descriptor.yaml \
          ghcr.io/${{ github.repository_owner }}/components

This approach ensures every service build produces a traceable component descriptor that captures the exact artifacts and their sources.

Aggregating Components for Platform Delivery

For platform-level deployments, create aggregate components that reference individual service components:

apiVersion: ocm.software/v3alpha1
kind: ComponentVersion
metadata:
  name: my-platform
  version: 2.1.0
  provider: appetizer-labs
spec:
  componentReferences:
  - name: user-service
    componentName: appetizer-labs/user-service
    version: 1.5.2
  - name: payment-service
    componentName: appetizer-labs/payment-service
    version: 2.3.1
  - name: notification-service
    componentName: appetizer-labs/notification-service
    version: 1.1.0
  resources:
  - name: platform-chart
    type: helmChart
    version: 2.1.0
    access:
      type: ociRegistry
      imageReference: ghcr.io/appetizer-labs/charts/platform:2.1.0

This creates a complete bill of delivery for your entire platform, with precise version tracking for each component.

Tracking Dependencies and Runtime Bindings

Capturing External Dependencies

Modern applications depend on external services, databases, and third-party APIs. OCM can capture these dependencies explicitly:

spec:
  references:
  - name: postgres
    componentName: external/postgresql
    version: "14.9"
    extraIdentity:
      deployment: production
  - name: redis
    componentName: external/redis
    version: "7.0"
    extraIdentity:
      deployment: production
  - name: stripe-api
    componentName: external/stripe
    version: "2023-10-16"
    extraIdentity:
      environment: production

This approach makes external dependencies visible in your supply chain, enabling better security scanning and compliance tracking.

Runtime Configuration Binding

OCM excels at capturing how components are configured for specific environments. Use labels and extra identity fields to track environment-specific bindings:

spec:
  resources:
  - name: app-config
    type: yaml
    version: 1.2.3
    access:
      type: github
      repository: github.com/appetizer-labs/configs
      ref: production/v1.2.3
    labels:
    - name: environment
      value: production
    - name: region
      value: us-west-2
    extraIdentity:
      cluster: prod-us-west-2
      namespace: my-app

This level of detail enables precise tracking of what configuration was deployed where, crucial for incident response and compliance audits.

Implementing Comprehensive Supply Chain Security

Signature Verification

OCM supports cryptographic signing of component descriptors, enabling end-to-end verification of your supply chain:

# Generate signing key
ocm create rsakeypair appetizer-labs-key

# Sign component
ocm sign componentversion \
  --signature appetizer-labs-signature \
  --private-key appetizer-labs-key.priv \
  component-descriptor.yaml

# Verify signature during deployment
ocm verify componentversion \
  --signature appetizer-labs-signature \
  --public-key appetizer-labs-key.pub \
  ghcr.io/appetizer-labs/components//my-platform:2.1.0

Integrate signature verification into your deployment pipelines to ensure only signed components reach production.

Vulnerability Scanning Integration

OCM component descriptors provide the perfect input for comprehensive vulnerability scanning. Here's how to integrate with popular scanning tools:

# Extract all container images from component
ocm download resource \
  --type ociImage \
  --output-format json \
  ghcr.io/appetizer-labs/components//my-platform:2.1.0 | \
  jq -r '.access.imageReference' | \
  xargs -I {} trivy image {}

# Scan Helm charts for misconfigurations
ocm download resource \
  --type helmChart \
  --output helm-chart.tgz \
  ghcr.io/appetizer-labs/components//my-platform:2.1.0
checkov -f helm-chart.tgz

This approach ensures you're scanning the exact artifacts that will be deployed, not just what's in your source repositories.

Policy Enforcement

Use OCM metadata to enforce deployment policies. For example, require all production components to be signed and scanned:

# admission-controller-policy.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ocm-policy
data:
  policy.rego: |
    package ocm.admission

    deny[msg] {
      input.metadata.labels.environment == "production"
      not input.metadata.annotations["ocm.software/signature"]
      msg := "Production components must be signed"
    }

    deny[msg] {
      input.metadata.labels.environment == "production"
      not input.metadata.annotations["security.scan.passed"]
      msg := "Production components must pass security scans"
    }

Monitoring and Observability for OCM Components

Runtime Correlation

One of OCM's most powerful features is the ability to correlate runtime behavior with specific component versions. Add OCM metadata to your application deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
  labels:
    ocm.software/component: appetizer-labs/user-service
    ocm.software/version: 1.5.2
    ocm.software/platform-component: appetizer-labs/my-platform
    ocm.software/platform-version: 2.1.0
spec:
  template:
    metadata:
      labels:
        ocm.software/component: appetizer-labs/user-service
        ocm.software/version: 1.5.2

This enables powerful queries in your monitoring systems:

# Alert on errors for specific component versions
rate(http_requests_total{code=~"5.."}[5m]) > 0.1
  and on(pod) 
  kube_pod_labels{label_ocm_software_component="appetizer-labs/user-service"}

Supply Chain Drift Detection

Monitor for unauthorized changes to your supply chain by comparing runtime state with OCM descriptors:

#!/bin/bash
# drift-detection.sh

# Get expected images from OCM
expected_images=$(ocm get resources \
  --type ociImage \
  ghcr.io/appetizer-labs/components//my-platform:2.1.0 | \
  jq -r '.spec.resources[].access.imageReference')

# Get actual running images
actual_images=$(kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}')

# Compare and alert on differences
diff <(echo "$expected_images" | sort) <(echo "$actual_images" | sort) || {
  echo "ALERT: Supply chain drift detected!"
  exit 1
}

Run this check regularly to catch unauthorized deployments or configuration drift.

Best Practices for Production OCM Implementation

Component Versioning Strategy

Align OCM component versions with your release strategy. For semantic versioning:

Patch versions (1.2.1 → 1.2.2): Bug fixes, security patches
Minor versions (1.2.0 → 1.3.0): New features, backward-compatible changes
Major versions (1.0.0 → 2.0.0): Breaking changes, architecture updates

Include build metadata in component descriptors:

metadata:
  labels:
  - name: build.commit
    value: a1b2c3d4
  - name: build.timestamp
    value: "2024-01-15T10:30:00Z"
  - name: build.pipeline
    value: github-actions

Repository Organization

Structure your repositories to support OCM workflows:

organization/
├── services/
│   ├── user-service/           # Individual service repos
│   ├── payment-service/
│   └── notification-service/
├── platform/
│   ├── charts/                 # Shared Helm charts
│   ├── configs/               # Environment configs
│   └── components/            # Platform component descriptors
└── infrastructure/
    ├── terraform/             # Infrastructure as code
    └── policies/              # Security and compliance policies

Automation and Integration

Automate OCM descriptor generation and validation in your CI/CD pipelines. Never manually create component descriptors – they should be generated from your build process to ensure accuracy and consistency.

Use GitOps principles with OCM by storing component descriptors in Git and using tools like ArgoCD to deploy based on OCM metadata:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-platform
spec:
  source:
    repoURL: ghcr.io/appetizer-labs/components
    targetRevision: 2.1.0
    chart: my-platform
  destination:
    server: https://kubernetes.default.svc
    namespace: production

Looking Forward: OCM in the Cloud-Native Ecosystem

The Open Component Model represents a significant step forward in supply chain security and observability. As the cloud-native ecosystem continues to evolve, OCM provides a foundation for more sophisticated supply chain management.

Key areas where OCM is expanding include integration with policy engines like Open Policy Agent, enhanced support for serverless deployments, and better tooling for multi-cloud scenarios. The specification is actively developed and backed by major cloud providers, ensuring long-term viability.

For organizations serious about supply chain security, OCM isn't just another tool – it's a comprehensive approach to understanding and controlling what gets delivered to production. Start with a pilot project, automate descriptor generation, and gradually expand coverage across your entire software portfolio. The investment in proper supply chain tracking pays dividends when you need to respond quickly to security incidents or compliance audits.

The future of software delivery is traceable, verifiable, and secure. OCM provides the foundation to build that future today.

React Performance Optimization: Profiling, Rendering, and Bundle Strategies That Scale

Matthias Bruns — Wed, 01 Apr 2026 07:43:18 +0000

React performance optimization isn't about micro-optimizations or premature optimization. It's about systematic identification and elimination of bottlenecks that actually impact user experience. When your React app starts feeling sluggish, users notice. When bundle sizes balloon, conversion rates drop. The good news? Most React performance issues follow predictable patterns, and the tooling to fix them has never been better.

Start with Profiling: Measure Before You Optimize

The React DevTools Profiler is your first stop for performance investigation. As the React team emphasizes, the Profiler "measures how often a React application renders and what the 'cost' of rendering is." This isn't guesswork—it's data.

Install React DevTools in your browser, then navigate to the Profiler tab. Hit record, interact with your app, and stop recording. You'll see a flame graph showing which components took the longest to render and how often they re-rendered.

Look for these red flags:

Components with unusually long render times
Frequent re-renders of expensive components
Deep component trees that update unnecessarily

// Use the Profiler component for programmatic measurement


function onRenderCallback(id, phase, actualDuration) {
  console.log('Component:', id);
  console.log('Phase:', phase); // "mount" or "update"
  console.log('Duration:', actualDuration);
}

function App() {
  return (

  );
}

Kent C. Dodds recommends starting with the development server and React DevTools, but don't stop there. Profile in production mode with npm run build and serve the built files. Development mode includes extra overhead that masks real performance characteristics.

Rendering Optimization: Stop Unnecessary Re-renders

The most common React performance issue isn't slow components—it's components that render too often. React's documentation states that you can "speed all of this up by overriding the lifecycle function shouldComponentUpdate, which is triggered before the re-rendering process starts."

Modern React gives us better tools than shouldComponentUpdate. Here's your optimization toolkit:

React.memo for Component Memoization

React.memo prevents re-renders when props haven't changed:

const ExpensiveComponent = React.memo(({ data, onUpdate }) => {
  // This only re-renders if data or onUpdate changes
  return (
    <div>
      {data.map(item => )}
    </div>
  );
});

// Custom comparison for complex props
const ExpensiveComponentWithCustomComparison = React.memo(
  ({ user, settings }) => {
    return ;
  },
  (prevProps, nextProps) => {
    return (
      prevProps.user.id === nextProps.user.id &&
      prevProps.settings.theme === nextProps.settings.theme
    );
  }
);

useMemo and useCallback for Value Stabilization

Stabilize expensive computations and function references:

function ProductList({ products, filters }) {
  // Expensive filtering only runs when products or filters change
  const filteredProducts = useMemo(() => {
    return products.filter(product => 
      filters.every(filter => filter.test(product))
    );
  }, [products, filters]);

  // Stable function reference prevents child re-renders
  const handleProductClick = useCallback((productId) => {
    analytics.track('product_clicked', { productId });
    navigate(`/products/${productId}`);
  }, [navigate]);

  return (
    <div>
      {filteredProducts.map(product => (

      ))}
    </div>
  );
}

State Structure Optimization

Poor state structure causes cascading re-renders. Flatten state and colocate updates:

// Bad: Nested state causes entire component tree to re-render
const [appState, setAppState] = useState({
  user: { name: '', email: '', preferences: {} },
  ui: { sidebar: false, theme: 'light' },
  data: { products: [], orders: [] }
});

// Good: Separate concerns, minimize re-render scope
const [user, setUser] = useState({ name: '', email: '' });
const [preferences, setPreferences] = useState({});
const [uiState, setUiState] = useState({ sidebar: false, theme: 'light' });
const [products, setProducts] = useState([]);

Bundle Splitting Strategies That Scale

Large bundles kill performance, especially on mobile networks. Modern React applications need intelligent code splitting strategies.

Route-Based Code Splitting

Start with route-level splits using React.lazy:




// Lazy load route components
const Home = lazy(() => import('./pages/Home'));
const Dashboard = lazy(() => import('./pages/Dashboard'));
const Analytics = lazy(() => import('./pages/Analytics'));

function App() {
  return (

      </Suspense>
    </BrowserRouter>
  );
}

Component-Based Code Splitting

Split heavy components that aren't always needed:



const HeavyChart = lazy(() => import('./HeavyChart'));
const DataTable = lazy(() => import('./DataTable'));

function Dashboard({ data }) {
  const [view, setView] = useState('summary');

  return (
    <div>


      }>
        {view === 'chart' && }
        {view === 'table' && }
      </Suspense>
    </div>
  );
}

Library Code Splitting

Split vendor libraries strategically:

// utils/dynamicImports.js
export const loadChartLibrary = () => import('chart.js');
export const loadDateLibrary = () => import('date-fns');

// components/Chart.jsx



function Chart({ data }) {
  const [ChartJS, setChartJS] = useState(null);

  useEffect(() => {
    loadChartLibrary().then(chartLib => {
      setChartJS(() => chartLib.Chart);
    });
  }, []);

  if (!ChartJS) return ;

  return ;
}

Advanced Optimization Patterns

Virtual Scrolling for Large Lists

Don't render thousands of DOM nodes. Use virtual scrolling:



function VirtualizedProductList({ products }) {
  const Row = ({ index, style }) => (
    <div style={style}>

    </div>
  );

  return (

  );
}

Debounced Input Handling

Prevent excessive API calls and re-renders:




function SearchInput({ onSearch }) {
  const [value, setValue] = useState('');

  const debouncedSearch = useCallback(
    debounce((searchTerm) => {
      onSearch(searchTerm);
    }, 300),
    [onSearch]
  );

  useEffect(() => {
    debouncedSearch(value);
  }, [value, debouncedSearch]);

  return (
    <input
      type="text"
      value={value}
      onChange={(e) => setValue(e.target.value)}
      placeholder="Search products..."
    />
  );
}

Web Workers for Heavy Computations

Move expensive operations off the main thread:

// workers/dataProcessor.js
self.onmessage = function(e) {
  const { data, operation } = e.data;

  let result;
  switch (operation) {
    case 'filter':
      result = data.filter(item => item.active);
      break;
    case 'sort':
      result = data.sort((a, b) => b.score - a.score);
      break;
  }

  self.postMessage(result);
};

// hooks/useWorker.js


export function useWorker(workerPath) {
  const [worker, setWorker] = useState(null);

  useEffect(() => {
    const w = new Worker(workerPath);
    setWorker(w);

    return () => w.terminate();
  }, [workerPath]);

  const runTask = (data, operation) => {
    return new Promise((resolve) => {
      worker.onmessage = (e) => resolve(e.data);
      worker.postMessage({ data, operation });
    });
  };

  return runTask;
}

Production Monitoring and Continuous Optimization

Performance optimization isn't a one-time task. Set up monitoring to catch regressions:

Bundle Analysis

Add bundle analysis to your build process:

{
  "scripts": {
    "analyze": "npm run build && npx webpack-bundle-analyzer build/static/js/*.js"
  }
}

Performance Budgets

Set performance budgets in your build configuration:

// webpack.config.js
module.exports = {
  performance: {
    maxAssetSize: 250000,
    maxEntrypointSize: 250000,
    hints: 'error'
  }
};

Real User Monitoring

Track Core Web Vitals in production:



function sendToAnalytics(metric) {
  // Send to your analytics service
  analytics.track('web_vital', {
    name: metric.name,
    value: metric.value,
    id: metric.id
  });
}

// Measure all Core Web Vitals
getCLS(sendToAnalytics);
getFID(sendToAnalytics);
getFCP(sendToAnalytics);
getLCP(sendToAnalytics);
getTTFB(sendToAnalytics);

The Performance Optimization Mindset

As discussed in the React community, "sometimes performance issues are just architecture issues." The most effective optimizations often involve rethinking component structure, state management, and data flow rather than micro-optimizing individual components.

Focus on these high-impact areas:

Eliminate unnecessary re-renders through proper memoization
Reduce bundle size with strategic code splitting
Optimize critical rendering path by loading essential code first
Monitor performance continuously to catch regressions early

Remember: React's performance optimization involves "a combination of strategies, from the fundamental understanding of React's diffing algorithm to leveraging built-in features and third-party tools." Start with profiling, fix the biggest bottlenecks first, and always measure the impact of your changes.

Performance optimization is an iterative process. Profile, optimize, measure, repeat. Your users will notice the difference, and your conversion metrics will thank you.

TypeScript Testing Patterns: Unit, Integration, and E2E Strategies That Scale

Matthias Bruns — Tue, 31 Mar 2026 07:44:08 +0000

TypeScript's type system catches many bugs at compile time, but that doesn't eliminate the need for comprehensive testing. In fact, TypeScript applications require a nuanced testing strategy that leverages both the language's static typing benefits and traditional testing practices to ensure code quality at scale.

The challenge isn't just writing tests—it's building a testing architecture that grows with your codebase without becoming a maintenance nightmare. This guide covers practical patterns for unit, integration, and end-to-end testing that work in real production environments.

Why TypeScript Testing Is Different

TypeScript unit testing differs from regular JavaScript testing in fundamental ways. The type system eliminates entire classes of runtime errors, which means you can focus your testing efforts on business logic rather than basic type mismatches.

However, this creates new challenges. You need test configurations that work with TypeScript's compilation process, and you must decide how much to rely on types versus runtime validation in your test assertions.

The payoff is significant: fewer tests overall, but higher-quality tests that focus on actual business value rather than catching trivial errors.

Setting Up Your TypeScript Testing Foundation

Compiler Configuration for Tests

Your testing setup needs a TypeScript configuration that balances compilation speed with debugging capabilities. Here's a proven tsconfig.json configuration for testing:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020", "DOM"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": false,
    "sourceMap": true,
    "outDir": "./dist"
  },
  "include": ["src/**/*", "tests/**/*"],
  "exclude": ["node_modules", "dist"]
}

Avoid using the outfile option in your test configuration—it breaks test discovery in most IDEs.

Framework Selection Strategy

The TypeScript testing ecosystem offers several mature options. Jest remains the most popular choice, but Vitest is gaining traction for its native TypeScript support and faster execution.

For Jest with TypeScript, use this configuration:

// jest.config.js
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node', // or 'jsdom' for frontend
  testMatch: ['**/__tests__/**/*.ts', '**/?(*.)+(spec|test).ts'],
  collectCoverageFrom: [
    'src/**/*.ts',
    '!src/**/*.d.ts',
    '!src/**/*.interface.ts'
  ],
  coverageThreshold: {
    global: {
      branches: 80,
      functions: 80,
      lines: 80,
      statements: 80
    }
  }
};

This configuration ensures ts-jest processes your TypeScript files correctly and maintains proper source map support for debugging.

Unit Testing Patterns That Scale

The Arrange-Act-Assert Pattern

The AAA pattern creates maintainable unit tests by organizing test code into three distinct sections. Here's how it works with TypeScript:

// userService.test.ts



describe('UserService', () => {
  let userService: UserService;

  beforeEach(() => {
    userService = new UserService();
  });

  it('should create user with valid data', () => {
    // Arrange
    const userData: Omit<User, 'id'> = {
      email: 'test@example.com',
      name: 'Test User',
      role: 'user'
    };

    // Act
    const result = userService.createUser(userData);

    // Assert
    expect(result).toHaveProperty('id');
    expect(result.email).toBe(userData.email);
    expect(result.name).toBe(userData.name);
    expect(result.role).toBe(userData.role);
  });
});

Test Data Management with the Prototype Pattern

The Prototype pattern helps manage test data efficiently by allowing you to clone and modify test objects:

// testDataFactory.ts
export class TestDataFactory {
  private static baseUser: User = {
    id: '1',
    email: 'default@example.com',
    name: 'Default User',
    role: 'user',
    createdAt: new Date('2024-01-01')
  };

  static createUser(overrides: Partial<User> = {}): User {
    return {
      ...this.baseUser,
      ...overrides,
      id: overrides.id || Math.random().toString(36)
    };
  }
}

// Usage in tests
const adminUser = TestDataFactory.createUser({ 
  role: 'admin', 
  email: 'admin@example.com' 
});

Reducing Mock Complexity

Minimize mocks to avoid brittle tests. Instead of mocking every dependency, use dependency injection and test doubles:

// Instead of heavy mocking
interface EmailService {
  sendEmail(to: string, subject: string, body: string): Promise<void>;
}

class MockEmailService implements EmailService {
  public sentEmails: Array<{to: string, subject: string, body: string}> = [];

  async sendEmail(to: string, subject: string, body: string): Promise<void> {
    this.sentEmails.push({ to, subject, body });
  }
}

// Clean test without complex Jest mocks
it('should send welcome email on user creation', async () => {
  const mockEmailService = new MockEmailService();
  const userService = new UserService(mockEmailService);

  await userService.createUser({
    email: 'new@example.com',
    name: 'New User'
  });

  expect(mockEmailService.sentEmails).toHaveLength(1);
  expect(mockEmailService.sentEmails[0].to).toBe('new@example.com');
});

Integration Testing Strategies

Integration tests verify that your application components work together correctly. In TypeScript applications, these tests often focus on API endpoints, database interactions, and service integrations.

Database Integration Testing

// userRepository.integration.test.ts



describe('UserRepository Integration', () => {
  let repository: UserRepository;
  let testDb: TestDatabase;

  beforeAll(async () => {
    testDb = new TestDatabase();
    await testDb.setup();
    repository = new UserRepository(testDb.connection);
  });

  afterAll(async () => {
    await testDb.teardown();
  });

  beforeEach(async () => {
    await testDb.clear();
  });

  it('should persist and retrieve user correctly', async () => {
    const userData = {
      email: 'integration@example.com',
      name: 'Integration Test User'
    };

    const savedUser = await repository.save(userData);
    const retrievedUser = await repository.findById(savedUser.id);

    expect(retrievedUser).toBeDefined();
    expect(retrievedUser!.email).toBe(userData.email);
    expect(retrievedUser!.createdAt).toBeInstanceOf(Date);
  });
});

API Integration Testing

// userApi.integration.test.ts




describe('User API Integration', () => {
  let testDb: TestDatabase;

  beforeAll(async () => {
    testDb = new TestDatabase();
    await testDb.setup();
  });

  afterAll(async () => {
    await testDb.teardown();
  });

  it('should create user via POST /users', async () => {
    const userData = {
      email: 'api@example.com',
      name: 'API Test User'
    };

    const response = await request(app)
      .post('/users')
      .send(userData)
      .expect(201);

    expect(response.body).toHaveProperty('id');
    expect(response.body.email).toBe(userData.email);
    expect(response.body.name).toBe(userData.name);
  });
});

End-to-End Testing Frameworks and Patterns

Framework Selection for E2E Testing

The E2E testing landscape offers several mature options. Playwright has emerged as the leading choice for TypeScript applications due to its native TypeScript support and comprehensive browser coverage.

// playwright.config.ts


export default defineConfig({
  testDir: './e2e',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',
  use: {
    baseURL: 'http://localhost:3000',
    trace: 'on-first-retry',
  },
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
  ],
  webServer: {
    command: 'npm run start',
    port: 3000,
  },
});

Page Object Pattern for Maintainable E2E Tests

// pages/LoginPage.ts


export class LoginPage {
  readonly page: Page;
  readonly emailInput: Locator;
  readonly passwordInput: Locator;
  readonly loginButton: Locator;
  readonly errorMessage: Locator;

  constructor(page: Page) {
    this.page = page;
    this.emailInput = page.getByTestId('email-input');
    this.passwordInput = page.getByTestId('password-input');
    this.loginButton = page.getByRole('button', { name: 'Login' });
    this.errorMessage = page.getByTestId('error-message');
  }

  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.loginButton.click();
  }

  async expectErrorMessage(message: string) {
    await expect(this.errorMessage).toHaveText(message);
  }
}

// tests/login.e2e.test.ts



test.describe('User Authentication', () => {
  test('should display error for invalid credentials', async ({ page }) => {
    const loginPage = new LoginPage(page);

    await page.goto('/login');
    await loginPage.login('invalid@example.com', 'wrongpassword');
    await loginPage.expectErrorMessage('Invalid email or password');
  });
});

Test Organization and Structure

Folder Structure That Scales

Good test structure requires clear naming conventions and logical organization:

src/
├── components/
│   ├── UserCard.tsx
│   └── __tests__/
│       └── UserCard.test.tsx
├── services/
│   ├── UserService.ts
│   └── __tests__/
│       ├── UserService.test.ts
│       └── UserService.integration.test.ts
├── utils/
│   ├── validation.ts
│   └── __tests__/
│       └── validation.test.ts
└── test-utils/
    ├── TestDatabase.ts
    ├── TestDataFactory.ts
    └── setupTests.ts

e2e/
├── pages/
│   ├── LoginPage.ts
│   └── DashboardPage.ts
├── fixtures/
│   └── testData.ts
└── tests/
    ├── authentication.e2e.test.ts
    └── userManagement.e2e.test.ts

Test Naming Conventions

Use descriptive test names that explain the scenario and expected outcome:

describe('UserService', () => {
  describe('createUser', () => {
    it('should create user with generated ID when valid data provided', () => {
      // Test implementation
    });

    it('should throw ValidationError when email format is invalid', () => {
      // Test implementation
    });

    it('should throw ConflictError when email already exists', () => {
      // Test implementation
    });
  });
});

Performance and Debugging Strategies

IDE Integration

Modern IDEs provide excellent TypeScript testing support. IntelliJ IDEA and VS Code both offer built-in test runners that work with ts-node, allowing you to run and debug tests without compilation.

For VS Code, add this configuration to your .vscode/launch.json:

{
  "type": "node",
  "request": "launch",
  "name": "Debug Jest Tests",
  "program": "${workspaceFolder}/node_modules/.bin/jest",
  "args": ["--runInBand"],
  "console": "integratedTerminal",
  "internalConsoleOptions": "neverOpen"
}

Test Performance Optimization

Use parallel execution for faster test runs:

// jest.config.js
module.exports = {
  preset: 'ts-jest',
  maxWorkers: '50%', // Use half of available CPU cores
  testTimeout: 10000,
  setupFilesAfterEnv: ['<rootDir>/src/test-utils/setupTests.ts'],
  globalSetup: '<rootDir>/src/test-utils/globalSetup.ts',
  globalTeardown: '<rootDir>/src/test-utils/globalTeardown.ts'
};

CI/CD Integration Patterns

GitHub Actions Configuration

# .github/workflows/test.yml
name: Test Suite

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:13
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Type check
        run: npm run type-check

      - name: Run unit tests
        run: npm run test:unit

      - name: Run integration tests
        run: npm run test:integration
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test

      - name: Run E2E tests
        run: npm run test:e2e

      - name: Upload coverage reports
        uses: codecov/codecov-action@v3

Building a Testing Strategy That Scales

The key to scalable TypeScript testing is layering your test types strategically. Unit tests should cover your business logic and utility functions. Integration tests should verify that your services work together correctly. E2E tests should validate critical user journeys.

Start with a solid foundation of unit tests, add integration tests for complex interactions, and use E2E tests sparingly for the most important user flows. This approach gives you confidence in your code without creating a maintenance burden.

Remember that TypeScript's type system is your first line of defense against bugs. Use it to eliminate entire classes of tests, then focus your testing efforts on the logic that actually matters to your users.

The testing patterns outlined here work in production environments because they balance comprehensive coverage with maintainability. Implement them incrementally, and you'll build a testing suite that grows with your application rather than holding it back.