The latest articles on DEV Community by Tullio Sebastiani (@tsebastiani). https://dev.to/tsebastiani https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4082801%2Fc1ae7dea-348a-4e80-81ef-ab2c19f409cb.jpg https://dev.to/tsebastiani en Tullio Sebastiani Tue, 18 Aug 2026 07:59:30 +0000 https://dev.to/tsebastiani/scaling-chaos-engineering-across-kubernetes-clusters-with-krkn-operator-1lmb https://dev.to/tsebastiani/scaling-chaos-engineering-across-kubernetes-clusters-with-krkn-operator-1lmb <p>Running a chaos experiment against a Kubernetes cluster is relatively straightforward.</p> <p>But what happens when you want to apply the same resilience validation consistently across a fleet of Kubernetes and OpenShift clusters?</p> <p>This is one of the problems we've been working on with <strong>Krkn Operator</strong>.</p> <p>I'm one of the maintainers of <strong>Krkn</strong>, a CNCF Sandbox project for chaos engineering. As Krkn evolved, we found that executing individual failure scenarios was only part of the problem.</p> <p>At multi-cluster scale, different questions start to emerge:</p> <ul> <li>How do you centrally manage target clusters and access?</li> <li>How do you compose repeatable experiments instead of running isolated scenarios?</li> <li>How do you execute the same resilience validation across multiple clusters?</li> <li>How do you compare system behavior before and after an upgrade, configuration change, or release?</li> </ul> <p>We built <strong>Krkn Operator</strong> around the Krkn core to address these problems.</p> <h2> A control plane for chaos engineering </h2> <p>Krkn Operator provides a Kubernetes-native control plane for orchestrating chaos experiments across multiple Kubernetes and OpenShift clusters.</p> <p>Instead of treating every experiment as an isolated execution, the goal is to make resilience testing something that can be defined, repeated, and evaluated consistently.</p> <p>One of the components we've built around this idea is <strong>Chaos Studio</strong>.</p> <p>Chaos Studio provides a visual workflow editor where multiple chaos scenarios and checks can be composed into reusable workflows.</p> <p>Scenarios can run sequentially or in parallel, allowing more complex failure conditions to be represented without turning the experiment itself into a collection of scripts.</p> <h2> From experiments to resilience validation </h2> <p>Injecting a failure is useful.</p> <p>Understanding how the system behaved during that failure is much more useful.</p> <p>Krkn Operator can combine experiments with observability data and <strong>resiliency scoring</strong>, allowing teams to establish a resilience baseline and compare results across different executions.</p> <p>This opens up some interesting use cases around CI/CD and infrastructure lifecycle management.</p> <p>For example, the same resilience workflow could be executed before and after:</p> <ul> <li>a Kubernetes upgrade</li> <li>an application release</li> <li>an infrastructure change</li> <li>a configuration change</li> </ul> <p>Instead of simply asking <em>"did the chaos experiment run?"</em>, we can start asking <em>"did the resilience of the system change?"</em></p> <h2> Multi-cluster execution with OCM and Red Hat ACM </h2> <p>Another area we've been focusing on is multi-cluster orchestration.</p> <p>Krkn Operator integrates with <strong>Open Cluster Management (OCM) and Red Hat Advanced Cluster Management (ACM)</strong>.</p> <p>Managed clusters can be discovered through the management hub and made available as targets for chaos experiments.</p> <p>This means an existing multi-cluster management architecture can also become the central point from which resilience testing is orchestrated.</p> <p>For organizations operating fleets of Kubernetes or OpenShift clusters, this removes the need to independently configure every target before running resilience experiments.</p> <h2> Everything is open source </h2> <p>Krkn Operator is <strong>Apache 2.0 licensed and fully open source</strong>, including the operator, the web console, and the OCM/ACM integration.</p> <p>If you're interested in the implementation, you can explore the project here:</p> <p>👉 <strong>GitHub:</strong> <a href="https://github.com/krkn-chaos/krkn-operator" rel="noopener noreferrer">https://github.com/krkn-chaos/krkn-operator</a></p> <p>We've also published a more detailed technical walkthrough on Red Hat Developer covering the architecture, Chaos Studio, resiliency scoring, multi-cluster execution, and the ACM integration:</p> <p>👉 <strong>Technical deep dive:</strong> <a href="https://krkn-chaos.dev/go/1.0/devto" rel="noopener noreferrer">https://krkn-chaos.dev/go/1.0/devto</a></p> <p>I'd be particularly interested in feedback from people working with Kubernetes at scale:</p> <p><strong>How are you currently integrating chaos or resilience testing into your infrastructure lifecycle or CI/CD pipelines?</strong></p> kubernetes devops opensource sre