Running a chaos experiment against a Kubernetes cluster is relatively straightforward.
But what happens when you want to apply the same resilience validation consistently across a fleet of Kubernetes and OpenShift clusters?
This is one of the problems we've been working on with Krkn Operator.
I'm one of the maintainers of Krkn, a CNCF Sandbox project for chaos engineering. As Krkn evolved, we found that executing individual failure scenarios was only part of the problem.
At multi-cluster scale, different questions start to emerge:
- How do you centrally manage target clusters and access?
- How do you compose repeatable experiments instead of running isolated scenarios?
- How do you execute the same resilience validation across multiple clusters?
- How do you compare system behavior before and after an upgrade, configuration change, or release?
We built Krkn Operator around the Krkn core to address these problems.
A control plane for chaos engineering
Krkn Operator provides a Kubernetes-native control plane for orchestrating chaos experiments across multiple Kubernetes and OpenShift clusters.
Instead of treating every experiment as an isolated execution, the goal is to make resilience testing something that can be defined, repeated, and evaluated consistently.
One of the components we've built around this idea is Chaos Studio.
Chaos Studio provides a visual workflow editor where multiple chaos scenarios and checks can be composed into reusable workflows.
Scenarios can run sequentially or in parallel, allowing more complex failure conditions to be represented without turning the experiment itself into a collection of scripts.
From experiments to resilience validation
Injecting a failure is useful.
Understanding how the system behaved during that failure is much more useful.
Krkn Operator can combine experiments with observability data and resiliency scoring, allowing teams to establish a resilience baseline and compare results across different executions.
This opens up some interesting use cases around CI/CD and infrastructure lifecycle management.
For example, the same resilience workflow could be executed before and after:
- a Kubernetes upgrade
- an application release
- an infrastructure change
- a configuration change
Instead of simply asking "did the chaos experiment run?", we can start asking "did the resilience of the system change?"
Multi-cluster execution with OCM and Red Hat ACM
Another area we've been focusing on is multi-cluster orchestration.
Krkn Operator integrates with Open Cluster Management (OCM) and Red Hat Advanced Cluster Management (ACM).
Managed clusters can be discovered through the management hub and made available as targets for chaos experiments.
This means an existing multi-cluster management architecture can also become the central point from which resilience testing is orchestrated.
For organizations operating fleets of Kubernetes or OpenShift clusters, this removes the need to independently configure every target before running resilience experiments.
Everything is open source
Krkn Operator is Apache 2.0 licensed and fully open source, including the operator, the web console, and the OCM/ACM integration.
If you're interested in the implementation, you can explore the project here:
👉 GitHub: https://github.com/krkn-chaos/krkn-operator
We've also published a more detailed technical walkthrough on Red Hat Developer covering the architecture, Chaos Studio, resiliency scoring, multi-cluster execution, and the ACM integration:
👉 Technical deep dive: https://krkn-chaos.dev/go/1.0/devto
I'd be particularly interested in feedback from people working with Kubernetes at scale:
How are you currently integrating chaos or resilience testing into your infrastructure lifecycle or CI/CD pipelines?
Top comments (0)