After building ledger-sentinel, a payment system focused on correctness under concurrent load, I wanted to go one level deeper: not just "does the system behave correctly," but "does the system recover when something actually breaks."
That's the premise behind fault-sentinel, a lightweight Kubernetes-native CLI I built in Go to deliberately inject failures (killing pods, adding network latency, generating CPU load) and observe what actually happens.
Why build your own tool instead of just using an existing one?
There are established chaos engineering tools (Chaos Mesh, LitmusChaos) that do this more comprehensively. I built fault-sentinel anyway for a specific reason: I wanted to understand the mechanics from the inside, not just run someone else's custom resources.
Writing the pod-eviction logic myself using k8s.io/client-go means I actually know what a graceful deletion versus a forced deletion does at the API level, rather than just knowing a CRD exists for it.
Note: This is an experimental learning project built to explore chaos injection concepts and Go platform engineering principles, not a production tool.
The Three Experiments
-
Pod Termination (
kill-pod): Selects a pod by label selector and deletes it to verify whether the deployment'sReplicaSetcontroller notices and replaces it, and how fast reconciliation occurs.
Network Latency (
network-delay): Uses Linux Traffic Control (tc netem) inside the target container's network namespace to inject artificial packet delay, automatically triggering a cleanup after the experiment window.CPU Stress (
stress-cpu): Spins up compute load across a configurable number of cores for a set duration to observe how the workload performs under compute starvation.
Each experiment executes through a Cobra-based CLI:
# Terminate a pod matching a label
./bin/fault-cli kill-pod --namespace default --label-selector app=target-app
# Inject 200ms network delay for 30 seconds
./bin/fault-cli network-delay --namespace default --label-selector app=target-app --delay 200ms --duration 30s
# Apply CPU stress across 2 cores for 30 seconds
./bin/fault-cli stress-cpu --duration 30s --cores 2
The design decision I spent the most time on: how much access does this tool actually need?
A chaos tool is, by definition, something that deletes pods and manipulates network behavior in a cluster. That's a meaningful amount of trust to grant a piece of software, so I wanted the access model to be as narrow as it could be.
Two things came out of that:
-
RBAC scoping:
kill-podonly needslistanddeletepermissions on pods in the target namespace — not cluster-wide access, not access to secrets or other resources. -
No persistent daemon: fault-sentinel doesn't run as a long-lived agent inside the cluster. It uses standard
client-gocalls and SPDY exec connections for the duration of an experiment, then it's done. There's no always-on process with standing permissions waiting to be exploited, the tool is only as dangerous as the moment you're actively running it.
The trade-off here is availability, not just access: a tool with no daemon can't schedule automatic recurring chaos experiments the way Chaos Mesh can. For a learning project focused on understanding failure mechanics rather than running a continuous chaos program, that trade-off made sense.
What the network-delay injection actually required
This was the most fiddly of the three experiments to get right. Injecting latency via tc netem requires the NET_ADMIN Linux capability inside the target container — without it, the tc command inside the pod simply fails with a permissions error. That means the target application's pod spec needs NET_ADMIN added to its securityContext, which is itself a decision worth thinking about: you're granting the target application's container a capability it doesn't normally need, purely so it can be a valid subject for chaos testing.
That's a real trade-off, not a footnote, running chaos experiments against a workload means loosening that workload's security posture slightly, at least for network-based experiments. In a real environment, you'd want a dedicated, tightly scoped service account and probably a separate sidecar with the capability, rather than granting it to the application container directly.
Telemetry: making sure the experiment itself is observable
An experiment you can't measure isn't really an experiment. fault-sentinel exposes Prometheus metrics on :8080/metrics during every run:
-
chaos_experiments_total{experiment_type, status}— how many experiments ran, and whether they completed or errored -
chaos_injected_faults_total{target_pod, fault_type}— which specific pods were hit, and with what -
chaos_experiment_duration_seconds{experiment_type}— how long each experiment actually took
Scraping this mid-experiment:
curl -s http://localhost:8080/metrics | grep chaos_
The
chaos_injected_faults_totalmetric tracks the cumulative count of successful fault injections labeled byfault_typeand targettarget_pod. In the screenshot above, the counter value of 1 reflects a single active 250ms latency experiment targeting podpayment-api-667fdccc65-gwmfl. Successive runs against other pods or repeated injections will increment this value per label combination across the lifetime of the telemetry server instance.
What I actually verified, not just designed
The CI pipeline (GitHub Actions) runs golangci-lint, unit tests with Go's race detector enabled (go test -race), builds a multi-stage Docker image, and runs a Trivy vulnerability scan on the built image for critical/high severities before anything is considered done. The race detector matters specifically here, a chaos tool that has its own concurrency bugs while testing other systems' concurrency behavior would be a bad joke.
What this project didn't try to be
This isn't a replacement for Chaos Mesh or Litmus, and I'm not claiming it should be used against a real production cluster. It doesn't have scheduled/recurring experiments, it doesn't have a web UI, and it doesn't handle multi-cluster orchestration. What it does have is a clear, small surface area that I understand completely, because I built every part of it... which was the actual point.
The lesson that stuck with me most: recovery isn't binary. A pod coming back after being killed isn't the same as the system recovering, you have to actually watch what happens to in-flight requests, latency, and error rates during the gap, not just confirm the pod count went back to normal. That's the difference between "it healed" and "it healed in a way that didn't hurt anyone using it."
Top comments (0)