DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Chaos Testing a Spring Stack: Inject Faults to Prove Resilience, Not Just Claim It

#devops #springboot #testing #java

Load testing proves a service is fast enough. Neither that nor "it deploys" answers the question on-call actually loses sleep over: what happens when a dependency breaks? A stack accumulates resilience primitives — a circuit breaker, retry/timeout/bulkhead, a dead-letter topic, a fail-fast producer — but "the breaker will save us" is a claim in a design doc until you break the thing and watch. Chaos testing turns each claim into a controlled experiment and, better, into an assertion in a test that gates every push.

A chaos experiment is controlled, not random

This isn't "break things in prod". Each experiment has three parts: a steady-state hypothesis (a measurable property that must stay true even during the fault), the smallest blast radius that tests it, and a recovery you assert. OrderHub ships four faults that map 1:1 to JVM fault-injection tests (the CI half) and Chaos Mesh manifests (the game-day half):

  • Dead downstream → the circuit breaker OPENs, fast-fails to a degraded fallback, then recovers CLOSED.
  • Injected latency → a @TimeLimiter cuts off the slow call and a @Bulkhead rejects overflow, so the caller is never blocked for the full latency.
  • Unreachable broker → the producer fails fast (max.block.ms=2s), the order still stands, and publishing resumes when the broker returns.
  • Poison message → retried a bounded number of times, then routed to a .DLT topic — never dropped, never an infinite loop.

The CI half: fault-injection tests that gate the build

The circuit-breaker experiment drives the real Resilience4j state machine with a tiny window so recovery is observable in milliseconds. It doesn't stop at "it opened" — it asserts graceful degradation and a clean recovery.

client.setFailureRate(1.0);                 // 💥 downstream is DOWN
for (int i = 0; i < 5; i++) guarded(breaker, client, "Keyboard");
assertThat(breaker.getState()).isEqualTo(OPEN);   // tripped

client.resetCallCount();                     // steady state during the fault:
for (int i = 0; i < 5; i++) {
    InventoryStatus s = guarded(breaker, client, "Keyboard");
    assertThat(s.degraded()).isTrue();       // graceful fallback, never a 500
}
assertThat(client.callCount()).isZero();     // OPEN -> the dead dep is never called

client.setFailureRate(0.0);                  // 🩹 downstream recovers
Thread.sleep(400);                           // > waitDurationInOpenState
guarded(breaker, client, "Keyboard");        // HALF_OPEN trial calls succeed
guarded(breaker, client, "Keyboard");
assertThat(breaker.getState()).isEqualTo(CLOSED);  // recovered
Enter fullscreen mode Exit fullscreen mode

The poison-message test builds the recovery policy explicitly and proves the retries are bounded — the failure mode to avoid is a record retried forever, blocking its partition:

DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 2L));
// ... a listener that ALWAYS throws for the poison record
template.send("chaos-poison", "ORD-POISON", "{...POISON-SKU...}");

ConsumerRecord<..> dead = getSingleRecord(dltConsumer, "chaos-poison.DLT", ofSeconds(20));
assertThat(dead.key()).isEqualTo("ORD-POISON");   // routed to .DLT, key preserved
assertThat(attempts.get()).isEqualTo(3);          // 1 initial + 2 retries — BOUNDED
Enter fullscreen mode Exit fullscreen mode

The game-day half: chaos as code

The same hypotheses become Chaos Mesh manifests you run against a real cluster. Where the JVM test proves the breaker logic, a PodChaos proves it against real pod rescheduling and kubelet timing — things an in-process test can't model. The manifest makes the blast radius explicit: one random pod, for a bounded duration, auto-reverted.

kind: PodChaos
spec:
  action: pod-kill
  mode: one                 # blast radius: ONE random pod, not the Deployment
  selector:
    labelSelectors: { app: inventory-service }
  gracePeriod: 0            # hard kill — the worst case for the caller
  duration: 60s             # bounded: Chaos Mesh reverts automatically
Enter fullscreen mode Exit fullscreen mode

Config rots, so even the chaos is tested: a ChaosManifestTest parses every manifest with SnakeYAML and asserts each is a well-formed, bounded, scoped experiment that documents a steady-state hypothesis. Drop a duration (an experiment that never reverts is dangerous), widen a selector, or delete the hypothesis comment and the build goes red.

Chaos in CI catches regressions on every push; the game-day catches what only real infrastructure can. Together, resilience stops being a claim and becomes evidence — 5 fault-injection tests plus 4 manifest-shape tests, mvn -B clean test → BUILD SUCCESS.

Pick a fault, press inject, and watch the steady-state hypothesis hold and then recover: https://dev48v.infy.uk/orderhub/day48-chaos-testing.html

Top comments (0)