DEV Community: Oriol Moura

Surviving the Chaos: Injecting Faults with the ChaosEngineeringMasterDeck

Oriol Moura — Thu, 22 Oct 2026 13:24:57 +0000

Abstract: How we turned "will it hold up?" into testable hypotheses using a deck of chaos experiments (the ChaosEngineeringMasterDeck), and what we uncovered when degrading the latency of a payment microservice in a real-world environment.

Why Break Things on Purpose In distributed systems, partial failure is the normal state, not an exception. With dozens of services, each with 99.9% availability, the composite availability of a request traversing 10 of them in series is approximately $0.999^{10} \approx 99.0\%$. That represents roughly 87 hours of downtime per year that no one anticipated by evaluating services individually. Unit and integration tests verify what code does when everything works; they reveal nothing about system behavior when the network slows down, a node disappears, or a downstream dependency stalls. Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions. Formalized in the Principles of Chaos Engineering (born out of Netflix's experience with Chaos Monkey), it follows the empirical scientific cycle:
Define steady state via observable business or system metrics (not just "the server is up").
Formulate a hypothesis: the steady state will persist despite fault $X$.
Inject realistic faults reflecting real-world events (latency, packet loss, node or availability zone outages).
Attempt to refute the hypothesis by contrasting control and experimental groups.
Contain the blast radius and abort immediately if safety thresholds are breached. The Mindset Shift The traditional approach focuses on preventing failures (maximizing MTBF). The resilience approach assumes failures are inevitable and optimizes for recovery time and graceful degradation (minimizing MTTR and user impact). In practice, this reframes the question from "How do we prevent the payment service from going down?" to "What does the user experience when it goes down, and how quickly do we recover?" Regarding staging versus production: mature implementations target production, as only production reflects real traffic, authentic data, and live configurations. However, the correct progression is staging first, followed by production with a minimal blast radius. An experiment should only transition environments once automated rollback and abort triggers are thoroughly validated.
Dissecting the ChaosEngineeringMasterDeck In our context, the Master Deck is a version-controlled catalog of chaos experiments modeled as playing cards, maintained in Git alongside application code. The card deck analogy serves a practical operational purpose: each card represents a self-contained unit that can be drawn during a Game Day, requiring mandatory fields. If any required field is empty, the card cannot be played.

Why This Format?
The value lies not in the novelty of the "deck" metaphor, but in enforcing explicit hypotheses and strict abort criteria. The most common pitfall in chaos experiments is executing without a hypothesis: injecting a fault, observing arbitrary behavior, and drawing unfounded conclusions. The deck enforces scientific rigor upfront. Stored in Git, each card undergoes code review just like production changes.
Example Card in YAML

id: REDE-03
titulo: Latência alta no serviço de pagamento
naipe: Rede
hipotese: >
  Com 2000 ms de latência adicional em payment-service para 5% do tráfego,
  o checkout mantém p99 < 800 ms e taxa de erro < 1%.
variaveis:
  independentes: { delay_ms: 2000, trafego_afetado: "5%", duracao: "10m" }
  dependentes: [p99_checkout, erro_5xx, threads_ativas, estado_circuit_breaker]
metricas_controle:
  p99_checkout_ms: { max: 800 }
  erro_5xx_pct: { max: 1.0 }
raio_explosao: { escopo: "namespace staging-canary", trafego: "5%", duracao_max: "10m" }
aborto:
  - "erro_5xx_pct > 3 por 60s"
  - "conversao_checkout cai > 2 p.p."
padroes_esperados: [circuit-breaker, timeout, fallback]

Case Study: Card NET-03 Architecture Under Test An e-commerce platform running on AWS with the following checkout flow: • API Gateway $\rightarrow$ order-service (Spring Boot 3, Java 21) $\rightarrow$ payment-service (Spring Boot) $\rightarrow$ External Payment Provider. • order-service persists state to Amazon RDS (PostgreSQL, Multi-AZ). • Workloads run on Amazon EKS with Horizontal Pod Autoscaler (HPA), distributed across three Availability Zones. • Observability powered by Micrometer + Prometheus + Grafana, with distributed tracing via OpenTelemetry. The critical failure domain is the synchronous invocation order-service $\rightarrow$ payment-service. It is a textbook candidate for cascading failure: if payment stalls, worker threads in order-service become blocked waiting for responses, starving the connection and thread pools until an otherwise healthy upstream service stops serving traffic entirely. Hypothesis With an additional 2000 ms of latency injected into 5% of traffic destined for payment-service, checkout maintains p99 < 800 ms and 5xx errors < 1%, because the circuit breaker will trip open and the fallback mechanism will enqueue the transaction for asynchronous processing. Fault Injection Using Chaos Mesh (equivalent to AWS Fault Injection Service, Litmus, or Toxiproxy):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: rede-03-latencia-pagamento
  namespace: staging-canary
spec:
  action: delay
  mode: fixed-percent
  value: "5"
  selector:
    namespaces: [staging-canary]
    labelSelectors:
      app: payment-service
  delay:
    latency: "2000ms"
    jitter: "200ms"
  direction: to
  duration: "10m"

Blast Radius & Containment
The blast radius is strictly bound along three dimensions prior to execution:
• Scope: Restricted exclusively to the staging-canary namespace; zero impact on other application pods or databases.
• Traffic: 5% of requests, isolated via header-based routing at the API Gateway.
• Duration: 10 minutes, configured natively in the resource definition so that if connection to the cluster is lost, the experiment terminates automatically.
The emergency abort command (kubectl delete networkchaos net-03-payment-latency) was tested beforehand, complemented by automated abort webhooks triggered by Alertmanager whenever safety thresholds were violated.

Observability and Results Telemetry Tracked

Note: Business metrics validate technical indicators. A degraded system might maintain low latency and low error rates by prematurely returning an empty screen, all while failing 100% of revenue-generating transactions.
Round 1: Hypothesis Refuted
During the first run, the hypothesis failed. Illustrative experimental data:

Distributed tracing quickly isolated root causes:

The underlying HTTP client (RestTemplate) lacked an explicit read timeout (defaulting to infinite).
The Resilience4j circuit breaker was only configured to trip on explicit application exceptions, not slow calls. Consequently, 2-second calls were never registered as failures. The circuit breaker remained closed while Tomcat's worker threads rapidly exhausted the pool. This yielded the primary architectural insight: a resilience pattern was declared in code, yet remained completely inoperative under real failure dynamics. Remediation We deployed three targeted fixes:

# application.yml (Resilience4j)
resilience4j:
  circuitbreaker:
    instances:
      payment:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        slowCallDurationThreshold: 800ms   # chamada lenta passa a contar
        slowCallRateThreshold: 50
        waitDurationInOpenState: 15s
        permittedNumberOfCallsInHalfOpenState: 3
  timelimiter:
    instances:
      payment:
        timeoutDuration: 1s
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 20             # isola o pool de threads

@CircuitBreaker(name = "payment", fallbackMethod = "enfileirarPagamento")
@Bulkhead(name = "payment")
public PagamentoResultado cobrar(Pedido pedido) {
    return paymentClient.cobrar(pedido);
}

// Fallback: não perde o pedido; marca como PAGAMENTO_PENDENTE e publica em fila
private PagamentoResultado enfileirarPagamento(Pedido pedido, Throwable t) {
    filaPagamentos.publicar(pedido.getId());
    return PagamentoResultado.pendente(pedido.getId());
}

Additionally, low-level socket and connect timeouts (500 ms connect, 1000 ms read) were set on the HTTP client pool, as TimeLimiter does not guarantee cancellation of blocking I/O calls on underlying threads.
Round 2: Hypothesis Upheld

The architecture maintained resilience: the circuit opened quickly, the bulkhead prevented thread pool exhaustion, and the asynchronous fallback preserved incoming orders. The engineering tradeoff was deliberate and accepted by business stakeholders: a small fraction of customers received an immediate "payment processing" notification rather than a hard checkout failure.
Resilience Patterns in Action
• Timeout: Enforces a ceiling on wait times, shedding resources promptly.
• Circuit Breaker: Fails fast against degraded dependencies, preventing backlog build-up.
• Bulkhead: Isolates critical resources (thread pools, concurrency quotas) per dependency.
• Queue-based Fallback (Graceful Degradation): Trades immediate strong consistency for system availability.
• Auto Scaling (HPA): Proved ineffective during the incident. Because the bottleneck was blocked I/O rather than CPU saturation, horizontal scaling would have merely amplified concurrency pressure against the already degraded payment backend.
Other Cards in the Deck
The framework accommodates wider failure domains, each governed by its own blast radius constraints:
• AZ-01 (Complete Availability Zone Outage): Verifies traffic reroutes cleanly across surviving AZs and confirms AWS Aurora / RDS Multi-AZ failover completes within SLA (typically 60–120s; actual failover time must be empirically measured rather than assumed). Executed via AWS FIS.
• DATA-02 (Database Connectivity Loss): Validates exponential backoff with full jitter on retries, alongside HikariCP connection pool recovery mechanics when pool connections become stale.

Conclusion Key Takeaways
Unverified resilience is not resilience. The circuit breaker existed in source control, yet failed to protect the system because it had never been tested against latency-based degradation.
Slowness is worse than outright failure. Downstream crashes fail fast; downstream latency silently exhausts upstream threads and memory.
Hypotheses must be falsifiable and validated by business metrics. Relying solely on technical signals masks functional edge cases.
Blast radiuses and automated aborts are prerequisites, not polish. They establish the safety boundary required to run experiments routinely.
Every experiment must drive traceable action: a bug fix, an alert adjustment, or an updated runbook, followed by re-executing the card to validate remediation (as shown in Round 2). Cultural Impact on DevOps The most enduring payoff of chaos engineering is cultural. By treating failure as an objective target of empirical study rather than human error, teams shift toward proactive architectural reviews and blameless post-mortems. Engineers begin developing with failure modes in mind ("What happens when this API call hangs?"), while operations teams validate runbooks in realistic conditions. Game Days double as practical incident-response training, cutting Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR). Next Steps: Automate low-risk cards inside CI/CD pipelines for continuous staging execution, reserve high-impact cards for scheduled Game Days, and track maturity through concrete KPIs: MTTR reduction, issues identified per experiment, and first-round card pass rates.

The Strategy Pattern in Practice

Oriol Moura — Sat, 12 Sep 2026 14:02:10 +0000

*1. Foundations — The Problem and the Theory *
_1.1 Introduction _
Strategy is a behavioral design pattern cataloged by the Gang of Four (Gamma, Helm, Johnson & Vlissides,
1994). Its purpose is to encapsulate a family of interchangeable algorithms behind a common interface,
allowing the algorithm used by an object to vary at runtime without changing the code of the client that
invokes it.
Unlike creational patterns (which deal with how objects are instantiated) or structural patterns (which deal
with how objects compose), Strategy solves a behavioral problem: which business rule should run in a given
context, and how to swap that rule without rewriting the class that uses it.
1.2 The Problem
Medium- and long-lived systems tend to accumulate payment methods, notification providers, shipping-cost
algorithms, discount policies — any rule variation that grows over time. The naive implementation
concentrates these variations in a single class, using a chain of conditionals:

if (method.equals("CREDIT_CARD")) { 
// Stripe logic 
} else if (method.equals("PIX")) { 
// Central Bank logic 
} else if (method.equals("BOLETO")) { 
// issuing bank logic 
} else if (method.equals("PAYPAL")) { 
// PayPal logic 
} // ... and so on

This design violates the Open/Closed Principle: adding a new payment method requires editing an existing
class that already concentrates logic for several distinct external integrations. In practice, on a real legacy
system, this was exactly the bottleneck faced: a PaymentService class with more than 900 lines, low test
coverage (testing one method risked breaking another due to shared side effects), and a commit history in
which any localized change — even to a single gateway — required recompiling, retesting, and redeploying
the entire service, increasing the risk of regression in already-stable payment flows.
1.3 Concept
Strategy solves this by extracting each algorithm variation into its own class, all implementing a common
interface. The object that originally held the logic (the "Context") now depends only on the interface and
receives — through dependency injection or configuration — which concrete implementation to use.
A direct analogy from software development: think of the JDBC interface. Data-access code programs against
java.sql.Connection and java.sql.Driver — never against a specific database implementation. Switching from
PostgreSQL to MySQL doesn't require rewriting the persistence layer, only swapping the injected driver.
Strategy applies exactly the same idea to business rules: the Context (the consumer of the behavior) never
knows the concrete implementation, only the contract.
*2. Development — Case Study, Architecture, and Code
2.1 Real-World Scenario *
Scenario: a Payment Service microservice, responsible for processing payments for an e-commerce platform,
needs to support four payment methods — credit card (via Stripe), Pix (via the Central Bank/SPI API), boleto
(via the issuing bank), and PayPal — with a strong likelihood that new methods will be added next quarter
(digital wallets, crypto assets). The architectural requirement is clear: adding a new payment method must
not require changing or retesting the existing orchestration logic, only adding a new class.
*2.2 UML Class Diagram *

**
2.3 High-Level Architecture Diagram**

*2.4 Java Implementation *
The interface defines the contract every payment strategy must fulfill:
public interface PaymentStrategy {
boolean supports(String method);
PaymentResult pay(Order order);
}
Each external gateway becomes a concrete implementation, isolated and independently testable:

@Component 
public class CreditCardPaymentStrategy implements PaymentStrategy { 
private final StripeClient stripeClient; 
public CreditCardPaymentStrategy(StripeClient stripeClient) { 
this.stripeClient = stripeClient; 
} 
@Override 
    public boolean supports(String method) { 
        return "CREDIT_CARD".equalsIgnoreCase(method); 
    } 

    @Override 
    public PaymentResult pay(Order order) { 
        StripeCharge charge = stripeClient.charge( 
                order.getAmount(), order.getCardToken()); 
        return new PaymentResult(charge.getId(), PaymentStatus.APPROVED); 
    } 
} 
@Component 
public class PixPaymentStrategy implements PaymentStrategy { 

    private final PixClient pixClient; 

    public PixPaymentStrategy(PixClient pixClient) { 
        this.pixClient = pixClient; 
    } 

    @Override 
    public boolean supports(String method) { 
        return "PIX".equalsIgnoreCase(method); 
    } 

    @Override 
    public PaymentResult pay(Order order) { 
        PixCharge charge = pixClient.createCharge(order.getAmount()); 
        return new PaymentResult(charge.getTxId(), PaymentStatus.PENDING); 
    } 
}

BoletoPaymentStrategy and PayPalPaymentStrategy follow exactly the same shape — each isolated in its
own class, with no coupling between them.
The Context handles orchestration only: it selects the correct strategy, delegates execution, persists the
result, and publishes the domain event. It never knows any gateway-specific detail:

@Service 
public class PaymentProcessor { 

    private final List<PaymentStrategy> strategies; 
    private final PaymentRepository paymentRepository; 
    private final PaymentEventPublisher eventPublisher; 

    public PaymentProcessor(List<PaymentStrategy> strategies, 
                             PaymentRepository paymentRepository, 
                             PaymentEventPublisher eventPublisher) { 
        this.strategies = strategies; 
        this.paymentRepository = paymentRepository; 
        this.eventPublisher = eventPublisher; 
    } 

    public PaymentResult processPayment(String method, Order order) { 
        PaymentStrategy strategy = strategies.stream() 
                .filter(s -> s.supports(method)) 
                .findFirst() 
                .orElseThrow(() -> new UnsupportedPaymentMethodException(method)); 

        PaymentResult result = strategy.pay(order); 

        paymentRepository.save(new PaymentRecord(order.getId(), method, result)); 
        eventPublisher.publish( 
                new PaymentProcessedEvent(order.getId(), result.getStatus())); 
return result; 
} 
} 
Note that Spring automatically injects every bean implementing PaymentStrategy into the constructor's 
List<PaymentStrategy> — no manual registration is required. Adding a fifth payment method comes down 
to creating a new class annotated with @Component; PaymentProcessor is not touched, and no existing test 
is affected. 
The REST controller simply delegates to the Context, keeping the presentation layer decoupled from any 
business rule: 
@RestController 
@RequestMapping("/payments") 
public class PaymentController { 
private final PaymentProcessor paymentProcessor; 
public PaymentController(PaymentProcessor paymentProcessor) { 
this.paymentProcessor = paymentProcessor; 
} 
} 
@PostMapping 
public ResponseEntity<PaymentResult> pay(@RequestBody PaymentRequest request) { 
PaymentResult result = paymentProcessor.processPayment( 
request.method(), request.toOrder()); 
return ResponseEntity.ok(result); 
}

*2.5 Pros and Cons *
Advantages
● Open/Closed Principle compliance: new payment methods are added by extension (a new class),
not by modifying the Context.
● Real testability: each strategy is tested in isolation with trivial mocks of its respective HTTP
client/SDK; the Context is tested with fake strategies, without depending on any external gateway.
● Elimination of cyclomatic complexity: the 900-line class with multiple if/else branches was replaced
by five small, cohesive classes, each with a single responsibility (SRP).
● Runtime flexibility: the strategy used depends on an input value (the payment method), not on
conditional compilation or reflection.
Costs and trade-offs
● More classes to manage: a small project with only two payment methods may not justify the
indirection — the pattern pays off once the variation is expected to grow.
● The supports(method) method decentralizes selection logic; if each strategy decides on its own
when it applies, discipline (or contract tests) is needed to prevent two strategies from claiming the
same method simultaneously.
● The API client still needs to supply a correct method identifier; Strategy does not by itself solve
routing — it solves execution once routing has already been decided.
● The indirection overhead (one extra interface call) is technically measurable but irrelevant next to
the network latency of any external payment gateway — it is not a real argument against the
pattern in this scenario.
*3. Conclusion
3.1 Summary *
The Strategy pattern resolved the architectural bottleneck by replacing a monolithic class coupled to multiple
payment gateways with a set of independent strategies orchestrated by a lean Context. The gain is not merely
cosmetic: it is a direct reduction in the cost and risk of evolving the system, measurable in the amount of
code changed and the test surface affected by each new integration.
3.2 Personal Take
The most valuable part of implementing this refactor was not the pattern itself — it is one of the simplest in
the GoF catalog — but the exercise of identifying exactly where behavioral variation was genuinely,
unnecessarily coupled to the Context. A practical warning is worth stating: Strategy should not be applied
preemptively to every if/else. It pays off when there is concrete evidence that the family of algorithms will
grow or has already changed more than once — applying it too early, in a still-unstable domain, only adds
indirection without return.
*3.3 Over to You *
Have you ever dealt with a "do-everything" class that grew until it became a risk with every deploy? What
strategy (pun intended) did you use to decouple that logic? Share in the comments — I'd like to know whether
you reached for Strategy, Chain of Responsibility, or another approach.