How I Built a Chaos Engineering Platform for a Multi-Region Serverless Payment Aggregator (AWS + Quarkus + DDD)

michel dre okoubi — Mon, 20 Jul 2026 12:47:54 +0000

The Problem with "It Works on My Machine"

Every payment system eventually faces the same hard truth: it's not a question of if something will fail, but when. A provider goes down. A DynamoDB partition disappears. A Lambda cold start cascades into a timeout spiral. Route53 takes 47 seconds to reroute traffic.

I built Payment Chaos Engineering — a multi-region serverless payment aggregator supporting 10 African and international providers (Orange Money, MTN MoMo, M-Pesa, Wave, Visa, Mastercard, Bitcoin, PI/SPI BCEAO…) — and chaos engineering was not an afterthought. It was baked into the architecture from day one.

This article covers the full technical stack, the DDD/CQRS/Event Sourcing architecture, and how we intentionally inject failures to validate resilience before production ever sees them.

Architecture at a Glance

┌─────────────────────────────────────────────────────┐
│            Multi-Region AWS Serverless               │
│                                                     │
│  us-east-1 (Primary)      eu-west-1 (Replica)       │
│  ┌──────────────┐          ┌──────────────┐          │
│  │ API Gateway  │          │ API Gateway  │          │
│  │ Lambda/Quark.│◄────────►│ Lambda/Quark.│          │
│  │ DynamoDB GT  │          │ DynamoDB GT  │          │
│  └──────────────┘          └──────────────┘          │
│                                                     │
│  Shared: Kafka · RabbitMQ · Prometheus · ELK        │
│          Chaos Monkey                               │
└─────────────────────────────────────────────────────┘

Tech stack:

Runtime: Quarkus 3.6 (native GraalVM compile → cold start < 100ms on Lambda)
Database: DynamoDB Global Tables (multi-region active-active)
Messaging: Kafka (event streaming) + RabbitMQ (retry queues + DLX)
Infra: Terraform 1.5+, AWS Lambda, Route53 Failover, SQS DLQ
Resilience: MicroProfile Fault Tolerance (@CircuitBreaker, @Retry, @Timeout, @Bulkhead)
Observability: Prometheus + Grafana + ELK Stack + OpenTelemetry

The Domain Model: DDD Done Right

The core aggregate is Payment. It owns its lifecycle and raises domain events — nothing outside can mutate its state directly.

public class Payment {
    private TransactionId transactionId;
    private UserId userId;
    private Money amount;
    private PaymentStatus status;
    private PaymentProvider provider;
    private List<DomainEvent> uncommittedEvents = new ArrayList<>();

    public static Payment create(InitiatePaymentCommand cmd) {
        Payment payment = new Payment();
        payment.apply(new PaymentInitiatedEvent(
            cmd.transactionId(), cmd.userId(), cmd.amount(), cmd.provider()
        ));
        return payment;
    }

    public void completePayment(String providerTransactionId) {
        apply(new PaymentCompletedEvent(this.transactionId, providerTransactionId));
    }

    public void failPayment(PaymentErrorCode errorCode, String reason) {
        apply(new PaymentFailedEvent(this.transactionId, errorCode, reason));
    }

    // Event sourcing: state IS the sum of events
    public static Payment reconstitute(List<DomainEvent> events) {
        Payment payment = new Payment();
        events.forEach(payment::apply);
        return payment;
    }

    private void apply(DomainEvent event) {
        // Mutate state
        when(event);
        // Record for publishing
        uncommittedEvents.add(event);
    }
}

9 domain events cover the full lifecycle:
PaymentInitiated → PaymentProcessing → PaymentCompleted / PaymentFailed → PaymentRetryInitiated → PaymentRefunded + FraudDetected + ChaosInjected

CQRS: Commands and Queries Never Mix

The command handler is where the heavy lifting happens:

@ApplicationScoped
public class InitiatePaymentCommandHandler {

    @Inject ChaosMonkey chaosMonkey;
    @Inject PaymentEventStore eventStore;
    @Inject EventPublisher eventPublisher;

    @CircuitBreaker(requestVolumeThreshold = 10, failureRatio = 0.5, delay = 5000)
    @Retry(maxRetries = 3, delay = 1000, jitter = 500)
    @Timeout(value = 30, unit = ChronoUnit.SECONDS)
    @Fallback(fallbackMethod = "queueForRetry")
    public CompletionStage<PaymentCommandResult> handle(InitiatePaymentCommand command) {
        // Chaos injection point
        chaosMonkey.maybeInjectFailure(command.providerId());

        Payment payment = Payment.create(command);
        eventStore.appendEvents(payment.getTransactionId(), payment.uncommittedEvents());
        eventPublisher.publishAll(payment.uncommittedEvents());

        return CompletableFuture.completedFuture(
            PaymentCommandResult.success(payment.getTransactionId())
        );
    }

    // Fallback: queue in RabbitMQ for async retry
    public CompletionStage<PaymentCommandResult> queueForRetry(InitiatePaymentCommand cmd) {
        eventPublisher.publishFallback(cmd);
        return CompletableFuture.completedFuture(
            PaymentCommandResult.queued(cmd.transactionId())
        );
    }
}

The query side is a separate service (payment-query) that maintains read-optimized projections in DynamoDB, updated asynchronously via Kafka events. Zero coupling between write and read paths.

Event Store: DynamoDB as the Source of Truth

@ApplicationScoped
public class DynamoDBEventStore implements PaymentEventStore {

    public void appendEvents(TransactionId aggregateId, List<DomainEvent> events) {
        // Optimistic concurrency: version check prevents lost updates
        List<TransactWriteItem> writes = events.stream()
            .map(event -> TransactWriteItem.builder()
                .put(Put.builder()
                    .tableName("PaymentEventStore")
                    .item(Map.of(
                        "aggregateId",     attr(aggregateId.value()),
                        "sequenceNumber",  attr(event.getVersion()),
                        "eventType",       attr(event.getClass().getSimpleName()),
                        "payload",         attr(serialize(event)),
                        "occurredAt",      attr(event.getOccurredAt().toString()),
                        "correlationId",   attr(event.getCorrelationId())
                    ))
                    // Fail if version already exists — prevents concurrent writes
                    .conditionExpression("attribute_not_exists(sequenceNumber)")
                    .build())
                .build())
            .toList();

        dynamoDb.transactWriteItems(req -> req.transactItems(writes));
    }
}

DynamoDB Global Tables replicate across us-east-1 ↔ eu-west-1 with < 1 second lag — every event is durable in two regions before the response returns.

The Chaos Monkey

The ChaosMonkey is injected into every critical path. It supports 11 failure scenarios:

public enum ChaosScenario {
    LATENCY,           // 100ms – 5000ms artificial delay
    ERROR,             // Simulated 5xx from provider
    THROTTLE,          // 429 Too Many Requests
    NETWORK_PARTITION, // Drop all packets between services
    REGION_FAILURE,    // 30s timeout → triggers Route53 failover
    DATA_CORRUPTION,   // Corrupt payload mid-flight
    PROVIDER_TIMEOUT,  // Provider takes 30s to respond
    DUPLICATE_TRANSACTION, // Replay attack simulation
    PARTIAL_FAILURE,   // 50% of operations succeed
    MEMORY_PRESSURE,   // OOM conditions
    CPU_SPIKE          // 100% CPU for N seconds
}

Configuration via application.properties:

chaos.enabled=true
chaos.failure-rate=0.10        # 10% of requests get chaos
chaos.latency.min-ms=100
chaos.latency.max-ms=5000
chaos.scenarios=LATENCY,ERROR,THROTTLE,NETWORK_PARTITION,REGION_FAILURE

Chaos is observable: every injection emits a Micrometer counter (chaos.injection{scenario,provider}) and a ChaosInjectedEvent to Kafka for audit.

Kafka Topology

7 topics, each with a purpose:

Topic	Partitions	Retention	Role
`payment-events`	12	7 days	Event Sourcing primary stream
`payment-commands`	6	1 day	CQRS command distribution
`chaos-events`	3	30 days	Chaos audit trail
`fraud-alerts`	6	30 days	Real-time fraud signals
`payment-notifications`	6	1 day	SMS/Email/Push triggers
`payment-events.DLT`	3	30 days	Unprocessable events
`payment-commands.DLT`	3	30 days	Failed command fallback

Messages carry headers: event-type, correlation-id, schema-version — every event is traceable end-to-end.

10 Chaos Experiments

The test suite (PaymentChaosTest) runs 10 ordered experiments that form a complete chaos engineering story:

#	Experiment	Hypothesis	SLO
1	Steady state baseline	All 10 providers respond 200/202	100%
2	Provider failure (Orange, MTN, M-Pesa, Visa)	Auto-failover to backup	< 15% failure
3	Artificial latency (1–5s)	Circuit breaker + timeout kicks in	Response < 30s
4	Circuit breaker activation (100% failure rate)	CB opens, fast-fails	Fast-fail after 10 failures
5	Cascade failure (80% rate, all providers)	System degrades gracefully	> 60% response rate
6	Data integrity under chaos	Event Sourcing is consistent	0 corrupted states
7	Region failover (Route53)	eu-west-1 takes over in < 60s	Failover < 60s
8	Load test + chaos (50 concurrent, 10% chaos)	System handles the load	> 85% success
9	Event sourcing consistency	Replay produces same state	100% deterministic
10	Recovery after chaos	System returns to steady state	> 80% success post-chaos

Results

After running the full suite against a real AWS environment:

Cold start: 87ms (Quarkus native)
p99 latency under 10% chaos: 1.8s
Region failover time: 34s (Route53 + health check)
Event replay accuracy: 100%
Test coverage: 87%
Zero 500 errors from circuit breaker or bulkhead saturation

What I Learned

1. Chaos engineering reveals assumptions, not bugs. The biggest issues we found weren't code bugs — they were assumptions ("DynamoDB will always respond in < 100ms", "the provider SDK retries automatically").

2. Event Sourcing is a chaos engineering superpower. When a chaos experiment corrupts in-flight state, the event log is immutable. Replay always wins.

3. Per-provider circuit breakers matter. A single global CB would have taken down all 10 providers when one failed. Isolation saved us.

4. Fallback is not a last resort — it's a first-class citizen. The RabbitMQ retry queue handles thousands of commands per hour that circuit breakers reject. Without it, those payments would be lost forever.

5. Quarkus native on Lambda is a game changer. The JVM version had 2–8s cold starts. Native dropped it to < 100ms, making Lambda viable for synchronous payment flows.

Open Source

The full project — Terraform, Docker Compose, all services, test suites, monitoring dashboards — is available on GitHub https://github.com/MichelRolandOkoubi/ChaosEngineeringPayment

If you're building payment systems in Africa or globally and want to talk chaos engineering, DDD, or event sourcing: find me on LinkedIn or leave a comment below.

Tags: #ChaosEngineering #AWS #Serverless #Java #Quarkus #DDD #CQRS #EventSourcing #PaymentSystems #Africa #MobileMoney #Kafka #DynamoDB

DEV Community: michel dre okoubi