The Problem with "It Works on My Machine"
Every payment system eventually faces the same hard truth: it's not a question of if something will fail, but when. A provider goes down. A DynamoDB partition disappears. A Lambda cold start cascades into a timeout spiral. Route53 takes 47 seconds to reroute traffic.
I built Payment Chaos Engineering — a multi-region serverless payment aggregator supporting 10 African and international providers (Orange Money, MTN MoMo, M-Pesa, Wave, Visa, Mastercard, Bitcoin, PI/SPI BCEAO…) — and chaos engineering was not an afterthought. It was baked into the architecture from day one.
This article covers the full technical stack, the DDD/CQRS/Event Sourcing architecture, and how we intentionally inject failures to validate resilience before production ever sees them.
Architecture at a Glance
┌─────────────────────────────────────────────────────┐
│ Multi-Region AWS Serverless │
│ │
│ us-east-1 (Primary) eu-west-1 (Replica) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ API Gateway │ │ API Gateway │ │
│ │ Lambda/Quark.│◄────────►│ Lambda/Quark.│ │
│ │ DynamoDB GT │ │ DynamoDB GT │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ Shared: Kafka · RabbitMQ · Prometheus · ELK │
│ Chaos Monkey │
└─────────────────────────────────────────────────────┘
Tech stack:
- Runtime: Quarkus 3.6 (native GraalVM compile → cold start < 100ms on Lambda)
- Database: DynamoDB Global Tables (multi-region active-active)
- Messaging: Kafka (event streaming) + RabbitMQ (retry queues + DLX)
- Infra: Terraform 1.5+, AWS Lambda, Route53 Failover, SQS DLQ
- Resilience: MicroProfile Fault Tolerance (@CircuitBreaker, @Retry, @Timeout, @Bulkhead)
- Observability: Prometheus + Grafana + ELK Stack + OpenTelemetry
The Domain Model: DDD Done Right
The core aggregate is Payment. It owns its lifecycle and raises domain events — nothing outside can mutate its state directly.
public class Payment {
private TransactionId transactionId;
private UserId userId;
private Money amount;
private PaymentStatus status;
private PaymentProvider provider;
private List<DomainEvent> uncommittedEvents = new ArrayList<>();
public static Payment create(InitiatePaymentCommand cmd) {
Payment payment = new Payment();
payment.apply(new PaymentInitiatedEvent(
cmd.transactionId(), cmd.userId(), cmd.amount(), cmd.provider()
));
return payment;
}
public void completePayment(String providerTransactionId) {
apply(new PaymentCompletedEvent(this.transactionId, providerTransactionId));
}
public void failPayment(PaymentErrorCode errorCode, String reason) {
apply(new PaymentFailedEvent(this.transactionId, errorCode, reason));
}
// Event sourcing: state IS the sum of events
public static Payment reconstitute(List<DomainEvent> events) {
Payment payment = new Payment();
events.forEach(payment::apply);
return payment;
}
private void apply(DomainEvent event) {
// Mutate state
when(event);
// Record for publishing
uncommittedEvents.add(event);
}
}
9 domain events cover the full lifecycle:
PaymentInitiated → PaymentProcessing → PaymentCompleted / PaymentFailed → PaymentRetryInitiated → PaymentRefunded + FraudDetected + ChaosInjected
CQRS: Commands and Queries Never Mix
The command handler is where the heavy lifting happens:
@ApplicationScoped
public class InitiatePaymentCommandHandler {
@Inject ChaosMonkey chaosMonkey;
@Inject PaymentEventStore eventStore;
@Inject EventPublisher eventPublisher;
@CircuitBreaker(requestVolumeThreshold = 10, failureRatio = 0.5, delay = 5000)
@Retry(maxRetries = 3, delay = 1000, jitter = 500)
@Timeout(value = 30, unit = ChronoUnit.SECONDS)
@Fallback(fallbackMethod = "queueForRetry")
public CompletionStage<PaymentCommandResult> handle(InitiatePaymentCommand command) {
// Chaos injection point
chaosMonkey.maybeInjectFailure(command.providerId());
Payment payment = Payment.create(command);
eventStore.appendEvents(payment.getTransactionId(), payment.uncommittedEvents());
eventPublisher.publishAll(payment.uncommittedEvents());
return CompletableFuture.completedFuture(
PaymentCommandResult.success(payment.getTransactionId())
);
}
// Fallback: queue in RabbitMQ for async retry
public CompletionStage<PaymentCommandResult> queueForRetry(InitiatePaymentCommand cmd) {
eventPublisher.publishFallback(cmd);
return CompletableFuture.completedFuture(
PaymentCommandResult.queued(cmd.transactionId())
);
}
}
The query side is a separate service (payment-query) that maintains read-optimized projections in DynamoDB, updated asynchronously via Kafka events. Zero coupling between write and read paths.
Event Store: DynamoDB as the Source of Truth
@ApplicationScoped
public class DynamoDBEventStore implements PaymentEventStore {
public void appendEvents(TransactionId aggregateId, List<DomainEvent> events) {
// Optimistic concurrency: version check prevents lost updates
List<TransactWriteItem> writes = events.stream()
.map(event -> TransactWriteItem.builder()
.put(Put.builder()
.tableName("PaymentEventStore")
.item(Map.of(
"aggregateId", attr(aggregateId.value()),
"sequenceNumber", attr(event.getVersion()),
"eventType", attr(event.getClass().getSimpleName()),
"payload", attr(serialize(event)),
"occurredAt", attr(event.getOccurredAt().toString()),
"correlationId", attr(event.getCorrelationId())
))
// Fail if version already exists — prevents concurrent writes
.conditionExpression("attribute_not_exists(sequenceNumber)")
.build())
.build())
.toList();
dynamoDb.transactWriteItems(req -> req.transactItems(writes));
}
}
DynamoDB Global Tables replicate across us-east-1 ↔ eu-west-1 with < 1 second lag — every event is durable in two regions before the response returns.
The Chaos Monkey
The ChaosMonkey is injected into every critical path. It supports 11 failure scenarios:
public enum ChaosScenario {
LATENCY, // 100ms – 5000ms artificial delay
ERROR, // Simulated 5xx from provider
THROTTLE, // 429 Too Many Requests
NETWORK_PARTITION, // Drop all packets between services
REGION_FAILURE, // 30s timeout → triggers Route53 failover
DATA_CORRUPTION, // Corrupt payload mid-flight
PROVIDER_TIMEOUT, // Provider takes 30s to respond
DUPLICATE_TRANSACTION, // Replay attack simulation
PARTIAL_FAILURE, // 50% of operations succeed
MEMORY_PRESSURE, // OOM conditions
CPU_SPIKE // 100% CPU for N seconds
}
Configuration via application.properties:
chaos.enabled=true
chaos.failure-rate=0.10 # 10% of requests get chaos
chaos.latency.min-ms=100
chaos.latency.max-ms=5000
chaos.scenarios=LATENCY,ERROR,THROTTLE,NETWORK_PARTITION,REGION_FAILURE
Chaos is observable: every injection emits a Micrometer counter (chaos.injection{scenario,provider}) and a ChaosInjectedEvent to Kafka for audit.
Kafka Topology
7 topics, each with a purpose:
| Topic | Partitions | Retention | Role |
|---|---|---|---|
payment-events |
12 | 7 days | Event Sourcing primary stream |
payment-commands |
6 | 1 day | CQRS command distribution |
chaos-events |
3 | 30 days | Chaos audit trail |
fraud-alerts |
6 | 30 days | Real-time fraud signals |
payment-notifications |
6 | 1 day | SMS/Email/Push triggers |
payment-events.DLT |
3 | 30 days | Unprocessable events |
payment-commands.DLT |
3 | 30 days | Failed command fallback |
Messages carry headers: event-type, correlation-id, schema-version — every event is traceable end-to-end.
10 Chaos Experiments
The test suite (PaymentChaosTest) runs 10 ordered experiments that form a complete chaos engineering story:
| # | Experiment | Hypothesis | SLO |
|---|---|---|---|
| 1 | Steady state baseline | All 10 providers respond 200/202 | 100% |
| 2 | Provider failure (Orange, MTN, M-Pesa, Visa) | Auto-failover to backup | < 15% failure |
| 3 | Artificial latency (1–5s) | Circuit breaker + timeout kicks in | Response < 30s |
| 4 | Circuit breaker activation (100% failure rate) | CB opens, fast-fails | Fast-fail after 10 failures |
| 5 | Cascade failure (80% rate, all providers) | System degrades gracefully | > 60% response rate |
| 6 | Data integrity under chaos | Event Sourcing is consistent | 0 corrupted states |
| 7 | Region failover (Route53) | eu-west-1 takes over in < 60s | Failover < 60s |
| 8 | Load test + chaos (50 concurrent, 10% chaos) | System handles the load | > 85% success |
| 9 | Event sourcing consistency | Replay produces same state | 100% deterministic |
| 10 | Recovery after chaos | System returns to steady state | > 80% success post-chaos |
Results
After running the full suite against a real AWS environment:
- Cold start: 87ms (Quarkus native)
- p99 latency under 10% chaos: 1.8s
- Region failover time: 34s (Route53 + health check)
- Event replay accuracy: 100%
- Test coverage: 87%
- Zero 500 errors from circuit breaker or bulkhead saturation
What I Learned
1. Chaos engineering reveals assumptions, not bugs. The biggest issues we found weren't code bugs — they were assumptions ("DynamoDB will always respond in < 100ms", "the provider SDK retries automatically").
2. Event Sourcing is a chaos engineering superpower. When a chaos experiment corrupts in-flight state, the event log is immutable. Replay always wins.
3. Per-provider circuit breakers matter. A single global CB would have taken down all 10 providers when one failed. Isolation saved us.
4. Fallback is not a last resort — it's a first-class citizen. The RabbitMQ retry queue handles thousands of commands per hour that circuit breakers reject. Without it, those payments would be lost forever.
5. Quarkus native on Lambda is a game changer. The JVM version had 2–8s cold starts. Native dropped it to < 100ms, making Lambda viable for synchronous payment flows.
Open Source
The full project — Terraform, Docker Compose, all services, test suites, monitoring dashboards — is available on GitHub https://github.com/MichelRolandOkoubi/ChaosEngineeringPayment
If you're building payment systems in Africa or globally and want to talk chaos engineering, DDD, or event sourcing: find me on LinkedIn or leave a comment below.
Tags: #ChaosEngineering #AWS #Serverless #Java #Quarkus #DDD #CQRS #EventSourcing #PaymentSystems #Africa #MobileMoney #Kafka #DynamoDB
Top comments (0)