System Design: Order Management System
A capstone system design walkthrough — designing an order management system (OMS) end to end — covering the core domain model, the order as a long-lived saga spanning inventory, payment, and fulfillment, the order state machine and its many legal (and illegal) transitions, idempotency and exactly-once-effect guarantees under retries, coordinating inventory reservation across services, handling cancellations and returns as first-class flows rather than exceptions, and the specific correctness, orchestration, and consistency demands that make order management a uniquely long-running, multi-service system design problem.
Table of Contents
- Introduction
- Why Order Management Is a Different Kind of Hard
- The Core Domain Model
- The Order Event Log: Immutable History as the Source of Truth
- Idempotency: The Single Most Important Property
- The Order State Machine
- Inventory Reservation and the Overselling Problem
- The Saga: Coordinating Order Placement Across Services
- Cancellations, Returns, and Modifications as First-Class Flows
- Fulfillment and Shipping Integration
- Reconciliation
- Data Security and Compliance
- Consistency, Availability, and the CAP Trade-off for Orders
- Scaling the System
- Observability for an Order Management System
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
An order management system takes the general system design vocabulary covered in this series' System Design guide — sagas, state machines, event logs, service coordination — and applies it to a domain where a single logical transaction (an order) can legitimately take hours or days to complete and cross the boundaries of inventory, payment, fulfillment, and shipping services along the way, each of which can independently fail, retry, or take its own sweet time to respond. This guide walks through designing such a system end to end, drawing directly on this series' DDD, Event-Driven Architecture, Microservices, and Payment Processing guides, each of which turns out to be load-bearing infrastructure for getting order management right rather than optional architectural polish.
Client → Order API → [validate, reserve inventory] → OrderSaga
↓ ↓
Order Log (source of truth) Payment Service, Inventory Service, Fulfillment Service
↓
Order Status Query Service (read replicas / CQRS projection)
1. Why Order Management Is a Different Kind of Hard
An order is a long-running process, not a single transaction
Most systems covered in this series can complete a unit of work — a request, a write — within a single request/response cycle. An order cannot: from placement to delivery, an order might legitimately take days, pass through inventory allocation, payment capture, warehouse picking, carrier handoff, and possibly a return, with genuine waiting between each step. This is why the order's state machine (Section 5) and the saga coordinating it (Section 7) dominate this guide's concerns more than any single database transaction ever could — an OMS is fundamentally a long-running process manager, not a CRUD service with a state field.
The system must stay correct even though it doesn't control most of the steps
Payment authorization can take seconds to minutes (3D Secure, per this series'
Payment Processing guide's discussion). Inventory reservation across a
multi-warehouse network isn't instantaneous. Carrier pickup happens on the
carrier's schedule, not the OMS's.
Unlike a system that owns its entire write path, an OMS is a coordinator over services it doesn't control the timing or reliability of — this is why sagas with explicit compensation (Section 7), rather than a single ACID transaction, are the correct mental model here, and why "the order is stuck in an intermediate state" needs to be a genuinely handled, monitored condition (Section 14), not an edge case.
You are almost never the sole source of truth for any single fact about the order
A critical, freeing realization for the design that follows: an order management system, in the overwhelming majority of real-world designs, does not itself hold inventory, does not itself move money, and does not itself ship packages — it orchestrates and records the outcome of calls to specialized services (inventory, payment, warehouse management, carriers) that own those facts. The OMS's job is to be the definitive, auditable record of what was decided and what happened, coordinating those services reliably, not to reimplement inventory management or payment processing itself — precisely the "don't build what a specialized service already does" discipline echoed in this series' Payment Processing and Microservices guides, applied here across an order's full lifecycle.
2. The Core Domain Model
Modeled with DDD, per this series' companion guide
public record OrderId(Guid Value);
public record LineItemId(Guid Value);
public record Money(long MinorUnits, string Currency);
public enum OrderStatus { Created, InventoryReserved, PaymentAuthorized, Confirmed, Fulfilling, Shipped, Delivered, Cancelled, Returned }
public class Order // the AGGREGATE ROOT, per this series' DDD guide
{
public OrderId Id { get; }
public IReadOnlyList<LineItem> Items { get; }
public Money Total { get; }
public OrderStatus Status { get; private set; }
private readonly List<OrderEvent> _domainEvents = new();
public void ConfirmPayment(string paymentAuthorizationId)
{
if (Status != OrderStatus.InventoryReserved)
throw new InvalidOperationException($"Cannot confirm payment for an order in status {Status}");
Status = OrderStatus.PaymentAuthorized;
_domainEvents.Add(new OrderPaymentAuthorizedEvent(Id, paymentAuthorizationId));
}
public void Cancel(string reason)
{
if (Status is OrderStatus.Shipped or OrderStatus.Delivered or OrderStatus.Cancelled)
throw new InvalidOperationException($"Cannot cancel an order in status {Status}");
Status = OrderStatus.Cancelled;
_domainEvents.Add(new OrderCancelledEvent(Id, reason));
}
}
This directly applies this series' DDD guide's aggregate pattern — Order is the aggregate root, enforcing its own state transitions (you cannot confirm payment before inventory is reserved) rather than trusting every caller to check status before mutating it, and raising domain events at exactly the points those transitions genuinely occur.
Line items as entities within the aggregate, not a separate top-level concept
public class LineItem
{
public LineItemId Id { get; }
public string Sku { get; }
public int Quantity { get; private set; }
public Money UnitPrice { get; }
public LineItemStatus Status { get; private set; } // an item can be partially fulfilled/returned independent of the order
}
As covered in this series' DDD guide's aggregate boundary discussion, keeping line items inside the Order aggregate (rather than as independent top-level entities) reflects the genuine business invariant that an order's items are only meaningful together — but per Section 9, individual line items still need their own status, since partial shipment and partial return are normal, not exceptional, in real order fulfillment.
3. The Order Event Log: Immutable History as the Source of Truth
Why a mutable "current order status" field alone is insufficient
-- ❌ A single mutable status column has no record of WHEN each transition happened,
-- what caused it, or how to answer "why is this order still Processing after 3 days"
UPDATE orders SET status = 'Shipped' WHERE id = 1;
An order management system needs more than "what is the order's current status" — it needs an immutable, ordered record of every transition the order went through, when, and why, both for customer support ("where is my order, what happened") and for the saga orchestrator (Section 7) itself to know exactly which compensating actions, if any, are needed on failure.
Event sourcing the order as the natural fit for this domain
CREATE TABLE order_event_log (
sequence_id BIGINT PRIMARY KEY,
order_id UUID NOT NULL, -- partition key: keeps one order's history strictly ordered
event_type VARCHAR NOT NULL, -- Created, InventoryReserved, PaymentAuthorized, Shipped, Cancelled, ...
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
As covered in this series' Event Sourcing guide (within the DDD and Event-Driven Architecture guides), the order domain is an unusually good fit for full event sourcing — the order's current state truly is the fold of everything that happened to it, and the audit/support value of a complete, replayable history (rebuild the order's state at any point in time, diagnose exactly which step of a saga stalled) outweighs the added complexity that event sourcing brings elsewhere in a system where it's less clearly justified.
The order log as the backbone for downstream consumers
Every order event is appended to the log BEFORE any downstream side effect (a
customer notification, an analytics update) is triggered from it — per this
series' Event-Driven Architecture guide's outbox pattern, avoiding dual-write
inconsistency between "record the transition" and "notify about it."
This gives a durable, replayable record and a backbone for downstream consumers via the outbox/CDC pattern, directly echoing this series' Event-Driven Architecture guide's discussion of avoiding dual-write inconsistency between "update order state" and "publish the event" — a notification service, an analytics pipeline, and a customer-facing status page can all consume from the same log without the order service needing to know about any of them individually.
4. Idempotency: The Single Most Important Property
Why this is even more critical here, given how many hops an order touches
As covered throughout this series' RabbitMQ, Kafka, and Event-Driven Architecture guides, every messaging technology provides at-least-once delivery, and every network call can time out ambiguously (did the "place order" request actually succeed server-side before the client gave up waiting?). For an order specifically, an un-idempotent retry means genuinely placing the same order twice — charging the customer twice, reserving inventory twice, shipping two packages for one intended purchase — which is precisely why idempotency is this guide's single most emphasized property, exactly as it is in this series' Payment Processing guide, applied here to the order itself rather than just its payment leg.
Idempotency keys: the standard mechanism, applied at order placement
[HttpPost("/orders")]
public async Task<IActionResult> PlaceOrder(
[FromHeader(Name = "Idempotency-Key")] string idempotencyKey,
PlaceOrderRequest request)
{
var existing = await _idempotencyStore.GetResultAsync(idempotencyKey);
if (existing is not null)
{
return Ok(existing); // the SAME order as the original request, no duplicate order created
}
var order = await _orderService.PlaceAsync(request);
await _idempotencyStore.SaveResultAsync(idempotencyKey, order);
return Ok(order);
}
This is the concrete implementation of the idempotency pattern introduced generally in this series' Redis guide's rate-limiting section and REST guide's discussion, applied at the client-facing order-placement endpoint exactly as this series' Payment Processing guide applies it to payment creation — a client generates a unique idempotency key per logical checkout attempt (not regenerated on retry) and includes it on every request, including retries.
Idempotency at every downstream hop the saga touches, not just order placement
Order API (idempotency key checked here)
→ Inventory reservation call (idempotency key passed through, per Section 6)
→ Payment authorization call (its OWN idempotency key, per this series' Payment
Processing guide — a payment gateway expects and enforces this natively)
→ Fulfillment/warehouse notification (idempotent against redelivery, per this
series' Event-Driven Architecture guide)
Idempotency needs to be enforced at every hop the saga (Section 7) makes, not just the client-facing entry point — each downstream service call carries its own idempotency key derived from the order's ID and the specific step, and each downstream service (inventory, payment, fulfillment) is expected to honor it, since at this many hops "we'll just be extra careful" is not an acceptable substitute for structural, enforced guarantees at every layer.
5. The Order State Machine
An explicit, enumerable set of states and legal transitions
Created → InventoryReserved → PaymentAuthorized → Confirmed → Fulfilling → Shipped → Delivered
↓ ↓ ↓
Cancelled Cancelled Cancelled
As covered in Section 2's Order aggregate, an order's lifecycle is a genuinely large, explicit state machine — larger and longer-lived than most domain objects covered elsewhere in this series — and the aggregate's own methods (ConfirmPayment(), Cancel()) are what enforce that only legal transitions are ever possible, throwing rather than silently succeeding if called out of order (attempting to ship an order whose payment was never authorized, for instance).
Why an explicit state machine matters more here than for most domain objects
Given this guide's emphasis on an order being a long-running, multi-service process (Section 1), having every legal and illegal state transition explicitly enumerated and enforced by the aggregate itself — rather than scattered conditional checks across application code calling into inventory, payment, and fulfillment services independently — is precisely the kind of rigor this series' DDD guide argues pays for itself most clearly in domains with genuinely complex, long-running lifecycles, and few domains fit that description more clearly than order management.
Terminal and semi-terminal states, and the transitions that remain legal from them
public void InitiateReturn(LineItemId itemId, string reason)
{
if (Status != OrderStatus.Delivered)
throw new InvalidOperationException($"Cannot initiate a return for an order in status {Status}");
// ... transitions the SPECIFIC line item, per Section 2, not necessarily the whole order
}
Delivered is terminal for the shipping lifecycle but not for the order as a whole — a return (Section 9) is a legitimate transition available from Delivered, while cancellation is not — encoding exactly which transitions remain legal from which states is what prevents an entire category of "this should never happen but somehow did" production incidents specific to an order's unusually long and branching lifecycle.
6. Inventory Reservation and the Overselling Problem
Why "check stock, then charge, then decrement" is a race condition waiting to happen
❌ Two concurrent orders both check "quantity available: 1", both pass the check,
both proceed to charge the customer — the classic overselling race condition
that plagues naive e-commerce inventory logic under real concurrent load.
As covered in this series' Database guide's concurrency discussion, checking availability and decrementing stock as two separate, unsynchronized steps is exactly the kind of race condition that looks fine in testing and fails constantly at real traffic volume — inventory reservation needs to be an atomic, conditional operation, not a check-then-act sequence.
Reserving inventory atomically, before payment is even attempted
UPDATE inventory
SET available_quantity = available_quantity - @requestedQty,
reserved_quantity = reserved_quantity + @requestedQty
WHERE sku = @sku AND available_quantity >= @requestedQty;
-- zero rows affected means insufficient stock — fail the reservation, don't proceed to payment
Reserving inventory with a single atomic, conditional UPDATE (or the equivalent in whatever inventory store is in use) — rather than a separate read-then-write — closes the race Section 6's opening example describes, and doing this before attempting payment authorization avoids the worse failure mode of charging a customer for an item that turns out to be unavailable.
Reservation has a lifetime — it must expire if the order doesn't complete
A reserved-but-never-confirmed order (customer abandoned checkout after inventory
was reserved but before payment completed) must not hold that inventory hostage
indefinitely — reservations carry a TTL (per this series' Redis/TTL discussion),
released back to available stock if the order doesn't progress within a bounded window.
Per this series' TTL and Background Services guides, an inventory reservation is a temporary hold, not a permanent decrement — a background process (or a TTL-based expiry directly in the inventory store) releases reservations for orders that stall before payment confirmation, since otherwise abandoned checkouts would gradually starve available stock for genuinely completing orders.
7. The Saga: Coordinating Order Placement Across Services
Why order placement is the textbook case for the saga pattern
OrderSaga:
1. OrderService: create order (Created) — compensating action: mark Cancelled
2. InventoryService: reserve stock (Section 6) — compensating action: release reservation
3. PaymentService: authorize payment — compensating action: void authorization / refund
4. OrderService: confirm order (Confirmed) — no compensation needed, this IS the completion
As covered directly in this series' Event-Driven Architecture and Microservices guides, "place an order" is one of the clearest real-world examples of the saga pattern's core motivation: the steps span separate services with separate databases, so there's no single ACID transaction spanning inventory, payment, and the order record itself — the saga replaces that with a sequence of local, fast transactions plus explicit compensating actions for anything that needs to be undone if a later step fails.
Orchestration vs. choreography for this specific saga
Orchestrated (a central OrderSaga coordinator explicitly calls each service in
sequence): easier to reason about the order's exact current step and to
implement timeouts/compensation centrally — the more common choice for order
placement specifically, per this series' Saga Pattern guide's trade-off discussion.
Choreographed (each service reacts to the previous service's published event):
looser coupling, but harder to answer "what step is this specific order on
right now" without piecing together events from multiple services.
Per this series' Saga Pattern guide's explicit comparison, order placement generally favors orchestration over choreography specifically because Section 1's "long-running, needs a clear current state" requirement is much easier to satisfy with a central coordinator that owns the order's state machine (Section 5) directly, rather than inferring it from a scattered sequence of events across services.
Timeouts and compensation for a saga step that never responds
public async Task<SagaStepResult> ReserveInventoryWithTimeoutAsync(Order order, TimeSpan timeout)
{
using var cts = new CancellationTokenSource(timeout);
try
{
return await _inventoryService.ReserveAsync(order, cts.Token);
}
catch (OperationCanceledException)
{
return SagaStepResult.TimedOut; // triggers compensation for any prior completed steps
}
}
Given Section 1's framing of downstream services as not fully within this system's control, every saga step needs an explicit timeout, with a timeout treated the same as an explicit failure for compensation purposes — per this series' Resilience guide's timeout discipline, a saga step that never responds must not leave the order stuck indefinitely in an intermediate state with inventory silently held or a payment silently unresolved.
8. Cancellations, Returns, and Modifications as First-Class Flows
Why these can't be modeled as "just cancel and re-create"
A customer requesting a QUANTITY CHANGE on one line item of an otherwise-shipping
order is a fundamentally different operation from cancelling the whole order —
modeling every change as "cancel + place a new order" loses the connection
between the two for support, accounting, and inventory purposes.
As covered in this series' DDD guide's domain-modeling discipline, cancellations, returns, and partial modifications are genuine, distinct domain operations with their own business rules (a return requires a completed delivery; a quantity reduction on an unshipped item is straightforward; a quantity reduction on an already-picked item requires warehouse coordination) — treating them as first-class flows, each with clear preconditions per Section 5's state machine, rather than approximating them via cancel-and-recreate, is what keeps the order's history (Section 3) honest and the accounting correct.
Partial fulfillment and partial returns at the line-item level
public void RecordPartialShipment(IReadOnlyList<LineItemId> shippedItems, string trackingNumber)
{
foreach (var itemId in shippedItems)
{
var item = Items.Single(i => i.Id == itemId);
item.MarkShipped(trackingNumber);
}
if (Items.All(i => i.Status == LineItemStatus.Shipped)) Status = OrderStatus.Shipped;
// otherwise the order stays Fulfilling — PARTIALLY shipped, a normal, expected intermediate state
}
Per Section 2's decision to give line items their own status within the aggregate, partial shipment (some items ship from stock immediately, others are backordered) and partial returns (a customer returns one item from a multi-item order) are handled naturally — the order's overall status reflects the aggregate of its items' statuses, rather than forcing an artificial "all or nothing" simplification that doesn't match how real fulfillment actually works.
Refund as its own saga, mirroring Section 7's original placement saga
ReturnSaga:
1. Mark line item(s) as ReturnInitiated
2. Await warehouse confirmation of the physical return (per Section 9)
3. PaymentService: issue refund — a NEW, auditable transaction (per this series'
Payment Processing guide), never a mutation of the original charge
4. Mark line item(s) as Returned, release/restock inventory if applicable
Exactly as this series' Payment Processing guide insists a refund is a new, ledger-recorded transaction rather than an erasure of the original charge, a return here is its own saga with its own compensation logic — not simply "undo the order" — preserving the complete, honest record of what was ordered, shipped, and subsequently returned.
9. Fulfillment and Shipping Integration
The warehouse management system (WMS) as another specialized external service
Per Section 1's framing: the OMS doesn't manage warehouse operations itself —
it hands off a confirmed order to a WMS (in-house or third-party) and tracks
the WMS's reported progress (picked, packed, handed to carrier) as events
feeding back into the order's state machine (Section 5).
As covered in this series' API Integration guide, the fulfillment leg is another instance of "orchestrate a specialist service, don't reimplement it" — the OMS's job is to hand off a confirmed order with enough information for the WMS to act, and to reliably ingest status updates the WMS reports back, translating them into the order's own state transitions.
Carrier integration and the asynchronous nature of shipping updates
Carrier tracking updates (per this series' Webhook/polling integration discussion)
arrive asynchronously and out of the OMS's control's timing — handled with the
same webhook-verification and idempotency discipline this series' Payment
Processing guide applies to gateway webhooks, since a forged or duplicated
tracking update is a real, if lower-stakes, integrity concern here too.
Carrier webhooks (or polling, depending on the carrier's integration model) are handled with the same discipline this series' Payment Processing guide applies to payment gateway webhooks — signature verification where the carrier supports it, idempotent processing keyed by tracking event ID, and tolerance for out-of-order delivery, since a "package delivered" update arriving before a "package in transit" update for the same shipment is a realistic occurrence, not a bug to assume away.
10. Reconciliation
Why "the order's status looks right" isn't sufficient — it must be proven against the services it coordinates
Order status says: Shipped
Inventory service says: reservation released, stock decremented
Payment service says: captured, amount matches order total
→ these must agree; any mismatch is a genuine defect in the saga's execution to
find and explain, not a display glitch to quietly ignore
Reconciliation is the (often scheduled, automated) process of comparing the order's own recorded state against the actual state reported by the services it coordinated — this is the concrete, continuously-enforced verification that the saga (Section 7) genuinely completed as recorded, not just an assumption resting on "no error was thrown at the time."
Detecting and resolving stuck sagas
public async Task ReconcileStuckOrdersAsync()
{
var stuck = await _orderRepository.FindOrdersInIntermediateStateOlderThanAsync(TimeSpan.FromHours(1));
foreach (var order in stuck)
{
var actualState = await _sagaStateProbe.ProbeDownstreamServicesAsync(order); // per Section 7
await _alerting.RaiseAsync("Order stuck in intermediate state", order.Id, actualState);
}
}
An order that's been sitting in InventoryReserved for hours without progressing to PaymentAuthorized is exactly the kind of stuck-saga signal Section 7's timeout handling is meant to catch proactively, but reconciliation exists as the safety net for cases the saga's own error handling missed — treated with genuine urgency, since a stuck order usually means a customer is waiting on something that silently isn't happening.
11. Data Security and Compliance
PII and payment data handling, deferring to the specialist guides that already cover them
An order inherently contains customer PII (shipping address, contact information) and touches payment data — per this series' Payment Processing guide's tokenization discussion (Section 5 of that guide), the OMS should never handle raw payment card data directly, only opaque tokens and the payment service's own references, and per this series' Data Privacy guide, order records should be scoped to the minimum PII genuinely needed for fulfillment and support, with retention driven by policy rather than kept indefinitely.
Audit logging as a compliance and dispute-resolution requirement
logger.LogInformation("Order {OrderId} transitioned to {Status} via {Actor}", orderId, newStatus, actor);
As covered in this series' Structured Logging and OWASP Top 10 guides, every order state transition needs to be logged with enough context (who or what system triggered it, when) to support both customer disputes ("I never authorized this cancellation") and regulatory audit requirements where applicable — this is a stricter logging bar than most systems require, precisely because an order's history (Section 3) is frequently the evidentiary record for exactly this kind of dispute.
12. Consistency, Availability, and the CAP Trade-off for Orders
Why order state transitions favor consistency, while status queries don't have to
As covered in this series' System Design guide's CAP theorem discussion, the actual write that advances an order's state — confirming payment, marking a shipment — needs strong consistency: it is generally preferable for a saga step to fail cleanly and trigger compensation (Section 7) than for the order to advance into an inconsistent state that reconciliation (Section 10) later has to painstakingly untangle. Order status queries, by contrast, are exactly where eventual consistency is not just acceptable but the right default.
Where eventual consistency is deliberately, explicitly scoped in
The order's OWN state transition (Section 5, via the saga) → strong consistency required, no compromise
A customer-facing "track my order" PAGE → eventual consistency, a few seconds, is fine
Analytics/reporting on order volume and fulfillment times → eventually consistent, standard CQRS
Not every part of an order management system needs the same consistency bar — the state transition itself absolutely does, but downstream, read-only projections (a tracking page, a reporting dashboard) can and should tolerate the eventual consistency this series' Event-Driven Architecture and CQRS discussions describe generally, since a tracking page being a few seconds stale carries none of the risk a genuinely inconsistent order state does.
13. Scaling the System
Applying this series' System Design guide's building blocks, with order-specific emphasis
CQRS (per this series' Event-Driven Architecture guide): the order-write path
(saga-driven state transitions) stays minimal and correctness-focused; a
separately-scaled, denormalized read model serves customer-facing status
queries and internal reporting without contending with the write path
Read replicas (per this series' PostgreSQL guide): safe for READ-heavy queries
(order history, status lookups) — never route a saga's own state-transition
write to a replica
Queues (per this series' RabbitMQ/Kafka guides): the backbone of asynchronous
saga steps and cross-service event propagation, decoupling the order service
from the availability characteristics of inventory, payment, and fulfillment
Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against Section 12's consistency requirements before being applied — the general principle "identify the bottleneck, then apply the specific technique" holds, but the order's own state-transition path narrows which techniques are safe to apply there versus which belong strictly on the read side.
Sharding, and why it matters less here than in a pure high-throughput system
Unlike a system processing tens of thousands of independent writes per second
(per this series' High-Volume Transaction Processing guide), a single order's
lifecycle involves relatively few writes over its (long) lifetime — sharding
by order_id or customer_id still helps distribute overall write volume, but
the PER-ORDER contention this guide worries about is comparatively rare.
Worth contrasting explicitly with this series' High-Volume Transaction Processing guide: while sharding by entity ID is still the right general approach for distributing an OMS's overall write volume across many concurrent orders, per-order contention (many concurrent writers to the same order) is a much smaller concern here than hot-row contention is in a pure transaction-processing system, since a single order is rarely being updated by more than one process at a time.
14. Observability for an Order Management System
Every guide in this series' observability trio, applied with saga-specific stakes
Structured logs (per this series' Structured Logging guide): every state
transition, every saga step attempt and its outcome, every compensation
triggered — with order ID and correlation ID, per this series' guidance
Distributed tracing (per this series' Distributed Tracing guide): tracing a
single order's saga across inventory, payment, and fulfillment calls — essential
for diagnosing exactly which step a specific stuck or slow order is stalled on
Metrics (per this series' Prometheus/Grafana guide): order placement success
rate, average and p99 time-to-confirmation, saga compensation rate, count of
orders stuck in an intermediate state beyond expected duration — the aggregate
health signals an operations team watches continuously
Every technique from this series' observability guides applies directly, with one order-specific addition worth stating explicitly: because an order's lifecycle genuinely spans hours or days (Section 1), "time since last state transition" is itself a meaningful health metric here in a way it wouldn't be for a short-lived request — an order that hasn't progressed in an unusually long time is a strong, early signal worth alerting on well before a customer complaint arrives.
Alerting on saga-health symptoms, distinct from ordinary error-rate alerting
# Per this series' Prometheus/Grafana guide's symptom-based alerting principle
count(order_state_age_seconds{status="InventoryReserved"} > 3600) > 0
A growing count of orders stuck in a specific intermediate state longer than expected is exactly the kind of symptom this series' Prometheus/Grafana guide argues alerts should be built around, and it's a genuinely different signal from a simple request-error-rate alert — a saga can be "succeeding" on every individual call and still be silently stuck if a downstream service's response never arrives, which is precisely the failure mode this metric is designed to catch.
15. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| No idempotency key on order placement | A network timeout retry genuinely places the same order twice | Idempotency keys enforced at order placement and at every downstream saga step |
| Check-then-decrement inventory logic | A classic race condition producing overselling under real concurrent traffic | Atomic, conditional inventory reservation (single conditional UPDATE), never read-then-write |
| No TTL on inventory reservations | Abandoned checkouts hold inventory hostage indefinitely, starving stock for completing orders | Reservations expire on a bounded TTL and release back to available stock |
| Modeling cancellations/returns as "cancel and re-create" | Loses the connection between the original and adjusted order for support, accounting, and audit | Model cancellations, returns, and modifications as first-class flows with their own preconditions |
| No timeout on saga steps calling downstream services | An order can get stuck indefinitely in an intermediate state if a downstream call never responds | Explicit timeouts on every saga step, triggering compensation just like an explicit failure |
| Treating a saga's "no error thrown" as proof of correct completion | A saga can silently drift from the state it believes it's in without reconciliation catching it | Scheduled reconciliation comparing recorded order state against actual downstream service state |
| Handling carrier/payment webhooks without signature verification or idempotency | Forged or duplicated status updates can corrupt order state | Verify webhook signatures; process by idempotency key, same discipline as payment gateway webhooks |
| Routing order state-transition writes to a read replica for "performance" | Introduces real risk of the saga acting on stale state | Order state transitions always go to the strongly consistent write path; replicas serve read-only status queries only |
Quick Reference Table
| Concept | Purpose |
|---|---|
Order aggregate + state machine |
Enforces only legal order state transitions across a genuinely long, branching lifecycle |
| Event-sourced order log | The provable, replayable history every support inquiry, audit, and saga decision depends on |
| Idempotency key at every hop | Prevents duplicate orders, duplicate charges, and duplicate reservations from routine retries |
| Atomic inventory reservation with TTL | Prevents overselling and keeps abandoned-checkout inventory from being held hostage |
| Orchestrated saga with compensation | Coordinates order placement across inventory, payment, and fulfillment without a single distributed transaction |
| First-class cancellation/return/return flows | Keeps partial fulfillment and partial returns honest and auditable, rather than approximated |
| Reconciliation against downstream services | Catches sagas that drifted from their recorded state despite no individual step erroring |
| CQRS read model for status queries | Keeps the correctness-critical write path separate from high-volume, latency-tolerant reads |
Conclusion
An order management system takes every general system design technique covered throughout this series and applies it to a process that is fundamentally longer-running and more multi-service than most systems are designed to assume — because an order genuinely spans hours or days, crosses services this system doesn't control the timing of, and must stay correct and explainable through cancellations, partial fulfillment, and returns along the way. The design that actually holds up under that reality rests on a small number of non-negotiable foundations: an event-sourced order log as the provable, replayable history of everything that happened and why; idempotency enforced at every hop the order's saga touches; atomic inventory reservation that closes the overselling race condition; an orchestrated saga with explicit compensation and timeouts rather than an assumed single transaction; and cancellations, returns, and modifications treated as first-class, auditable flows rather than approximated as cancel-and-recreate.
Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — DDD's aggregates enforcing a genuinely large and branching state machine, Event-Driven Architecture's sagas and idempotent, outbox-backed event propagation, Payment Processing's webhook and refund discipline applied to carrier integration and returns, and the full observability trio watching over a process where "time since last progress" matters as much as any error rate. Order management is, in that sense, less a distinct discipline from everything else in this series than the place where its cumulative lessons about long-running coordination, honest state representation, and reconciliation with reality matter more constantly, and more visibly, than almost anywhere else.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the stuck-order-in-InventoryReserved-for-three-days incident that turned out to matter far more than a clean happy-path demo ever should.
Top comments (0)