An order looks simple from the outside.
A customer clicks Buy.
The system says:
Order confirmed.
Maybe an email arrives.
Maybe the warehouse starts packing.
Maybe a payment is captured.
Maybe the customer sees a tracking number a few hours later.
From the customer's perspective, this is one action.
From the perspective of software architecture, it is a small distributed universe.
An order processing engine sits at the center of that universe.
It coordinates inventory, payments, pricing, customers, fulfillment, shipping, notifications, fraud checks, refunds, cancellations, and state transitions.
And suddenly, something that looked like:
POST /orders
becomes one of the most interesting systems you can build.
Because an order is not merely data.
An order is a process.
It moves.
It changes state.
It triggers other systems.
It must survive retries.
It must handle failures.
It must prevent duplicate actions.
It must preserve history.
And, most importantly, it must never lose the truth about what happened.
That is what makes an order processing engine interesting.
We are not going to build a shopping cart.
We are going to build the machine underneath the shopping cart.
1. The Real Problem
Imagine a customer orders three products.
The system needs to do something like this:
Customer
|
v
Create Order
|
v
Validate Items
|
v
Calculate Price
|
v
Reserve Inventory
|
v
Authorize Payment
|
v
Confirm Order
|
v
Create Fulfillment
|
v
Notify Customer
Looks straightforward.
Now introduce reality.
The customer clicks the button twice.
The network times out after payment succeeds.
The inventory service responds slowly.
One item is out of stock.
The payment provider is temporarily unavailable.
The customer cancels the order while fulfillment is being created.
A notification service crashes.
The application server dies halfway through processing.
A worker receives the same message three times.
A database transaction commits, but the process crashes before publishing an event.
Now our simple order endpoint has become a distributed systems problem.
This is the first important principle:
Order processing is not CRUD. It is state orchestration.
CRUD asks:
What data should I store?
An order engine asks:
What is allowed to happen next?
That difference changes everything.
2. Start With the Order State Machine
The first thing I would design is not the API.
Not the database.
Not Kafka.
Not Redis.
The state machine.
An order needs explicit states.
For example:
PENDING
|
v
VALIDATED
|
v
RESERVED
|
v
PAYMENT_AUTHORIZED
|
v
CONFIRMED
|
v
FULFILLING
|
v
SHIPPED
|
v
DELIVERED
But reality also requires failure paths:
+----------------+
| |
v |
PENDING -> VALIDATED -> RESERVED |
| | | |
| | v |
| | PAYMENT |
| | FAILED |
| | | |
| v v |
| REJECTED CANCELLED
|
v
CANCELLED
The important part is that not every state can transition into every other state.
For example:
DELIVERED -> PENDING
should be impossible.
So we define transitions explicitly.
PENDING -> VALIDATED
VALIDATED -> RESERVED
VALIDATED -> REJECTED
RESERVED -> PAYMENT_AUTHORIZED
RESERVED -> PAYMENT_FAILED
PAYMENT_AUTHORIZED -> CONFIRMED
CONFIRMED -> FULFILLING
FULFILLING -> SHIPPED
SHIPPED -> DELIVERED
And cancellation:
PENDING -> CANCELLED
VALIDATED -> CANCELLED
RESERVED -> CANCELLED
But perhaps:
SHIPPED -> CANCELLED
is not allowed.
Instead, it might require a return workflow.
This is why state machines are powerful.
They turn business rules into architecture.
3. An Order Is a Historical Object
A naive database might contain:
orders
id
customer_id
status
total
created_at
updated_at
That is useful.
But it doesn't tell us everything.
Suppose yesterday:
status = RESERVED
and today:
status = CONFIRMED
What happened between those states?
An advanced order engine should preserve the history.
We can introduce:
order_events
with:
id
order_id
event_type
payload
created_at
Now we can have:
ORDER_CREATED
ORDER_VALIDATED
INVENTORY_RESERVED
PAYMENT_AUTHORIZED
ORDER_CONFIRMED
FULFILLMENT_CREATED
ORDER_SHIPPED
The order becomes more than its current state.
It becomes:
Current State
+
Historical Events
This distinction becomes extremely valuable when debugging production systems.
A customer says:
"I paid but my order wasn't shipped."
Instead of staring at:
status = CONFIRMED
we can inspect:
ORDER_CREATED
ORDER_VALIDATED
INVENTORY_RESERVED
PAYMENT_AUTHORIZED
ORDER_CONFIRMED
and discover:
FULFILLMENT_CREATION_FAILED
Now we know what actually happened.
4. The Architecture
A basic architecture could look like this:
+----------------+
| Client |
+-------+--------+
|
v
+---------------+
| Order API |
+-------+-------+
|
v
+-------------------+
| Order Processing |
| Engine |
+---------+---------+
|
+-----------------+-----------------+
| | |
v v v
+-------------+ +-------------+ +-------------+
| Inventory | | Payment | | Pricing |
+-------------+ +-------------+ +-------------+
| | |
+-----------------+-----------------+
|
v
+------------------+
| Event Bus |
+--------+---------+
|
+----------------+----------------+
| | |
v v v
+-----------+ +-----------+ +-----------+
|Fulfillment| |Notification| | Analytics |
+-----------+ +-----------+ +-----------+
The processing engine coordinates the workflow.
It does not necessarily own every capability.
That distinction matters.
The order engine might know:
Reserve inventory.
Authorize payment.
Confirm order.
But it doesn't need to know the internal mechanics of payment processing.
The payment service owns payment.
The inventory service owns inventory.
The fulfillment service owns fulfillment.
The order engine coordinates them.
5. Designing the Order Model
Let's start with a simplified model.
class Order:
id
customer_id
status
currency
subtotal
tax
shipping
discount
total
version
created_at
updated_at
Then:
class OrderItem:
id
order_id
product_id
quantity
unit_price
discount
subtotal
Notice something important.
We store the price on the order item.
We don't simply reference the current product price.
Why?
Because product prices change.
Imagine:
Monday:
Laptop = $1,000
Customer orders it.
Tuesday:
Laptop = $1,200
If we calculate the old order using the current product price, we have corrupted history.
The order must contain the commercial truth that existed when the transaction occurred.
Therefore:
Product Price
and:
Order Item Price
are different concepts.
6. Order Creation
Our API could expose:
POST /orders
with:
{
"customer_id": "cus_123",
"items": [
{
"product_id": "prod_10",
"quantity": 2
},
{
"product_id": "prod_20",
"quantity": 1
}
],
"currency": "USD"
}
The engine should not immediately charge the customer.
First, it creates the order.
PENDING
Then:
ORDER_CREATED
is recorded.
The engine can now begin processing.
This separation is important because order creation and order completion are different operations.
7. Validation
Validation should happen before expensive operations.
We check:
Does the customer exist?
Are all products valid?
Are quantities positive?
Is the currency supported?
Are products currently sellable?
Are required shipping details present?
Are prices still valid?
Are there restrictions?
For example:
def validate_order(order):
if not order.items:
raise InvalidOrder("Order has no items")
for item in order.items:
if item.quantity <= 0:
raise InvalidOrder("Invalid quantity")
if not product_exists(item.product_id):
raise InvalidOrder("Product does not exist")
If validation succeeds:
PENDING
|
v
VALIDATED
If it fails:
PENDING
|
v
REJECTED
8. Pricing Should Be Deterministic
Pricing is often more complicated than it looks.
The total might be:
Subtotal
- Discounts
+ Tax
+ Shipping
= Total
But discounts can have rules.
For example:
10% off orders above $100
or:
Buy 2, get 1 free
or:
Customer gets $20 credit
This is where a pricing engine can become its own subsystem.
The order engine should ideally receive a pricing result:
{
"subtotal": 240,
"discount": 20,
"tax": 22,
"shipping": 10,
"total": 252
}
Then persist that calculation.
Again:
Never assume you can reconstruct historical commercial decisions from today's configuration.
Store the result.
9. Inventory Reservation
This is one of the most important parts of the system.
Suppose inventory says:
Laptop:
available = 1
Two customers place orders simultaneously.
Both see:
available = 1
Both attempt to purchase.
Without proper coordination:
Customer A -> 1 laptop
Customer B -> 1 laptop
Now we've sold two laptops that don't exist.
This is an overselling problem.
The solution is reservation.
Instead of immediately consuming inventory:
available = 1
we reserve it:
available = 0
reserved = 1
The reservation belongs to the order.
For example:
reservation_id
order_id
product_id
quantity
expires_at
status
Now inventory can enforce:
available quantity >= requested quantity
atomically.
10. Reservations Need Expiration
Imagine:
Order #100
reserves:
2 phones
but the customer never pays.
Those phones should not remain locked forever.
So reservations can have:
expires_at
For example:
created_at = 20:00
expires_at = 20:15
A background worker can release expired reservations.
RESERVATION_EXPIRED
Then:
reserved -> released
This introduces another important concept:
Order processing engines often require both synchronous APIs and asynchronous workers.
11. Payment
Once inventory is reserved, payment can begin.
The order engine might call:
POST /payments/authorize
with:
{
"order_id": "ord_123",
"amount": 252,
"currency": "USD"
}
The payment service responds:
{
"status": "authorized",
"transaction_id": "txn_987"
}
The order transitions:
RESERVED
|
v
PAYMENT_AUTHORIZED
Then:
CONFIRMED
But what happens if payment fails?
RESERVED
|
v
PAYMENT_FAILED
Now the system should release inventory.
Payment failed
|
v
Release reservation
|
v
Cancel order
This is a distributed transaction problem.
12. The Distributed Transaction Trap
We might wish we could do this:
BEGIN TRANSACTION
reserve inventory
charge payment
confirm order
COMMIT
But inventory and payment probably live in different systems.
A database transaction cannot magically span the entire internet.
Therefore, we need a different strategy.
One common pattern is the Saga pattern.
Instead of one giant transaction, we execute a sequence of local transactions with compensating actions.
For example:
Create Order
|
v
Reserve Inventory
|
v
Authorize Payment
|
v
Confirm Order
If payment fails:
Release Inventory
|
v
Cancel Order
The compensation is not:
ROLLBACK
It is an actual business operation.
This is a profound distinction.
Distributed systems don't always undo history.
Sometimes they perform another action that compensates for the previous action.
13. Idempotency
Now we encounter one of the most important concepts in payment and order systems.
Suppose the customer sends:
POST /orders
The server processes it.
Then the network dies.
The client doesn't know whether the request succeeded.
So the client retries.
Now we receive:
POST /orders
POST /orders
If we blindly process both:
Order A
Order B
The customer might get charged twice.
This is why we need idempotency.
The client sends:
Idempotency-Key: 7f93e...
The server stores:
idempotency_key
request_hash
response
created_at
When another request arrives with the same key:
Already processed.
Return previous result.
The key becomes a fingerprint for the operation.
14. Idempotency Is Bigger Than Payments
Many developers associate idempotency with payments.
It is much broader.
Consider:
Reserve inventory
Create fulfillment
Send refund
Create shipment
Generate invoice
Send notification
Any operation that may be retried should be considered for idempotency.
For example:
event_id = evt_123
A worker receives:
ORDER_CONFIRMED
It processes the event.
Then the message broker delivers it again.
Without protection:
Create shipment
Create shipment
With an idempotency table:
processed_events
we check:
if event_id in processed_events:
return
Now:
at-least-once delivery
can be made safe through:
idempotent consumers
15. The Event Bus
After an order is confirmed, many systems might care.
For example:
ORDER_CONFIRMED
could be consumed by:
Fulfillment Service
Notification Service
Analytics Service
Customer Loyalty Service
Invoice Service
Fraud Service
The order engine shouldn't have to call each one synchronously.
Instead:
Order Engine
|
v
Event Bus
|
+----> Fulfillment
|
+----> Notifications
|
+----> Analytics
|
+----> Loyalty
This gives us decoupling.
The order engine publishes:
{
"event_id": "evt_123",
"type": "ORDER_CONFIRMED",
"order_id": "ord_456",
"timestamp": "2026-09-18T18:00:00Z"
}
Consumers decide what to do.
16. But There Is a Dangerous Race
Consider this:
BEGIN TRANSACTION
UPDATE orders
SET status = 'CONFIRMED'
COMMIT
publish ORDER_CONFIRMED
The application crashes after the database commit but before the event is published.
Now:
Database:
CONFIRMED
but:
Event Bus:
nothing
The order is confirmed, but fulfillment never hears about it.
This is one of those bugs that may never appear during local development.
It appears at 2:13 AM in production.
The solution is the Transactional Outbox Pattern.
17. Transactional Outbox
Instead of publishing directly:
Database
+
Event Bus
we write the event into an outbox table within the same database transaction.
BEGIN
UPDATE orders
SET status = 'CONFIRMED'
INSERT INTO outbox_events (...)
COMMIT
Now both changes succeed or fail together.
A separate worker reads:
outbox_events
and publishes them to the message broker.
Database
|
v
Outbox
|
v
Publisher
|
v
Event Bus
If the publisher crashes, it can retry.
This turns an unreliable boundary into a recoverable workflow.
18. Workers
The engine needs workers for asynchronous processing.
A worker might consume:
ORDER_CONFIRMED
and create fulfillment.
Another might process:
RESERVATION_EXPIRED
Another:
PAYMENT_FAILED
Another:
ORDER_DELIVERED
The architecture becomes:
+----------------+
| Order API |
+-------+--------+
|
v
+-------------+
| Database |
+------+------+
|
v
+-------------+
| Outbox |
+------+------+
|
v
+-------------+
| Event Bus |
+------+------+
|
+-------------+-------------+
| | |
v v v
Worker A Worker B Worker C
Now the system can scale workers independently.
19. Retry Strategy
Failures are inevitable.
The question isn't:
"How do we prevent every failure?"
That's impossible.
The better question is:
"How does the system recover from failure?"
Suppose the payment provider returns:
503 Service Unavailable
We shouldn't immediately mark the order as permanently failed.
We can retry.
For example:
Attempt 1 -> immediately
Attempt 2 -> 1 second
Attempt 3 -> 5 seconds
Attempt 4 -> 30 seconds
Attempt 5 -> 2 minutes
This is exponential backoff.
But retries should have limits.
Eventually:
FAILED
or:
MANUAL_REVIEW
depending on the business rules.
20. Dead-Letter Queues
What if an event repeatedly fails?
Suppose:
ORDER_CONFIRMED
cannot be processed because fulfillment has a persistent bug.
We don't want:
retry forever
retry forever
retry forever
Instead:
Event
|
+--> retry
|
+--> retry
|
+--> retry
|
v
Dead Letter Queue
The dead-letter queue gives operators a place to investigate.
We can inspect:
event_id
order_id
error
attempt_count
first_failed_at
last_failed_at
Now operational debugging becomes possible.
21. Concurrency Control
Orders can be processed concurrently.
That means two workers could attempt to modify the same order.
Imagine:
Worker A:
CONFIRMED -> FULFILLING
Worker B:
CONFIRMED -> CANCELLED
Both execute simultaneously.
Which one wins?
This is where optimistic concurrency control can help.
Our order contains:
version = 7
Worker A executes:
UPDATE orders
SET status = 'FULFILLING',
version = 8
WHERE id = ?
AND version = 7;
If the update affects one row:
success
If it affects zero rows:
someone else changed the order
Worker A reloads the latest state.
This prevents silent overwrites.
22. The Order Engine as a State Machine
We can formalize transitions.
TRANSITIONS = {
"PENDING": {
"VALIDATE": "VALIDATED",
"CANCEL": "CANCELLED"
},
"VALIDATED": {
"RESERVE": "RESERVED",
"CANCEL": "CANCELLED"
},
"RESERVED": {
"PAYMENT_SUCCESS": "PAYMENT_AUTHORIZED",
"PAYMENT_FAILED": "PAYMENT_FAILED",
"CANCEL": "CANCELLED"
},
"PAYMENT_AUTHORIZED": {
"CONFIRM": "CONFIRMED"
},
"CONFIRMED": {
"FULFILL": "FULFILLING"
},
"FULFILLING": {
"SHIP": "SHIPPED"
},
"SHIPPED": {
"DELIVER": "DELIVERED"
}
}
Then:
def transition(order, action):
transitions = TRANSITIONS.get(order.status, {})
if action not in transitions:
raise InvalidTransition(
f"{action} is not valid from {order.status}"
)
order.status = transitions[action]
This tiny structure contains an enormous amount of business logic.
It says:
The system cannot simply set status.
The system must perform a valid transition.
That is the difference between:
status management
and:
workflow management
23. Commands and Events
Another useful architectural distinction is between commands and events.
A command says:
Do this.
An event says:
This happened.
For example:
ReserveInventory
is a command.
While:
InventoryReserved
is an event.
Similarly:
AuthorizePayment
is a command.
And:
PaymentAuthorized
is an event.
This distinction gives the system a cleaner mental model.
Command
|
v
Action
|
v
State Change
|
v
Event
For example:
AuthorizePayment
|
v
Payment Provider
|
v
Payment Authorized
|
v
PAYMENT_AUTHORIZED
The event represents the fact.
24. Exactly-Once Is Usually an Illusion
Distributed systems often talk about:
exactly-once processing
But it is dangerous to build an architecture around the assumption that every network operation will happen exactly once.
Messages can be duplicated.
Requests can be retried.
Workers can crash.
Connections can disappear.
Instead, build around:
at-least-once delivery
+
idempotent processing
+
durable state
For example:
Event received twice
|
v
Same event_id
|
v
Idempotency check
|
v
Process once
The goal is not to pretend duplication cannot happen.
The goal is to make duplication harmless.
25. Observability
A production order engine needs more than logs saying:
Processing order...
We need correlation.
Every request and event should have identifiers.
For example:
request_id
order_id
event_id
customer_id
transaction_id
Now we can trace:
Request
|
+--> Order
|
+--> Inventory Reservation
|
+--> Payment Transaction
|
+--> Fulfillment
|
+--> Notification
A useful log might look like:
{
"event": "PAYMENT_AUTHORIZED",
"order_id": "ord_123",
"transaction_id": "txn_987",
"request_id": "req_555"
}
Now production debugging becomes archaeology.
We can reconstruct the journey.
26. Metrics
Important metrics might include:
orders_created_total
orders_confirmed_total
orders_failed_total
orders_cancelled_total
payment_failures_total
inventory_reservation_failures_total
order_processing_duration
fulfillment_latency
event_processing_latency
retry_count
dead_letter_count
One particularly useful metric is:
Order Processing Success Rate
For example:
successful orders
-----------------
total orders
Another:
Time from creation -> confirmation
If that suddenly increases:
20 seconds
|
v
2 minutes
|
v
8 minutes
something has probably degraded.
Metrics turn invisible system behavior into measurable behavior.
27. Database Design
A relational database is a natural starting point.
We might have:
orders
order_items
order_events
reservations
payments
outbox_events
processed_events
For example:
orders
----------------
id
customer_id
status
currency
subtotal
tax
shipping
discount
total
version
created_at
updated_at
order_items
----------------
id
order_id
product_id
quantity
unit_price
discount
subtotal
order_events
----------------
id
order_id
event_type
payload
created_at
outbox_events
----------------
id
aggregate_id
event_type
payload
published
created_at
Indexes matter.
For example:
CREATE INDEX idx_orders_customer
ON orders(customer_id);
And:
CREATE INDEX idx_orders_status
ON orders(status);
For workers:
CREATE INDEX idx_outbox_unpublished
ON outbox_events(published, created_at);
As the system grows, database design becomes part of performance engineering.
28. What Happens When the Server Dies?
Let's walk through a realistic failure.
Suppose:
Order #100
has reached:
RESERVED
Payment succeeds.
The payment provider returns:
transaction_id = txn_123
Then our server crashes.
What happens?
If we designed the system correctly, recovery is possible.
The payment record is durable.
The order state might be updated.
The event can be retried from the outbox.
Workers can resume.
The system doesn't depend on memory.
This is a crucial principle:
Anything necessary for recovery must exist in durable state.
Don't keep workflow truth only in:
RAM
or:
application process state
because processes die.
Databases and durable queues survive them.
29. Cancellation
Cancellation sounds simple:
POST /orders/123/cancel
But cancellation is actually another state machine.
If the order is:
PENDING
we can cancel immediately.
If:
RESERVED
we must release inventory.
If:
PAYMENT_AUTHORIZED
we might need to void the authorization.
If:
CONFIRMED
we might need to initiate a refund.
If:
SHIPPED
we may need a return workflow.
Therefore cancellation isn't:
UPDATE orders
SET status = 'CANCELLED'
It is:
Cancellation Request
|
v
Evaluate Current State
|
+--> Release Inventory
|
+--> Void Payment
|
+--> Refund
|
+--> Notify Customer
|
v
CANCELLED
The correct action depends on history.
30. Refunds
Refunds deserve their own lifecycle.
For example:
REFUND_REQUESTED
|
v
REFUND_PROCESSING
|
v
REFUNDED
with:
REFUND_FAILED
as a possible branch.
Never assume:
refund requested = money returned
Those are different facts.
The payment provider may take time.
The refund might fail.
The customer may receive a partial refund.
So the system should represent reality rather than collapsing multiple states into one.
31. Partial Fulfillment
Now imagine an order contains:
2 laptops
1 monitor
3 keyboards
The warehouse only has:
2 laptops
1 monitor
1 keyboard
Do we cancel everything?
Not necessarily.
The engine may support partial fulfillment.
Order #500
Shipment A
- 2 laptops
- 1 monitor
Shipment B
- 1 keyboard
This means the order itself isn't identical to fulfillment.
We might model:
Order
|
+---- Fulfillment
|
+---- Fulfillment
This distinction becomes important as systems become sophisticated.
32. Scaling the Engine
Imagine we begin with:
100 orders/day
A single application server and PostgreSQL database may be enough.
Then:
10,000 orders/day
Then:
1,000,000 orders/day
Architecture changes.
We might introduce:
Load Balancer
|
v
+-----+-----+-----+
| | | |
API API API
| | |
+-----+-----+
|
v
Database
Workers can scale independently:
Worker x 5
Worker x 20
Worker x 100
Depending on workload.
The event bus becomes the shock absorber between producers and consumers.
Instead of:
Order API
|
+--> Payment
+--> Inventory
+--> Fulfillment
+--> Email
we can use:
Order API
|
v
Event Bus
|
+--> Payment Workers
+--> Inventory Workers
+--> Fulfillment Workers
+--> Notification Workers
33. Partitioning
At very high scale, an event stream may be partitioned.
A useful partition key could be:
order_id
Then events for the same order tend to remain ordered within the same partition.
For example:
Partition 1
---------
Order A
Order A
Order A
Partition 2
---------
Order B
Order B
Partition 3
---------
Order C
Order C
This helps preserve ordering where it matters.
But it introduces another architectural question:
Which operations actually require global ordering?
Often, they don't.
You only need ordering within an aggregate such as an order.
This is another distributed systems lesson:
Don't demand global coordination when local ordering is enough.
34. Security
Orders contain valuable information.
Potentially:
customer information
addresses
payment references
purchase history
The engine should enforce authorization.
For example:
Customer
|
+--> View own orders
+--> Cancel eligible order
Admin
|
+--> View orders
+--> Manage fulfillment
Warehouse
|
+--> View fulfillment information
Never let:
GET /orders/123
mean:
Give me order 123.
It should mean:
Give me order 123 if the authenticated actor is authorized to access it.
Authorization belongs inside the architecture.
Not as an afterthought.
35. Testing the State Machine
The order engine is perfect for automated tests.
We should test valid transitions:
PENDING -> VALIDATED
VALIDATED -> RESERVED
RESERVED -> PAYMENT_AUTHORIZED
And invalid ones:
DELIVERED -> PENDING
SHIPPED -> RESERVED
CANCELLED -> CONFIRMED
We can also test failure scenarios:
payment fails
inventory fails
worker crashes
duplicate event
duplicate request
expired reservation
concurrent update
One particularly useful technique is property-based testing.
Instead of testing only known scenarios, generate sequences of operations and assert invariants.
For example:
An order can never be both:
CANCELLED
and
DELIVERED
Another invariant:
reserved inventory cannot exceed available inventory
Another:
total >= 0
depending on the business domain.
The engine becomes a set of invariants rather than a collection of endpoints.
36. The Most Important Invariants
A mature order engine should define its truths explicitly.
For example:
1. An order cannot contain zero items.
2. Order total must equal its persisted pricing breakdown.
3. Inventory cannot be reserved beyond available quantity.
4. A payment cannot be captured twice for the same operation.
5. Invalid state transitions are rejected.
6. Events have unique identifiers.
7. Consumers must tolerate duplicate events.
8. Historical order prices cannot change because product prices changed.
9. A cancelled order cannot silently become confirmed.
10. Every confirmed order must eventually have a fulfillment outcome.
These aren't just test cases.
They are architectural laws.
37. A Simplified Processing Pipeline
We can now visualize the entire system:
CREATE ORDER
|
v
+---------+
| PENDING |
+----+----+
|
v
VALIDATE
|
v
+--------------+
| VALIDATED |
+------+-------+
|
v
RESERVE INVENTORY
|
v
+--------------+
| RESERVED |
+------+-------+
|
v
AUTHORIZE PAYMENT
|
+----------+----------+
| |
v v
SUCCESS FAILURE
| |
v v
PAYMENT_AUTHORIZED RELEASE STOCK
| |
v v
CONFIRMED CANCELLED
|
v
CREATE FULFILLMENT
|
v
FULFILLING
|
v
SHIPPED
|
v
DELIVERED
And around the entire workflow:
Idempotency
Retries
Events
Outbox
Observability
Concurrency Control
Compensation
These are not decorations.
They are what make the workflow reliable.
38. Start Simple
You don't need to build Amazon on day one.
A reasonable first version might be:
API
|
v
PostgreSQL
|
v
Background Worker
Implement:
Order creation
Order validation
Pricing snapshot
Inventory reservation
Payment integration
Order state machine
Outbox events
Idempotency
Then add:
Message broker
Multiple workers
Dead-letter queues
Distributed tracing
Advanced fulfillment
Fraud detection
Partial shipments
Architecture should evolve with actual requirements.
Complexity has a cost.
A system with twenty distributed services isn't automatically more sophisticated than a well-designed modular monolith.
Sometimes the most advanced architecture is knowing what not to distribute.
39. The Modular Monolith
A beautiful first implementation could look like:
order-engine/
│
├── orders/
│ ├── models
│ ├── service
│ ├── state_machine
│ └── repository
│
├── inventory/
│ ├── reservation
│ └── service
│
├── payments/
│ ├── gateway
│ └── service
│
├── fulfillment/
│ └── service
│
├── events/
│ ├── publisher
│ ├── consumers
│ └── outbox
│
├── idempotency/
│
└── workers/
One deployable application.
Multiple logical modules.
Strong internal boundaries.
Later, if inventory becomes a bottleneck:
inventory/
can become:
Inventory Service
without rewriting the entire business model.
Good modular architecture gives you this escape route.
40. The Deeper Idea
Building an order processing engine teaches something bigger than e-commerce.
It teaches us how software represents change.
A database stores what is true now.
A workflow engine models how truth changes.
An event system records what happened.
A state machine defines what is allowed to happen.
A distributed architecture defines how multiple systems cooperate while things are failing.
And idempotency recognizes a fundamental property of the real world:
The same instruction may arrive more than once.
The network does not care about our intentions.
Packets disappear.
Requests retry.
Servers crash.
Workers restart.
Messages duplicate.
Customers double-click.
Payment providers timeout.
Software must be designed for reality, not the happy path.
41. The Order Engine Is Really a Coordination Machine
At first glance, we might say:
We're building software to process orders.
But that description is too small.
We're actually building a coordination machine.
It coordinates:
Customer Intent
|
v
Order State
|
+---- Pricing
|
+---- Inventory
|
+---- Payment
|
+---- Fulfillment
|
+---- Shipping
|
+---- Notifications
|
+---- Accounting
Each system sees only part of reality.
The order engine connects those realities.
The inventory system knows:
We have 4 units.
The payment system knows:
We received $200.
The fulfillment system knows:
The package was shipped.
The notification system knows:
The customer was informed.
The order engine needs to understand the relationship between them.
That is why order processing is fundamentally an architectural problem.
42. Final Architecture
Putting everything together:
CLIENT
|
v
+---------------+
| API Gateway |
+-------+-------+
|
v
+-----------------------+
| ORDER PROCESSING |
| ENGINE |
+-----------+-----------+
|
+---------------+----------------+
| |
v v
+--------------+ +--------------+
| Database | | Idempotency |
+------+-------+ +--------------+
|
v
+--------------+
| Outbox |
+------+-------+
|
v
+--------------+
| Event Bus |
+------+-------+
|
+-------+--------+----------------+----------------+
| | | |
v v v v
Inventory Payment Fulfillment Notification
Service Service Service Service
| | | |
+----------------+----------------+----------------+
|
v
ORDER EVENTS
|
v
Analytics / BI
Around it:
+-----------------------+
| Observability |
| |
| Logs |
| Metrics |
| Tracing |
| Alerts |
+-----------------------+
+-----------------------+
| Reliability |
| |
| Retries |
| Backoff |
| DLQ |
| Idempotency |
| Compensation |
+-----------------------+
That is no longer:
POST /orders
It is a system.
Conclusion
The interesting part of an order isn't the order.
It's everything that has to happen around it.
A serious order processing engine must answer questions like:
What state is this order in?
What happened before?
What can happen next?
What if the request is repeated?
What if payment succeeds but our server crashes?
What if inventory disappears?
What if an event is delivered twice?
What if a worker dies?
What if cancellation arrives during fulfillment?
What if two workers modify the same order?
What if the customer disputes the history six months later?
These questions push us away from simple CRUD and toward state machines, event-driven architecture, transactional outboxes, sagas, idempotency, concurrency control, durable workflows, and observability.
And that is where the engineering gets interesting.
The code for an order engine might eventually be thousands of lines.
But the real system is defined by a much smaller collection of ideas:
STATE
EVENTS
INVARIANTS
TRANSITIONS
IDEMPOTENCY
DURABILITY
COMPENSATION
OBSERVABILITY
Once those ideas are correct, implementation becomes an engineering exercise.
Without them, even a beautiful codebase can collapse under real-world behavior.
An order processing engine therefore teaches a broader lesson about software architecture:
Good systems don't merely perform actions. They preserve truth while actions are happening.
The customer sees:
Order Confirmed.
Behind that sentence is an entire machine coordinating state across databases, services, workers, networks, and failures.
That machine is the real product.
And building it from scratch is one of the best ways to understand modern distributed software.
I write code. I write songs. And when I build systems, I want to understand the machine underneath the interface.
Top comments (0)