Designing a Production Serverless API: Beyond API Gateway + Lambda
A serverless API can begin with a diagram that looks almost too easy:
Client
↓
API Gateway
↓
Lambda
↓
DynamoDB
And for many applications, this is a perfectly reasonable starting point.
But now imagine we are building an order API.
Traffic is unpredictable. Most of the day we receive only a few requests, but during promotions thousands of customers can arrive within a short period.
A customer sends:
POST /orders
with something like:
{
"customerId": "C101",
"items": [
{
"productId": "P10",
"quantity": 2
}
]
}
Our requirements initially sound straightforward:
- Create the order.
- Check inventory.
- Take payment.
- Send a confirmation email.
We could put everything inside one Lambda:
async function createOrder(order) {
validate(order);
await saveOrder(order);
await reserveInventory(order);
await chargePayment(order);
await sendEmail(order);
return {
success: true
};
}
It works.
Until it doesn't.
This article isn't about memorizing AWS services.
Instead, we'll start with that simple implementation and make the requirements progressively harder.
Every time the design breaks, we'll ask:
What problem do we need to solve next?
That is how our simple Lambda gradually becomes a production serverless architecture.
Why serverless in the first place?
Our traffic is unpredictable.
Running servers continuously for occasional spikes could mean maintaining capacity that sits mostly unused.
Lambda is attractive because AWS manages the underlying execution infrastructure and can increase concurrent execution as demand grows, within the applicable scaling and account limits.
So our first version is:
Customer
↓
API Gateway
↓
CreateOrder Lambda
↓
DynamoDB
DynamoDB also fits the serverless model well for workloads where its access-pattern-oriented data model makes sense.
But selecting serverless doesn't remove architecture.
We are about to discover why.
Problem #1: One Lambda is doing too much
Consider this sequence:
Save order ✅
Reserve inventory ✅
Charge payment ✅
Send email ❌ timeout
Our Lambda throws an error.
From the client's perspective, the request failed.
So the client retries.
What should happen?
Should we:
- create another order?
- reserve inventory again?
- charge the customer again?
- attempt the email again?
We have mixed operations with very different failure characteristics inside one request.
Sending an email should not determine whether an already successful payment is considered successful.
So our first architectural improvement is decoupling.
Email doesn't belong in the critical request path
After payment succeeds, we don't need the customer to wait while an email provider processes a message.
Instead:
Payment succeeds
↓
SQS
↓
Email Lambda
↓
Email provider
Now the payment workflow can finish while email processing happens independently.
Amazon SQS acts as a buffer between the producer and consumer.
This gives us an important principle:
A queue does not make the consumer faster. It allows producers and consumers to operate at different rates.
It also isolates failures.
If the email provider is temporarily unavailable, that does not need to break order creation.
But asynchronous systems introduce a new issue.
Lambda processing from SQS can result in messages being processed more than once, so AWS recommends making processing idempotent.
That brings us to one of the most important concepts in serverless systems.
Problem #2: Retries can duplicate business operations
Imagine the customer sends:
POST /orders
The server creates:
ORD-9001
but the response is lost because of a network timeout.
The customer retries.
Without protection, we might create:
ORD-9002
for exactly the same checkout.
Worse, the same problem can happen during payment.
Payment provider charges €100 ✅
↓
Lambda loses connection ❌
↓
Lambda retries
↓
Charge another €100? 😬
A normal database rollback cannot solve this.
The payment provider exists outside our database transaction.
Idempotency
An operation is idempotent when repeating the same logical request does not create an additional side effect.
For order creation, the client might send:
Idempotency-Key: checkout-abc-123
On the first request:
checkout-abc-123
↓
Create ORD-9001
If that exact request is retried:
checkout-abc-123
↓
Already processed
↓
Return ORD-9001
Not:
Create ORD-9002
For payment, we want the same concept.
payment idempotency key = ORD-9001
If a network failure causes us to retry, the retry represents:
“Give me the result of the same payment operation.”
not:
“Perform a brand-new payment.”
This gives us a useful mental rule:
Retry makes systems resilient. Idempotency makes retries safe.
Problem #3: We now have a workflow
Our order process has grown:
Create order
↓
Check inventory
↓
Reserve inventory
↓
Process payment
↓
Update order
↓
Queue email
We could manually call Lambda A from Lambda B, Lambda B from Lambda C and keep adding flags to DynamoDB.
But now we have questions such as:
- What if payment times out?
- What if the card is declined?
- Which errors should retry?
- How long should we wait?
- What happens after the last retry?
- How do we know which step the order is currently in?
At this point we are no longer dealing with a simple chain of functions.
We are dealing with a workflow.
That is where AWS Step Functions becomes useful.
Step Functions lets us model workflows as state machines and orchestrate distributed applications and microservices.
Our order process can now be represented explicitly:
Create Order
↓
Reserve Inventory
↓
Process Payment
/ \
/ \
Success Failure
↓ ↓
Update Handle
Order failure
↓
Queue Email
↓
Complete
Step Functions without the mystery
A few states are enough to understand most of our example.
Task
Do something.
ReserveInventory
ProcessPayment
UpdateOrder
Choice
Make a decision.
Payment result
/ \
/ \
SUCCESS DECLINED
↓ ↓
Continue Failure path
A Choice state adds conditional branching to a Step Functions workflow.
Retry
Try a failed operation again.
But only when retrying makes sense.
Catch
If the operation cannot recover, route execution to another state.
Succeed / Fail
End the workflow intentionally as successful or failed.
The important part is not memorizing state names.
It's understanding that business flow is now visible outside our Lambda implementation.
Technical failure is not business failure
Consider payment.
A provider timeout might mean:
temporary network problem
That is potentially retryable.
But:
CARD_DECLINED
is not a temporary infrastructure failure.
Retrying the same declined card three times probably makes no sense.
So:
Payment
|
+-- TIMEOUT
| ↓
| Retry
|
+-- CARD_DECLINED
↓
Do not retry
↓
Ask customer
for another method
Step Functions supports error-specific retry and catch behavior.
This gives us another useful distinction:
Temporary technical failure
→ Retry
Valid negative business result
→ Business workflow path
Not every undesirable result should become an exception.
So what is a DLQ?
Retry/Catch and a Dead-Letter Queue solve related but different problems.
Imagine SQS contains a message that repeatedly fails:
message
↓
attempt 1 ❌
attempt 2 ❌
attempt 3 ❌
attempt 4 ❌
Instead of allowing it to continually interfere with normal processing, it can eventually be moved to a dead-letter queue for investigation or later redrive.
So remember:
Retry
→ Try this operation again.
Catch
→ The operation did not recover.
Which workflow path should run?
DLQ
→ This queued message repeatedly failed.
Move it away from normal processing.
They are not interchangeable concepts.
Problem #4: We don't have one giant rollback anymore
Now suppose the workflow is:
Create Order ✅
Reserve Inventory ✅
Charge Payment ❌
If this were one relational database transaction, we might think:
ROLLBACK;
But inventory, payment and shipping may be completely separate services.
There is no global database transaction covering everything.
So if payment permanently fails, we must undo the earlier successful business action:
Release inventory
This leads us to the Saga pattern.
Saga in simple language
A Saga breaks a distributed transaction into smaller local transactions.
For important forward actions, we define an appropriate compensating action.
For example:
Forward action Compensation
Create order → Cancel order
Reserve inventory → Release inventory
Charge payment → Refund payment
Create shipment → Cancel shipment
Suppose:
Create Order ✅
Reserve Inventory ✅
Charge Payment ✅
Create Shipment ❌
We might compensate:
Shipment failed
↓
Refund payment
↓
Release inventory
↓
Cancel order
A compensation isn't necessarily a magical reversal of history.
A refund, for example, is another business operation that compensates for a successful charge.
That distinction matters.
There are two common Saga coordination styles.
Choreography
Services react to events:
OrderCreated
↓
Inventory service
InventoryReserved
↓
Payment service
PaymentFailed
↓
Inventory service releases stock
Orchestration
A central workflow explicitly coordinates the steps:
Step Functions
|
+--------+--------+
| | |
Order Inventory Payment
Our example is a natural fit for orchestration because the workflow has clear business states, retries and compensations.
Problem #5: Some work can happen in parallel
Payment has succeeded.
Now we need to:
Send confirmation email
Update loyalty points
Notify warehouse
These operations do not necessarily depend on one another.
Running:
Email
↓
Loyalty
↓
Warehouse
would unnecessarily serialize them.
Step Functions has a Parallel state for independent branches:
Payment Success
↓
Parallel
/ | \
↓ ↓ ↓
Email Loyalty Warehouse
\ | /
\ | /
complete
A Parallel state starts its branches concurrently and waits for all branches to terminate before moving on.
But this introduces an architectural choice.
If the main workflow must know that all three operations completed, Parallel makes sense.
If these are independent side effects that the main workflow doesn't need to wait for, event-driven fan-out may be cleaner:
PaymentSucceeded
↓
Event/Event Bus
/ | \
↓ ↓ ↓
Email Loyalty Warehouse
So:
Parallel state means “do these things concurrently and my workflow cares about their completion.”
while event choreography often means:
“These consumers can react independently and the producer doesn't need to coordinate them.”
Map: parallel processing for collections
Parallel is useful when the branches perform different jobs.
What if an order contains 500 items and we need to check every item?
That's the same operation repeated over a collection.
A Step Functions Map state fits that problem.
500 order items
↓
Map
↓
Check inventory for each item
For large workloads, Step Functions also provides Distributed Map; AWS recommends that mode for scenarios such as concurrency above 40 iterations or very large execution histories/datasets.
But this raises another question.
Should all 500 checks execute simultaneously?
Maybe not.
Problem #6: Scaling can become dangerous
This is one of the most important serverless lessons.
Imagine:
500 inventory checks
and the inventory dependency can safely handle:
10 concurrent operations
Running all 500 at once would simply move the bottleneck downstream.
The exact same principle appears in ordinary Node.js:
await Promise.all(hugeArray.map(processItem));
can create too much concurrency.
Different technology, same engineering rule:
Parallelism improves throughput until the dependency becomes the bottleneck.
Now apply that to Lambda itself.
Suppose our API normally receives:
20 requests/second
but a promotion suddenly sends thousands.
Lambda can scale concurrency.
That sounds excellent until every execution calls a payment provider that safely supports only 100 requests per second.
Thousands of requests
↓
Lambda
scales aggressively
↓
Payment Provider
🔥🔥🔥
The Lambda tier can scale faster than the dependency behind it.
Reserved concurrency
Lambda supports reserved concurrency, which can reserve concurrency for a function while also placing an upper bound on that function's concurrency. AWS specifically notes it can help prevent overwhelming downstream resources such as database connections.
Conceptually:
Payment Lambda
maximum concurrency = 100
↓
Payment Provider
Now scaling has a guardrail.
But simply throttling thousands of requests isn't always a good customer experience.
For asynchronous workloads, SQS can absorb the burst:
5,000 jobs
↓
SQS
====================
jobs waiting safely
====================
↓
Lambda consumers
controlled throughput
↓
Dependency
SQS/Lambda event source mappings also support concurrency controls.
Again:
Automatic scaling is not automatically safe architecture.
Always ask:
Can Lambda scale?
↓ yes
Can everything behind Lambda
scale at the same rate?
Cold starts and provisioned concurrency
When Lambda needs a new execution environment, initialization occurs before the handler processes the request.
That extra initialization latency is commonly called a cold start.
A simplified comparison:
Warm
Request
↓
Existing environment
↓
handler()
versus:
Cold
Request
↓
Initialize environment
↓
Load runtime/code/dependencies
↓
handler()
This is also why reusable clients are often initialized outside the handler when appropriate:
const client = createClient();
export const handler = async (event) => {
// reuse client
};
For latency-sensitive workloads, Lambda also offers provisioned concurrency, which keeps execution environments initialized ahead of demand.
Don't confuse the two concepts:
Reserved concurrency
→ controls/reserves concurrency capacity
Provisioned concurrency
→ keeps execution environments initialized
for predictable startup latency
Problem #7: DynamoDB scales, but our key design still matters
A serverless database does not remove data-modeling decisions.
DynamoDB uses partition-key values to determine how items are distributed internally. Items sharing a partition-key value are grouped together, and when a sort key exists, they are ordered by sort-key value.
Suppose we choose:
PK = merchantId
Usually traffic is distributed:
Merchant A → moderate traffic
Merchant B → moderate traffic
Merchant C → moderate traffic
But during a huge campaign:
Merchant A → enormous traffic 🔥
Merchant B → small
Merchant C → small
Now one partition-key value becomes disproportionately busy.
AWS recommends designing partition keys with many distinct values and reasonably uniform activity.
This is why DynamoDB design begins with access patterns.
Partition key and sort key, simplified
A useful mental model:
Partition Key
→ Which logical group does this item belong to?
Sort Key
→ How do I identify/order items inside that group?
For example:
PK = MERCHANT#M123
SK = ORDER#2026-08-10#9001
SK = ORDER#2026-08-11#9002
SK = ORDER#2026-08-12#9003
Now:
PK = MERCHANT#M123
can retrieve that merchant's order collection.
The timestamp in the sort key can help support chronological/range-oriented access.
GSI: another access path
Suppose our base table is optimized for:
Get an order by orderId
but another requirement is:
Get orders by merchant and date
A Global Secondary Index (GSI) can use a different partition and sort key from the base table.
Conceptually:
Base table
PK = ORDER#9001
while:
GSI
PK = MERCHANT#M123
SK = 2026-08-10#ORDER#9001
Same underlying item.
Different access path.
The key lesson is:
DynamoDB schema design is strongly driven by how the application needs to retrieve data.
That is a different mental model from designing relational tables first and later adding indexes to optimize queries.
Eventual consistency: correct doesn't always mean immediately visible everywhere
Suppose payment succeeds:
ORDER#9001
status = PAID
Immediately afterward, another component reads through a GSI.
Could it briefly observe older data?
Yes.
DynamoDB table/LSI reads can use eventual or strong consistency, while GSI reads are eventually consistent.
Simplified:
Write:
status = PAID
↓
immediate eventual read
↓
possibly PENDING briefly
↓
later read
↓
PAID
This doesn't mean the write disappeared.
It reflects the consistency guarantee of that read path.
For an immediately authoritative order confirmation, we may prefer reading the exact order from the base table using the appropriate consistency requirement instead of assuming every secondary view has already converged.
Another distributed-systems lesson:
Correct state and immediate visibility of that state everywhere are different guarantees.
Problem #8: Every Lambda should not have permission to everything
Our architecture now contains several functions:
CreateOrder Lambda
Inventory Lambda
Payment Lambda
Email Lambda
The easiest IAM policy would be:
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
It would also be a terrible production boundary.
A better architecture uses least privilege.
CreateOrder Lambda
→ write required order data
→ start the workflow
Payment Lambda
→ read/update payment state
→ access payment secret
Email Lambda
→ consume email messages
→ access email secret
Lambda uses execution roles to determine what AWS resources a function can access.
Each component should receive the permissions required for its responsibility, not unrestricted authority over the whole application.
For example:
Email Lambda
CAN:
✓ consume email queue
✓ read email-provider credentials
CANNOT:
✗ modify payment records
✗ read payment secrets
✗ delete orders
That reduces the blast radius if one component is compromised.
Secrets are not configuration strings in Git
This should never happen:
const PAYMENT_API_KEY = "live-secret-key";
inside source control.
Sensitive credentials can be stored in AWS Secrets Manager and accessed only by functions whose IAM role allows it.
Payment Lambda
↓
Secrets Manager
↓
payment-provider credentials
Meanwhile:
Email Lambda
does not need permission to read those credentials.
Authorization boundaries matter just as much as network/application boundaries.
And don't leak the secret through logs
This:
console.log({
cardNumber,
cvv,
paymentToken
});
creates a new security problem.
Prefer operational context:
{
"orderId": "ORD-9001",
"paymentAttemptId": "PAY-72",
"status": "FAILED",
"errorType": "PROVIDER_TIMEOUT"
}
which takes us to our final production concern.
Problem #9: Order 9001 failed. Where?
Our order can now cross:
API Gateway
↓
Lambda
↓
DynamoDB
↓
Step Functions
↓
Inventory Lambda
↓
Payment Lambda
↓
SQS
↓
Email Lambda
A customer says:
“My order failed.”
Which component failed?
Without observability, debugging a distributed system becomes archaeology.
Logs: what happened?
Instead of:
console.log("payment failed");
prefer structured logs:
{
"correlationId": "REQ-ABC-123",
"orderId": "ORD-9001",
"service": "payment",
"status": "FAILED",
"errorType": "PROVIDER_TIMEOUT"
}
AWS Powertools for Lambda includes utilities for structured logging, tracing and metrics.
Correlation IDs: connect the story
Carry an identifier across components:
REQ-ABC-123
|
API Gateway
|
CreateOrder
|
Step Functions
|
Payment
|
Email
Now logs across several services can be correlated to the same business operation.
Metrics: is the system healthy?
Useful metrics might include:
Lambda error rate
Lambda duration
Lambda throttles
Concurrent executions
Payment failure rate
SQS queue depth
Oldest message age
Failed Step Functions executions
Workflow duration
Orders stuck in PAYMENT_PENDING
Notice the last one.
Observability isn't only:
technical metrics
It also needs:
business metrics
Your infrastructure could technically be healthy while thousands of orders remain stuck in an invalid business state.
Alarms: when should humans care?
Dashboards are useful.
But humans shouldn't have to stare at them continuously.
Examples:
Payment failures suddenly > normal threshold
SQS oldest message > 5 minutes
Lambda throttling increasing
Workflow failures spike
Those signals should trigger alarms and operational action.
Step Functions execution history: observe the business workflow
One advantage of explicit orchestration is that we can inspect a workflow execution:
CreateOrder ✅
ReserveInventory ✅
ProcessPayment ❌
Retry ❌
Retry ✅
UpdateOrder ✅
QueueEmail ✅
Instead of reconstructing the entire process from Lambda-to-Lambda calls, we can see the state-machine execution itself.
Traces: where did the request spend time?
Suppose checkout completes but takes eight seconds.
Logs tell us that everything executed.
Metrics tell us latency increased.
Tracing helps answer:
API Gateway 40 ms
CreateOrder 90 ms
DynamoDB 12 ms
Inventory 140 ms
Payment 6.8 sec ← bottleneck
Lambda integrates with AWS X-Ray, which can produce service maps and searchable traces for diagnosing errors and latency bottlenecks.
So a useful observability mental model is:
Logs
→ What happened?
Metrics
→ Is the system healthy overall?
Traces
→ Where did this request spend its time?
The architecture we ended up with
We started here:
API Gateway
↓
Lambda
↓
DynamoDB
After introducing real production requirements, we arrived at something closer to:
Client
|
v
API Gateway
|
v
CreateOrder Lambda
|
v
DynamoDB
|
v
Step Functions
/ | \
/ | \
v v v
Inventory Payment Other Tasks
| |
| Retry / Catch
| |
+---- Saga compensation
|
Success
|
v
SQS
|
v
Email Lambda
Surrounding that application:
IAM
→ least-privilege permissions
Secrets Manager
→ credentials
CloudWatch
→ logs, metrics, alarms
X-Ray / tracing
→ request path and latency
Correlation IDs
→ connect one order across components
The interesting part is that we didn't start by saying:
“Let's use API Gateway, Lambda, DynamoDB, SQS, Step Functions and ten other AWS services.”
Every component appeared because a requirement exposed a limitation in the simpler design.
We needed asynchronous side effects.
→ SQS.
Retries introduced duplicate operations.
→ Idempotency.
The process became a business workflow.
→ Step Functions.
Distributed operations couldn't share one rollback.
→ Saga and compensation.
Independent work didn't need serial execution.
→ Parallel and Map.
Automatic scaling threatened downstream systems.
→ Concurrency controls and queues.
DynamoDB needed multiple efficient read patterns.
→ PK/SK/GSI design.
Distributed reads did not always become visible simultaneously.
→ Consistency awareness.
More components increased security exposure.
→ Least privilege and secrets management.
More components made failures harder to diagnose.
→ Observability.
The serverless lesson I would keep
Serverless is often introduced as:
“You don't manage servers.”
That's true, but it is only the beginning.
You still have to reason about:
failure
retries
duplicates
idempotency
concurrency
downstream capacity
distributed transactions
compensation
data access patterns
consistency
permissions
security
observability
AWS can manage servers for us.
AWS cannot decide our business guarantees for us.
And perhaps the most useful architectural question throughout this entire example was not:
Which AWS service should I use?
It was:
What problem does my current simple design fail to solve next?
That question is what turns a collection of serverless services into an architecture.
Top comments (0)