Most serverless articles end at the architecture diagram. API Gateway in front, Lambda in the middle, DynamoDB at the back, a queue or two, and the system is "done." I have spent the last five years running systems like that for products used by millions of students, and the diagram was never the hard part. The hard part is everything that happens after launch, when retries, cold starts, throttles and bills show up at the same time.
These are the six decisions I now make before a serverless system ships, because each one was painful to make afterwards.
1. Every asynchronous handler is idempotent
Lambda invoked from SQS, EventBridge, SNS or a DynamoDB stream is at-least-once. The same event will arrive twice. Not often, but at the worst time, and usually when a downstream service is slow and your function timed out after doing half its work.
The fix is boring and it has to be everywhere: an idempotency key per event, stored before the side effect happens. For DynamoDB that is a conditional write on the key. For anything else it is a small table of processed keys with a TTL. AWS Lambda Powertools has an idempotency utility that does exactly this, and it is worth adopting on day one rather than retrofitting it into forty functions later.
If a handler cannot be made idempotent, it needs to be the only consumer of a FIFO queue with a deduplication ID. That is a design constraint, so decide it early.
2. A dead-letter queue on every asynchronous path, with an alarm
An asynchronous Lambda without a dead-letter queue drops failed events on the floor after its retries are exhausted. You will not know. The event is simply gone.
Every SQS queue gets a redrive policy to a DLQ. Every asynchronous Lambda invocation gets an on-failure destination. Every DLQ gets a CloudWatch alarm on ApproximateNumberOfMessagesVisible greater than zero, routed to the on-call channel. The alarm matters as much as the queue. A DLQ nobody watches is a slower way of losing data.
3. Concurrency limits protect the things that cannot scale
Lambda scales faster than almost everything it talks to. A traffic spike that Lambda absorbs happily becomes a connection storm against a relational database, a throttled third-party API, or a downstream service owned by another team.
Reserved concurrency on the function that talks to the fragile thing is the simplest brake. It turns a downstream outage into a queue backlog, which is recoverable, instead of a cascade, which is not. For the small number of functions on the synchronous request path where cold starts are visible to users, provisioned concurrency is worth its cost. For everything else, it is not.
4. Observability is a correlation ID and four alarms
Logs from a hundred functions are useless without a way to follow one request through them. Every function logs structured JSON with a correlation ID that comes in on the request or event and goes out on everything it emits. Powertools does this too. Without it, debugging a production issue is grep across a hundred log groups.
Then four alarms per function, no more to start with: error rate, p95 duration against the timeout, throttles, and for stream or queue consumers, iterator age or oldest message age. Those four catch almost everything. Dashboards are for understanding; alarms are for waking up, and they should be rare enough that every one is acted on.
5. The bill is decided by three settings
Serverless is cheap by default and expensive by accident. The three settings that decide most of the bill:
- Log retention. CloudWatch log groups default to never expire. Set a retention period on every group, in infrastructure code, from the start.
- Memory size. Lambda bills memory multiplied by duration, and more memory often means a shorter duration and a lower cost. Run AWS Lambda Power Tuning once per function and pick the cheapest setting that meets the latency target, instead of guessing 1024 MB.
- DynamoDB capacity mode. On-demand for spiky or unknown traffic, provisioned with auto scaling for steady traffic. Switching later is allowed, so start with on-demand and look at the numbers after a month.
6. A runbook exists before the first alarm fires
Every alarm links to a runbook that says what the alarm means, what to check first, and what the safe actions are. It does not need to be long. "DLQ depth above zero: open the queue, read one message, check the function's error logs by correlation ID, fix or redrive" is enough.
The runbook is also where the team's knowledge stops living in one person's head. On a team of six to eight engineers with a mix of junior and senior people, the runbook is what lets a junior engineer handle a Tuesday night alarm without paging me.
What I would tell my past self
None of this is clever. All of it is the difference between a serverless system that is quiet in production and one that generates a steady stream of mysteries. If you are about to ship one, do these six things in the infrastructure code before the first deploy, because adding them later means touching every function and relearning every lesson at 2 a.m.
Top comments (0)