How I built an event-driven order pipeline that reacts to every database write in real time — using API Gateway, DynamoDB Streams, two purpose-built Lambdas, and SNS.
Disclaimer: this is a learning build, not a production-hardened system. Test thoroughly before adapting any part of this to a real workload.
Most "real-time" systems people build when they're starting out actually aren't. They poll. A cron job checks the database every 30 seconds, a script diffs the last snapshot, something eventually notices a new row exists. It works, but it's slow, wasteful, and falls apart the moment you need sub-second reactions.
Change Data Capture (CDC) flips that model. Instead of asking "did anything change?" on a timer, your database itself tells you the instant something changes — insert, update, or delete — and your infrastructure reacts automatically. No polling, no cron jobs, no wasted invocations.
I built a small but complete CDC pipeline on AWS to see this pattern end-to-end: an order comes in through an API, gets written to DynamoDB, and the moment it lands, a separate Lambda picks up the change from DynamoDB Streams and fires off a notification — all without either Lambda knowing the other exists.
The Architecture
curl POST /order (API key)
│
USER
"Order 999 placed successfully"
┌───────────────────────────────────────────────────────────────────┐
│ AWS Cloud / Region │
│ │
│ API Gateway ──① feeds the table──▶ Lambda·Ingest ──▶ DynamoDB │
│ POST /order · API key writes + answers orders │
│ 50 req/s · 1,000/day fast on-demand │
│ │ │
│ │ ② listens │
│ ▼ │
│ SNS ◀────────────── Lambda·Processor ◀── Streams │
│ fans it out auto-triggered every change │
│ │ by Streams → event │
│ ▼ │
│ your email │
│ "New Order Alert: Gaming..." │
│ │
│ IAM: ingest → write | processor → read + SNS (least privilege) │
│ CloudWatch: all logs, one place │
└───────────────────────────────────────────────────────────────────┘
Two Lambdas, two IAM roles, one DynamoDB table with Streams enabled, and SNS on the other end. That's the whole pipeline — and each piece only knows about its immediate neighbor.
How It Works
Ingest Service (Lambda A + API Gateway) — receives order data via HTTP POST and writes it straight to DynamoDB. Nothing else. It doesn't know or care what happens after.
The Database (DynamoDB + Streams) — stores the order, and with Streams enabled, captures every INSERT, UPDATE, and DELETE as a discrete event the moment it happens.
Event Processor (Lambda B) — auto-triggered by the stream. Reads the change event, pulls out the new order details, and publishes a notification. It never talks to API Gateway or the client directly.
Notification Service (SNS) — fans the message out to an email subscriber (or, easily, to Slack/SMS/another Lambda) the moment a new order lands.
The key design idea: ingest and notification are fully decoupled. The ingest Lambda's only job is to respond to the client fast. It has zero knowledge that a downstream processor even exists — DynamoDB Streams is what bridges the two.
Step 1 — DynamoDB Table with Streams Enabled
Table name: cdc-orders
Partition key: order_id (String)
Capacity mode: On-demand
Once the table exists, the important part happens under Exports and streams:
Turn on DynamoDB stream details
View type: New and old images — this captures both what changed and what it changed from, which matters if you ever want to build diff-based logic later.
This one toggle is what turns a plain table into an event source.
Step 2 — Two IAM Roles, Not One
It would be easy to give both Lambdas the same broad role. I deliberately split them:
iam_role_lambda_ingest — Write only
AmazonDynamoDBFullAccessAWSLambdaBasicExecutionRole
iam_role_lambda_processor — Read stream + notify
-
AWSLambdaDynamoDBExecutionRole(stream read) AmazonSNSFullAccessAWSLambdaBasicExecutionRole
The ingest Lambda can never publish to SNS. The processor Lambda can never directly write new orders. If either function were ever compromised or misconfigured, the blast radius is contained to exactly what that function needs to do — nothing more.
Step 3 — The Ingest Lambda
This is the front door of the pipeline — the only function the client ever talks to. It receives the POST body from API Gateway (Lambda proxy integration), writes the order straight to DynamoDB, and responds immediately. It has zero awareness of Streams, the processor, or SNS — that separation is deliberate.
import json
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('cdc-orders')
def lambda_handler(event, context):
try:
body = json.loads(event['body'])
order_id = body['order_id']
item = body['item']
price = body['price']
table.put_item(
Item={
'order_id': order_id,
'item': item,
'price': price
}
)
return {
'statusCode': 200,
'body': json.dumps(f'Order {order_id} placed successfully!')
}
except Exception as e:
print(f"Error processing order: {str(e)}")
return {
'statusCode': 500,
'body': json.dumps('Failed to place order')
}
Two things worth calling out:
json.loads(event['body']) is required because with Lambda proxy integration, API Gateway hands the entire HTTP request to the function as an event object — the actual JSON payload arrives as a string inside event['body'], not as a pre-parsed dict.
This function does nothing beyond the write. No SNS call, no stream check, no notification logic. That's not a limitation — it's the whole point of CDC. The ingest path stays fast and simple, and every downstream reaction (email alerts, future Slack alerts, audit logging) hangs off DynamoDB Streams instead of being bolted onto this function.
Step 4 — The Processor Lambda
This is the piece that actually reacts to change events. Deployed with Python 3.14, triggered directly by the DynamoDB stream (batch size 1, starting position Latest — I want new events only, not a replay of history):
import json
import boto3
sns = boto3.client('sns')
TOPIC_ARN = "" # your SNS topic ARN goes here
def lambda_handler(event, context):
for record in event['Records']:
if record['eventName'] == 'INSERT':
new_image = record['dynamodb']['NewImage']
order_id = new_image['order_id']['S']
item = new_image.get('item', {}).get('S', 'Unknown Item')
message = f"🆕 NEW ORDER RECEIVED!\nID: {order_id}\nItem: {item}"
response = sns.publish(
TopicArn=TOPIC_ARN,
Message=message,
Subject="New Order Alert"
)
print(f"Alert sent for Order {order_id}")
return {
'statusCode': 200,
'body': json.dumps('Successfully processed DynamoDB Stream records')
}
Notice this function has no idea an API Gateway or an ingest Lambda even exists. It only knows: "a stream record arrived, was it an INSERT, pull the fields I care about, publish." That's the whole point of CDC — each consumer of the stream can be added or removed independently without touching the write path.
Step 5 — Exposing Ingest via API Gateway
REST API, Regional endpoint, name: CDC-Shop-API
Resource: /order
Method: POST, Lambda proxy integration enabled, pointed at the ingest function
Deployed to a prod stage
Step 6 — Locking It Down with a Usage Plan + API Key
This was the part I almost skipped, and it's the part that actually makes the API safe to hand out. A usage plan (cdc-usage-plan) sits in front of the stage:
Throttling: 50 requests/second, burst 50
Quota: 1,000 requests/day
Associated to CDC-Shop-API → prod stage, and tied to an API key (CDC Key)
Without an API key requirement on the method and a usage plan behind it, anyone with the invoke URL could hammer the endpoint indefinitely. This is a five-minute step that's easy to forget under a tutorial's momentum, and it's the difference between a demo and something you'd actually expose.
One gotcha worth calling out explicitly: the ARN shown on the method execution page is not your invoke URL. It looks superficially similar and it's tempting to paste it straight into curl, but ARNs are for IAM policies, not HTTP calls. The real invoke URL only shows up on the Stages page after deployment, in this format:
https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/{resource}
Testing End to End
curl -X POST https://abc123xyz.execute-api.us-east-1.amazonaws.com/prod/order \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_API_KEY_HERE" \
-d '{"order_id": "999", "item": "Gaming Laptop", "price": "1500"}'
Immediate response:
Order 999 placed successfully!
Within 1–5 seconds, an email lands:
🆕 NEW ORDER RECEIVED!
ID: 999
Item: Gaming Laptop
And a quick check in DynamoDB (cdc-orders → Explore table items) confirms the record landed with order_id: 999, item: Gaming Laptop, price: 1500 — proving the whole chain, from HTTP request to email, actually fired off a single write.
Lessons From This Build
Decoupling isn't just a buzzword here — it's what makes the pipeline extensible. Because the processor only reacts to stream events, I can bolt on a second consumer (Slack alerts, an audit log writer, a cache invalidation function) without touching the ingest Lambda or the API at all. Streams effectively turn one write into as many downstream reactions as I want.
Batch size 1 was a deliberate choice for this demo, not a default to keep. At batch size 1, the processor runs once per record, which is easy to reason about and debug. In a real high-throughput system, you'd increase batch size to reduce Lambda invocation count and cost — but you accept slightly more complex error handling per invocation as a result.
Least-privilege IAM roles caught a mistake early. While wiring this up, I originally pointed the wrong role at the processor function and it failed silently on stream reads until CloudWatch logs made the missing permission obvious. Splitting the roles didn't just improve security — it made the failure mode immediately diagnosable instead of a vague "nothing happens" symptom.
The usage plan step is the one people skip, and it's the one that matters most before sharing a demo publicly. A working pipeline with no throttling is one bad script away from a surprise bill or a self-inflicted denial of service.
Real-World Use Cases
This exact pattern extends directly to:
Order processing and fulfillment notifications
Real-time data synchronization between systems
Audit logging and compliance tracking
Cache invalidation the instant underlying data changes
Analytics and reporting pipelines that need to react, not poll
I hope you enjoyed reading this article. Please feel free to write @awais684@gmail.com | LinkedIn https://www.linkedin.com/in/awais684 for any queries on AWS/DevOps & stay tuned for the next write-up.
If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇
Top comments (0)