DEV Community: Renaldi

Cost-Aware Serverless Architecture Reviews: A Practical Framework

Renaldi — Sun, 12 Apr 2026 23:00:00 +0000

Cost optimization in serverless is often discussed as a pricing exercise. In practice, I have found it is much more effective when treated as an architecture review discipline.

That is the lens I use in this post.

This is a practical framework I use to review serverless systems for cost efficiency without weakening reliability, operability, or delivery speed. It is designed for engineers, architects, and technical leads who want to go beyond “reduce Lambda memory” and make better design decisions across services.

I will focus on AWS serverless building blocks and the tradeoffs between:

AWS Lambda
AWS Step Functions
Amazon EventBridge Pipes
Direct service integrations (for example, Step Functions SDK/service integrations)
Supporting services like SQS, EventBridge, CloudWatch Logs, DynamoDB, and S3

I will cover:

Cost hotspots in serverless (invocations, state transitions, logs, data transfer)
Architecture changes that reduce cost without harming reliability
Tradeoffs: Lambda vs Pipes vs service integrations vs Step Functions
Workload profiling and right-sizing patterns
End-to-end walkthrough and implementation discussion
Architecture and code examples you can adapt

Why this topic stands out

I like this topic because it is senior-level but immediately useful.

Almost every serverless system starts cost-efficient, then gradually accumulates “cost drag” as features are added:

Extra orchestration steps
Helper Lambdas that only transform payloads
Verbose logging left on in production
Duplicate event fan-out and reprocessing
Cross-AZ / cross-region / internet egress paths that nobody revisits
Overbuilt retries and polling loops

None of these are necessarily “wrong.” In many cases they are the fastest way to ship.

But once traffic grows, it becomes worth doing a structured review.

The goal is not to squeeze every cent out of the system. The goal is to spend intentionally, with a clear understanding of where cost comes from and what reliability/maintainability benefits we are buying with that spend.

What I mean by a cost-aware serverless architecture review

A cost-aware review is a design review that asks four questions for each path in the workload:

What is the unit of work?
- One event, one request, one file, one state transition, one batch, one customer action
What services does that unit of work touch?
- Lambda, Step Functions, EventBridge, SQS, DynamoDB, logs, storage, network paths
Which of those touches scales linearly with traffic, and which scales with payload size, duration, or retries?
- This is where the real hotspots show up
Can I reduce cost by changing the architecture instead of only tuning a single service?
- Example: removing three “glue Lambdas” with native integrations

That last question is the most important one.

Framework Overview

This is the practical framework I use in reviews:

Phase 1: Profile the workload

Understand traffic shape and execution shape:

request rate, concurrency, burstiness
payload sizes
durations
retries
fan-out counts
error rates
cold-start sensitivity
async vs sync paths

Phase 2: Map the cost path

Trace the cost of one unit of work across services:

invocations
transitions
messages
logs ingested/stored
storage reads/writes
data transfer
optional downstream analytics/monitoring copies

Phase 3: Identify hotspots

Rank cost drivers by:

total monthly impact
growth rate with scale
ease of remediation
risk of change

Phase 4: Apply architecture changes

Prefer changes that:

remove unnecessary compute hops
reduce chatty orchestration
batch operations safely
right-size compute from measured data
preserve observability and failure isolation

Phase 5: Re-measure

I treat cost optimization as an engineering loop:

baseline
change
compare
document tradeoffs
repeat

End-to-End Walkthrough

To make this concrete, I will walk through a realistic example I often see:

Example workload: Event-driven order processing pipeline

A retail platform emits an OrderCreated event. The serverless pipeline:

Validates the payload
Enriches customer metadata
Writes the order to DynamoDB
Sends a fulfillment request to an internal API
Publishes notifications
Updates operational dashboards
Triggers async follow-up workflows

Baseline architecture (common version)

A common initial design looks like this:

EventBridge rule receives OrderCreated
Step Functions orchestrates the flow
Multiple Lambda functions perform:
- input validation
- JSON transformation
- service API calls
- retries / status polling
- notification dispatch
Every function logs full payloads
Step Functions uses many small states to represent each transformation
Synchronous calls are used where async could work
Duplicate events are emitted for multiple downstream consumers

This is usually functional and maintainable, but over time I see four cost hotspots appear.

Cost hotspots in serverless (what I look for first)

1) Invocation-driven hotspots (Lambda, messaging, API calls)

This is the most obvious one, but not always where the biggest savings are.

I look for:

High-frequency helper Lambdas that do only simple mapping/filtering
Repeated invocations caused by retries at multiple layers (client, service, workflow)
Tiny per-item processing when batching is safe
Polling loops implemented in Lambda instead of orchestration or event callbacks

Common anti-pattern

A Lambda is invoked just to:

rename fields
add a constant
route based on one attribute
pass the payload to another service

In many cases this can be replaced with:

EventBridge input transformation
EventBridge Pipes filtering/enrichment
Step Functions Parameters / ResultSelector
Step Functions direct service integrations

That removes both compute cost and operational surface area.

2) State transition hotspots (Step Functions)

I see this frequently in mature systems.

Step Functions is excellent, but cost can grow when workflows become overly chatty:

One state per trivial transformation
Frequent polling loops with short waits
Repetitive branching for small decisions
Using Standard workflows for high-volume, very short paths without reviewing fit
Nested workflows for steps that could be in one integration

What I optimize carefully

I do not optimize by blindly collapsing everything into one Lambda. That usually hurts observability and retry control.

Instead, I look for:

eliminating trivial pass/transform states
replacing glue Lambdas with direct integrations
increasing polling intervals or switching to callback/event-driven completions
splitting hot short-lived paths from long-running orchestration

3) Logging hotspots (CloudWatch Logs)

This one is often underestimated because logging is “cheap enough” until scale increases.

Cost growth comes from:

Logging full request/response payloads for every invocation
Debug-level logs enabled permanently
Duplicate logs across Lambda, workflow, and application layers
Large structured objects written multiple times
High-cardinality logs that are rarely queried

Practical rule I use

I keep logs useful and minimal in production:

log identifiers, not entire payloads
sample verbose payload logs
gate debug logs by environment/feature flag
separate audit records from debug logs
set explicit retention periods

In serverless, “just logging more” can become a real cost driver.

4) Data transfer hotspots (often missed in reviews)

Data transfer is easy to miss because the architecture diagram looks “serverless,” but network paths still matter.

I review:

Cross-region calls between Lambda and downstream services
Internet egress to SaaS APIs
Repeated large payload movement between services
Download/re-upload patterns (for example, pulling S3 objects through Lambda when direct processing is possible)
VPC-attached Lambda traffic patterns if applicable (including NAT-related patterns)
Sending oversized event payloads when a pointer (S3 key/object reference) would work

Pattern I prefer

Move references more often than full payloads:

Put large content in S3
Pass object keys / IDs in events
Fetch only where necessary
Trim event payloads to fields required for the next step

This reduces both transfer and execution overhead.

A Practical Review Framework I Use in Real Projects

Below is the review structure I use during architecture reviews. It is intentionally simple so teams can repeat it.

Step 1: Define the unit economics

I start with a unit of work, for example:

“One OrderCreated event processed successfully”
“One failed event retried and sent to DLQ”
“One batch of 100 events processed”

Then I estimate or measure the service touches for that unit.

Example (baseline, illustrative)

For one order:

EventBridge ingress: 1 event
Step Functions states: 22 transitions
Lambda invocations: 8
DynamoDB writes: 2
CloudWatch log lines: ~120
One external API call
One notification event
Occasional retries on fulfillment call

The exact numbers vary, but this gives me a concrete path to improve.

Step 2: Build a cost-path table (without overcomplicating it)

I do not start with exact pricing spreadsheets. I start with a ranked table:

Driver (Lambda, Step Functions, logs, transfer, etc.)
Scaling factor (requests, duration, payload size, retries)
Observed value
Risk/benefit if changed
Optimization candidate

This quickly identifies which optimizations are architectural vs micro-tuning.

Step 3: Profile the workload (shape matters more than averages)

Averages hide the truth in serverless systems.

I profile:

p50/p95/p99 duration
payload size distribution
retry rate by dependency
burst concurrency
fan-out ratio
failure modes (timeouts, throttles, validation errors)
cold-start frequency (only when relevant to SLOs)

Why this matters

A system with the same monthly volume can have very different costs if:

one workload is smooth and small-payload
the other is spiky, retry-heavy, and large-payload

Architecture decisions should follow the shape, not just total count.

Step 4: Apply architecture changes in this order

This is the sequence I use because it usually yields high value with lower risk:

Remove unnecessary service hops
Replace glue compute with native integrations
Batch where safe
Right-size Lambda from measured duration/memory
Tune logs and retention
Revisit orchestration granularity
Optimize transfer/payload strategy
Only then consider deeper redesigns

Architecture Changes That Reduce Cost Without Harming Reliability

Here are the changes I most commonly recommend, with the tradeoffs explained.

1) Replace glue Lambdas with service integrations

Before

A Lambda receives a payload and only:

transforms JSON
calls DynamoDB / SQS / EventBridge
returns a small result

After

Use:

Step Functions AWS SDK/service integrations
EventBridge input transformers
EventBridge Pipes filter/enrichment steps (where appropriate)

Why this helps

Removes invocation and duration cost
Reduces operational burden (fewer deployments, alarms, IAM roles)
Improves determinism for simple operations

Reliability note

This can improve reliability because fewer components means fewer failure points. The key is preserving observability and explicit error handling.

2) Use Pipes for simple event movement and filtering

EventBridge Pipes is a strong fit when the path is mostly:

source -> filter -> optional transform/enrichment -> target

Good fit examples

SQS -> target Lambda (with filtering)
DynamoDB stream -> EventBridge bus
Kinesis -> SQS
SQS -> Step Functions (selective forwarding)

When I do not force Pipes

If the workflow needs:

complex branching
human approvals
multi-step retries with business semantics
long waits
compensation logic

then Step Functions is usually the better fit, even if the raw “per event movement” cost might not be the lowest.

3) Reduce Step Functions chatty orchestration

What I change

Collapse trivial transform-only states
Use Parameters, ResultSelector, and JSONPath/JSONata features instead of helper Lambdas
Shift from polling to callback/event completion where feasible
Split short high-volume flows from long-running exception flows

What I keep

Distinct states for failure boundaries
States that need targeted retries/catches
Business-visible milestones (auditability)
Human review / approval checkpoints

The goal is not “fewer states at any cost.” The goal is intentional state design.

4) Batch small units where the business semantics allow it

Batching can reduce cost significantly across:

Lambda invocations
downstream API calls
logs
orchestration overhead

Examples

Process 10 records per Lambda invocation instead of 1
Aggregate notifications
Batch writes to DynamoDB when access patterns permit
Buffer events briefly for downstream efficiency

Caution

Batching changes failure semantics. I only use it when I can answer:

What happens if item 7 of 10 fails?
Do I need partial success reporting?
Can the downstream system handle duplicates on retries?

Reliability comes first. Cost savings that break recovery are not real savings.

5) Right-size Lambda with measured data, not guesswork

Right-sizing is more than memory reduction. Since Lambda allocates CPU proportionally to memory, a lower-memory function can become slower and more expensive overall.

What I measure

p50/p95 duration
memory usage
init duration (where relevant)
timeout headroom
downstream latency contribution
retry behavior after timeouts/throttles

What I look for

Functions with high duration and low memory usage (often downstream-bound)
Functions with CPU-bound work that benefit from more memory
Functions with large package sizes causing slower init
Functions using /tmp heavily (may benefit from memory/storage review)

I optimize for cost per successful unit of work, not cost per invocation in isolation.

6) Tune logging intentionally (not minimally)

I do not recommend “turn off logs.” I recommend structured, useful, cost-conscious logging.

Practical changes:

Move full payload dumps behind a sampled debug flag
Log IDs and metrics by default
Use explicit log retention (for example, shorter for noisy operational logs)
Avoid duplicate payload logging at every layer
Emit metrics for counts/latency instead of reconstructing them from logs

This usually reduces cost while improving signal-to-noise.

7) Pass references, not large payloads

Instead of pushing large documents or objects through multiple services:

Store payloads in S3
Pass a pointer in the event/workflow
Fetch on demand

Benefits:

lower transfer volume
smaller event payloads
faster orchestration payload handling
cleaner replayability

This pattern is especially helpful in document, media, and analytics workflows.

Tradeoffs: Lambda vs Pipes vs Service Integrations vs Step Functions

This is the part teams usually want summarized, so here is the practical version I use in reviews.

Lambda is best when:

I need custom code/business logic
I need third-party SDKs or protocols
I need non-trivial validation/transformation
I need specialized retry behavior inside the function
I need to encapsulate logic reused across multiple flows

Lambda is overkill when:

I am only mapping fields
I am just forwarding to another AWS service
I can use native service integration safely and clearly

EventBridge Pipes is best when:

The path is primarily source-to-target movement
I need filtering and simple enrichment
I want low operational overhead for plumbing
I want to avoid writing a “routing Lambda”

Pipes is not the best fit when:

I need complex multi-step orchestration
I need long waits or human callbacks
I need rich compensation logic across steps

Step Functions is best when:

I need explicit orchestration and auditability
I need retries/catches at step boundaries
I need long-running workflows
I need human-in-the-loop or callback patterns
I need clear operational visualization for a business process

Step Functions can become costly when:

I model every trivial transform as a state
I use frequent polling instead of event-driven completion
I keep hot paths in Standard without reviewing alternatives
I use nested workflows where direct integrations would do

Direct service integrations are best when:

The step is simple and deterministic
I can express the call declaratively
I want fewer moving parts
I want to remove glue code and reduce operational burden

Direct integrations may be a poor fit when:

I need complex custom serialization/validation
I need cross-service custom logic not represented well declaratively
The readability of the workflow would suffer too much

End-to-End Optimization Walkthrough (Baseline -> Improved)

Now I will apply the framework to the example order pipeline.

Baseline (illustrative)

8 Lambda invocations per order
22 Step Functions transitions
verbose logs in every function
large payload passed between multiple steps
polling Lambda for fulfillment status
duplicate notification paths

Review findings

Top cost hotspots:

Step Functions state transitions from chatty orchestration
Lambda invocations for glue transforms and simple service calls
CloudWatch Logs ingestion due to payload dumps
Data transfer/execution overhead from oversized payloads
Retries multiplying costs during downstream API flakiness

Improved architecture (same reliability goals)

Changes I make

Replace 3 glue Lambdas with:
- Step Functions Parameters / ResultSelector
- direct DynamoDB/EventBridge integrations
Replace one routing Lambda with EventBridge Pipes
Move large enrichment blobs to S3 and pass object references
Replace polling Lambda loop with callback/event-driven completion
Keep Step Functions states at clear failure boundaries only
Reduce default log verbosity and set retention explicitly
Batch non-critical notifications

What I do not change

Idempotency controls
DLQs and retry policies
Key audit checkpoints
Alarms and traces for critical dependencies

This is important. I want lower cost without creating an opaque, fragile pipeline.

Architecture pattern (review-ready version)

The optimized pattern I recommend for many teams looks like this:

Ingress: EventBridge
Plumbing/filtering: EventBridge Pipes (where simple)
Orchestration: Step Functions for business process + callbacks
Compute: Lambda only for genuine custom logic
Storage: S3 for large payloads, DynamoDB for metadata/state
Observability: metrics first, targeted logs, traces on critical paths

This is cost-aware because each service is doing the kind of work it is best suited for.

Implementation Discussion

Below are code examples that demonstrate how I apply the framework in practice.

1) Step Functions workflow using service integrations instead of glue Lambdas (ASL fragment)

This fragment shows a flow that:

validates and prepares payload once
writes to DynamoDB via a direct integration
emits an event via EventBridge integration
keeps Lambda only for actual custom logic

{
  "Comment": "Cost-aware serverless orchestration example",
  "StartAt": "ValidateAndEnrich",
  "States": {
    "ValidateAndEnrich": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${ValidateAndEnrichFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Next": "PutOrderMetadata",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 2,
          "BackoffRate": 2.0,
          "MaxAttempts": 4
        }
      ]
    },
    "PutOrderMetadata": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:putItem",
      "Arguments": {
        "TableName": "${OrdersTable}",
        "Item": {
          "pk": { "S.$": "States.Format('ORDER#{}', $.orderId)" },
          "sk": { "S": "META" },
          "status": { "S": "RECEIVED" },
          "customerId": { "S.$": "$.customerId" },
          "payloadRef": { "S.$": "$.payloadRef.s3Uri" },
          "createdAt": { "S.$": "$.timestamps.receivedAt" }
        },
        "ConditionExpression": "attribute_not_exists(pk)"
      },
      "Next": "PublishOrderReceived"
    },
    "PublishOrderReceived": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Arguments": {
        "Entries": [
          {
            "Source": "com.example.orders",
            "DetailType": "order.received",
            "EventBusName": "${EventBusName}",
            "Detail": {
              "orderId.$": "$.orderId",
              "customerId.$": "$.customerId",
              "payloadRef.$": "$.payloadRef",
              "correlationId.$": "$.correlationId"
            }
          }
        ]
      },
      "Next": "WaitForFulfillmentCallback"
    },
    "WaitForFulfillmentCallback": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Arguments": {
        "FunctionName": "${CreateFulfillmentRequestFnArn}",
        "Payload": {
          "taskToken.$": "$$.Task.Token",
          "orderId.$": "$.orderId",
          "payloadRef.$": "$.payloadRef",
          "correlationId.$": "$.correlationId"
        }
      },
      "TimeoutSeconds": 900,
      "ResultPath": "$.fulfillment",
      "Next": "FinalizeOrder"
    },
    "FinalizeOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:updateItem",
      "Arguments": {
        "TableName": "${OrdersTable}",
        "Key": {
          "pk": { "S.$": "States.Format('ORDER#{}', $.orderId)" },
          "sk": { "S": "META" }
        },
        "UpdateExpression": "SET #s = :s, fulfillmentId = :f, fulfilledAt = :t",
        "ExpressionAttributeNames": {
          "#s": "status"
        },
        "ExpressionAttributeValues": {
          ":s": { "S": "FULFILLED" },
          ":f": { "S.$": "$.fulfillment.fulfillmentId" },
          ":t": { "S.$": "$.fulfillment.completedAt" }
        }
      },
      "End": true
    }
  }
}

Why this is cost-aware

I removed two common glue Lambdas (put item, publish event)
I use callback instead of chatty polling
I keep orchestration only where it adds reliability/visibility

2) EventBridge Pipes (CDK TypeScript) for low-overhead routing/filtering

This example shows a common “plumbing” path I do not want to solve with a custom Lambda.

import * as cdk from 'aws-cdk-lib';
import * as pipes from 'aws-cdk-lib/aws-pipes';
import * as iam from 'aws-cdk-lib/aws-iam';

const pipeRole = new iam.Role(this, 'OrdersPipeRole', {
  assumedBy: new iam.ServicePrincipal('pipes.amazonaws.com'),
});

pipeRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'sqs:ReceiveMessage',
    'sqs:DeleteMessage',
    'sqs:GetQueueAttributes',
    'sqs:ChangeMessageVisibility',
  ],
  resources: [ordersQueue.queueArn],
}));

pipeRole.addToPolicy(new iam.PolicyStatement({
  actions: ['states:StartExecution'],
  resources: [orderWorkflow.stateMachineArn],
}));

new pipes.CfnPipe(this, 'FilteredOrdersPipe', {
  roleArn: pipeRole.roleArn,
  source: ordersQueue.queueArn,
  target: orderWorkflow.stateMachineArn,
  sourceParameters: {
    sqsQueueParameters: {
      batchSize: 10,
      maximumBatchingWindowInSeconds: 5,
    },
    filterCriteria: {
      filters: [
        {
          pattern: JSON.stringify({
            body: {
              eventType: ['OrderCreated'],
              priority: ['standard', 'express'],
            },
          }),
        },
      ],
    },
  },
  targetParameters: {
    stepFunctionStateMachineParameters: {
      invocationType: 'FIRE_AND_FORGET',
    },
    inputTemplate: JSON.stringify({
      orderId: '<$.body.orderId>',
      customerId: '<$.body.customerId>',
      priority: '<$.body.priority>',
      payloadRef: {
        s3Uri: '<$.body.payloadRef.s3Uri>',
      },
      correlationId: '<$.body.correlationId>',
    }),
  },
});

Why this is cost-aware

No routing Lambda to maintain
Built-in filtering avoids unnecessary workflow starts
Batching reduces per-message overhead while preserving throughput

3) Lambda right-sizing profiling helper (Python)

I use a simple embedded metric approach to log what matters for right-sizing reviews. This is not a full observability solution, but it is enough to drive architecture decisions.

import json
import os
import time
from datetime import datetime, timezone

def emit_emf(namespace: str, metrics: dict, dimensions: dict):
    emf = {
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [
                {
                    "Namespace": namespace,
                    "Dimensions": [list(dimensions.keys())],
                    "Metrics": [{"Name": k, "Unit": "None"} for k in metrics.keys()],
                }
            ],
        },
        **dimensions,
        **metrics,
    }
    print(json.dumps(emf))

def lambda_handler(event, context):
    start = time.perf_counter()

    payload_size_bytes = len(json.dumps(event).encode("utf-8"))

    # ... custom logic here ...
    # Keep this function for genuine business logic, not simple routing

    duration_ms = (time.perf_counter() - start) * 1000
    configured_memory_mb = int(os.getenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "128"))

    emit_emf(
        namespace="ServerlessReview",
        dimensions={
            "FunctionName": context.function_name,
            "Environment": os.getenv("STAGE", "prod"),
        },
        metrics={
            "DurationMs": round(duration_ms, 2),
            "PayloadSizeBytes": payload_size_bytes,
            "ConfiguredMemoryMB": configured_memory_mb,
        },
    )

    return {
        "ok": True,
        "processedAt": datetime.now(timezone.utc).isoformat()
    }

How I use this data in reviews

I compare:

duration by memory setting
duration by payload size bucket
timeout proximity
retry rates correlated with duration

Then I decide whether to:

increase memory (reduce duration and total cost)
decrease memory (if clearly overprovisioned)
batch inputs
move logic out of Lambda entirely

4) Production-safe logging pattern (Python)

This pattern reduces log cost without sacrificing debuggability.

import json
import os
import random

LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO").upper()
DEBUG_SAMPLE_RATE = float(os.getenv("DEBUG_SAMPLE_RATE", "0.01"))  # 1% default

def should_log_debug_payload() -> bool:
    if LOG_LEVEL == "DEBUG":
        return True
    return random.random() < DEBUG_SAMPLE_RATE

def log_info(message: str, **kwargs):
    print(json.dumps({"level": "INFO", "message": message, **kwargs}))

def log_debug(message: str, **kwargs):
    if LOG_LEVEL == "DEBUG":
        print(json.dumps({"level": "DEBUG", "message": message, **kwargs}))

def handler(event, context):
    correlation_id = event.get("correlationId", "unknown")
    order_id = event.get("orderId", "unknown")

    log_info(
        "processing_order",
        correlationId=correlation_id,
        orderId=order_id
    )

    if should_log_debug_payload():
        print(json.dumps({
            "level": "DEBUG_SAMPLED",
            "message": "event_payload_sample",
            "correlationId": correlation_id,
            "orderId": order_id,
            "payload": event
        }))

    # ... business logic ...

    log_info(
        "order_processed",
        correlationId=correlation_id,
        orderId=order_id,
        status="success"
    )

    return {"status": "ok"}

Why I like this pattern

Production logs stay useful and cheaper
I still have sampled payload visibility for troubleshooting
It works well with structured log queries and metrics extraction

Workload Profiling and Right-Sizing Patterns

This is the part that makes the review credible. I do not want architecture recommendations based on intuition alone.

1) Profile by workload class, not just service

I segment workloads into classes, for example:

Latency-sensitive sync API path
Async event processing path
Burst batch path
Low-frequency admin path

Each class has different priorities and cost levers.

Example:

Sync API path may justify higher Lambda memory for lower latency
Async batch path may prioritize batching and reduced transitions
Admin path may not be worth heavy optimization effort

2) Build a “cost to reliability” tradeoff table

For each proposed change, I document:

Change
Expected cost impact
Reliability impact
Operability impact
Rollback plan
Measurement plan

This keeps reviews grounded and avoids “optimize everything” behavior.

Example entries

Replace routing Lambda with Pipe -> lower cost, lower ops burden, neutral reliability, easy rollback
Collapse 5 workflow states into 2 -> moderate cost savings, possible loss of debug granularity, test carefully
Batch writes by 25 -> potentially high savings, changes failure semantics, requires idempotency validation

3) Right-size from p95 and timeout headroom, not p50 alone

A function that looks fine at p50 may:

time out at p99
retry frequently during bursts
create downstream contention that increases total cost

I tune using:

p95/p99 duration
timeout headroom
retry counts
dependency latency distribution

This avoids false savings that increase retries and operational incidents.

4) Profile payload size and serialization overhead

In event-driven systems, large payloads can create hidden costs:

more transfer
longer execution time
larger logs
larger Step Functions state payloads
more memory pressure in Lambda

I explicitly bucket payload sizes and look for outliers.

A common improvement is:

move large optional fields to S3
pass references
fetch lazily only in the steps that need them

5) Profile retries as a cost multiplier

Retries are often the hidden multiplier in “unexpected” serverless cost growth.

I measure:

retry rate per dependency
retry amplification across layers (SDK retry + Lambda retry + workflow retry)
cost of failed attempts vs successful attempts
DLQ rate and replay volume

Then I redesign retry placement:

retry at the layer with the best context
avoid stacking retries unnecessarily
use backoff and circuit-breaking patterns where appropriate

This improves both cost and stability.

Review Checklist I Actually Use

Here is the short version of the checklist I use in architecture reviews.

Hotspots

[ ] Which cost components scale with request count?
[ ] Which scale with duration, payload size, retries, or fan-out?
[ ] Where are we logging too much?
[ ] Where are we moving data unnecessarily?

Architecture fit

[ ] Is Lambda only used where custom logic is truly needed?
[ ] Can any glue Lambda be replaced with Pipes or direct integration?
[ ] Is Step Functions state granularity intentional?
[ ] Are polling loops justified, or can callbacks/events replace them?

Profiling

[ ] Do we have p95/p99 duration data?
[ ] Do we know payload size distribution?
[ ] Do we know retry rate by dependency?
[ ] Do we know burst concurrency patterns?

Reliability guardrails

[ ] Will the optimization change failure semantics?
[ ] Is idempotency preserved?
[ ] Are DLQ/replay paths intact?
[ ] Is observability still sufficient?

Rollout

[ ] Can we canary or phase the change?
[ ] Do we have before/after metrics defined?
[ ] Do we have a rollback path?

Common Mistakes in Cost Reviews

I see these often, so I call them out explicitly.

Mistake 1: Only tuning Lambda memory

Useful, but incomplete. The biggest cost issue may be:

Step Functions transitions
logs
retries
data movement
unnecessary service hops

Mistake 2: Replacing orchestration with one “mega Lambda”

This can reduce one visible cost but create:

worse reliability
harder retries
worse observability
slower delivery velocity

Mistake 3: Optimizing by averages

p50 hides burst pain and retry amplification.

Mistake 4: Ignoring developer productivity

A design that is slightly cheaper but much harder to maintain may cost more overall.

Mistake 5: Removing logs indiscriminately

The right answer is better logging design, not blind log suppression.

Closing Thoughts

A strong cost-aware serverless review is not about chasing the cheapest architecture on paper. It is about making architecture choices that fit the workload, preserve reliability, and keep operating costs proportional to business value.

The framework I use is simple:

Profile the workload
Map the cost path
Find hotspots
Apply architecture changes first
Right-size from measurements
Re-measure and document tradeoffs

If I am reviewing a production serverless system, I want the outcome to be more than a list of tips. I want a clear plan that the team can implement safely, measure, and iterate on.

That is what makes the review practical.

References

(Consolidated at the end per request)

AWS Lambda pricing, performance tuning guidance, and memory/CPU behavior documentation
AWS Step Functions pricing and workflow design guidance (Standard vs Express)
AWS Step Functions service integrations and callback/task token patterns
Amazon EventBridge and EventBridge Pipes documentation (filtering, enrichment, targets)
Amazon SQS retry, DLQ, and batching guidance
Amazon DynamoDB write/read patterns and idempotency design guidance
Amazon CloudWatch Logs pricing, retention, and observability best practices
AWS Well-Architected Framework (Cost Optimization and Reliability pillars)
Serverless Lens and workload-specific architecture review guidance

My Study Guide for the Microsoft Certified SQL AI Developer Associate Beta Exam

Renaldi — Fri, 10 Apr 2026 23:00:00 +0000

I put this guide together for people who want something more useful than a recycled checklist. When a certification is still in beta, the most valuable preparation is not memorizing fragments. It is understanding how Microsoft is defining the role, what real skills sit underneath the blueprint, and how those skills connect to actual work. For this exam, that means designing and developing AI enabled database solutions across Microsoft SQL Server, Azure SQL, and SQL database in Microsoft Fabric, while also being comfortable with T-SQL, CI/CD, security, performance, and modern AI integration patterns.

What makes this beta especially interesting to me is that it does not treat the database as a passive storage layer. The role itself is framed around building AI enabled database solutions. That is a very different mindset from traditional database development. It means the exam is really about whether you can treat the database as an active part of an intelligent application architecture rather than just the place where rows happen to live.

Why this exam needs a different kind of preparation

Beta exams are different from mature exams in a very practical way. The public prep ecosystem is usually thinner, the patterns are less familiar, and the most useful clue is often the study guide itself. That matters here because the official certification page says the practice assessment is not currently available while the exam is in beta, and beta exams are not scored immediately. In other words, this is not the kind of exam where you can rely on a polished prep routine and coast through it.

My approach would be simple. Study this as a role, not as a quiz. If you approach it like a collection of disconnected SQL facts, you will miss what makes it valuable. If you approach it like a role centered on application design, database engineering, delivery discipline, and AI capability, the blueprint starts to make a lot more sense.

Who this guide is for

This guide is for you if you already know basic T SQL and database development, but you want a more structured way to prepare for a beta that blends modern SQL engineering with AI features. It is especially useful if any of these feel true for you.

You are comfortable with databases but less comfortable with AI features in SQL platforms
You know SQL development but want a clearer CI CD and deployment mindset
You are strong in Azure SQL or SQL Server but have not yet thought deeply about how AI changes database design choices
You want a study plan that is grounded in the official exam scope without being dry and mechanical

If you are already experienced in SQL development, application data design, and delivery automation, this exam should feel relevant. If your experience is mostly limited to writing ad hoc queries, you will likely need to spend more time on lifecycle thinking, deployment practices, and AI integration patterns.

What the exam is really testing

The current study guide breaks the exam into three major skill areas.

Design and develop database solutions with a weight of 35 to 40 percent
Secure, optimize, and deploy database solutions with a weight of 35 to 40 percent
Implement AI capabilities in database solutions with a weight of 25 to 30 percent

That distribution tells you a lot. This is not an AI first exam that only lightly touches databases. It is still a database developer exam at its core. The AI section matters, but it sits on top of strong expectations around schema design, query logic, programmability, security, deployment, and performance engineering.

That is why I would prepare for this in layers.

First, make sure your database engineering fundamentals are genuinely strong.

Second, make sure you can think about database delivery as code, not just as changes made manually in a tool.

Third, add the AI layer on top so that you can reason clearly about vector search, embeddings, RAG style scenarios, and model integration without losing your grounding in database design.

My study philosophy for this beta

My rule for this exam would be to study every objective from four angles.

What the feature does
Why a team would choose it in a real application
What tradeoffs it introduces
What can go wrong in production

That last point is where a lot of real understanding lives. Many candidates can explain what a feature is. Fewer can explain when it is a good choice, when it is a risky choice, or how it changes operational behavior after deployment. This exam feels much more valuable when you prepare for that deeper level.

Domain one

Design and develop database solutions

This domain carries the heaviest weight and sets the foundation for everything else. The study guide includes database objects, specialized tables, JSON columns and indexes, constraints, sequences, stored procedures, functions, triggers, views, transactions, query logic, window functions, common table expressions, error handling, and support for semi structured data.

What this domain really means

This domain is about whether you can design a database that supports application behavior cleanly, predictably, and at scale. It is not just about knowing syntax. It is about choosing the right structures and patterns so the data model works for real workloads.

The more I look at this blueprint, the more I think this section rewards engineers who can connect schema design with application behavior. A table definition is not just a table definition. It affects performance, maintainability, integrity, and how easy it is to add AI driven features later.

What I would make sure I can do

Design tables with sensible keys, data types, defaults, and constraints
Explain when specialized table types such as temporal, graph, ledger, in memory, or external tables make sense
Work confidently with JSON data and know how it affects design and querying
Write solid stored procedures, functions, and views without relying on fragile shortcuts
Use transactions intentionally and know when isolation and consistency matter most
Read complex query logic without getting lost when joins, window functions, and common table expressions are involved
Think of schema design as application design, not just as data storage

Hands on tasks I would actually practice

Create a small schema from scratch for a realistic application and justify each table and constraint
Add JSON based attributes to one part of the design and compare the tradeoffs against full normalization
Write queries that use window functions for ranking, running totals, and partitioned logic
Build a stored procedure that includes error handling and transaction control
Review a design and ask where future AI style features such as semantic search or hybrid structured and unstructured access might fit

Traps I would avoid

Treating schema design as a memorization exercise
Ignoring JSON and semi structured data because it feels secondary
Overfocusing on rare features before mastering the core objects and query patterns
Studying stored procedures and functions only as syntax rather than as maintainable application building blocks

My view on this domain

This domain is exactly why I think the exam has substance. It still expects you to be a real database developer. That is a good thing. Without strong fundamentals here, the AI portions become superficial very quickly.

Domain two

Secure, optimize, and deploy database solutions

The study guide includes authentication and authorization, roles and permissions, data protection, monitoring, performance tuning, indexing, query plan awareness, deployment using SQL database projects, Git based workflows, schema drift, and CI CD practices.

What this domain really means

This domain is about whether you can treat database work as part of a modern engineering workflow rather than a one off administrative activity. To me, this is one of the most important parts of the exam because it separates people who can build something from people who can maintain and ship something responsibly.

The database is often where teams become inconsistent with engineering discipline. Application code gets source control, pipelines, reviews, and repeatable releases. Database changes too often get handled manually or as an afterthought. This domain pushes in the right direction by expecting proper delivery discipline.

What I would make sure I can do

Explain core authentication and authorization choices in Azure SQL and related SQL platforms
Understand how permissions should be scoped and why least privilege matters
Read performance problems through the lens of indexing, plan quality, and workload behavior
Use SQL database projects as a source of truth for schema
Understand how CI CD pipelines apply to database changes
Explain schema drift and why it becomes dangerous when teams skip disciplined deployment practices
Think about deployment safety and repeatability rather than just whether the change works once

Hands on tasks I would actually practice

Create a simple SQL database project and put it under source control
Build and validate a DACPAC based deployment flow
Walk through a GitHub Actions or Azure DevOps example for database deployment
Compare a safe controlled deployment process against a manual direct change process
Review indexing choices on a small schema and explain how they support or hurt workload patterns
Practice reading execution plans at a conceptual level, especially for common tuning scenarios

Traps I would avoid

Thinking that database CI CD is too operational to be important for the exam
Treating security as a list of terms instead of a design decision
Memorizing individual performance tips without understanding why they help
Ignoring schema drift because it sounds niche

My view on this domain

I really like that this exam gives so much space to secure delivery and optimization. That makes it feel much more relevant to the actual role. Modern database development is not just about writing objects. It is about shipping them safely and keeping them fast.

Domain three

Implement AI capabilities in database solutions

This is the part that makes the exam stand out. The study guide includes AI integration patterns, embeddings, vector search, retrieval augmented generation style design, intelligent search, and building AI aware data solutions directly on Microsoft SQL platforms.

What this domain really means

This domain asks whether you can connect database engineering with modern AI application patterns without losing rigor. That matters because there is a big difference between talking about AI in general and actually designing a database solution that supports intelligent retrieval, vector based matching, model integration, and reliable application behavior.

What I find most interesting is that Microsoft is clearly positioning SQL platforms as active participants in AI architectures. This is not just a certification about wiring a model to an app. It is about understanding how the database itself can support embeddings, retrieval, structured filtering, hybrid search behavior, and data shaped for AI driven scenarios.

What I would make sure I can do

Explain what embeddings are at a practical level and why they matter for semantic similarity scenarios
Understand where vector search fits and when it is appropriate
Compare keyword style retrieval with semantic or hybrid approaches
Understand how structured data and semi structured data can work together in intelligent applications
Think clearly about where SQL based AI patterns are a good fit and where a dedicated search or AI service may still be the better choice
Understand the role of T SQL in orchestrating AI aware retrieval patterns
Be comfortable talking about RAG style design in database terms rather than only in model terms

Hands on tasks I would actually practice

Walk through an Azure SQL vector search sample and explain each step in plain language
Store and query JSON based data alongside relational data and think about how that supports intelligent application scenarios
Compare a full text or keyword style search approach against a vector similarity approach
Read documentation on intelligent applications in Azure SQL and summarize when SQL centric AI design is strong and when it becomes limiting
Sketch a simple RAG style flow where the database plays an intentional retrieval role

Traps I would avoid

Studying AI buzzwords without grounding them in actual SQL platform behavior
Assuming vector search replaces all other search patterns
Ignoring the importance of structured filters in intelligent applications
Treating AI features as magic instead of as design choices with cost, performance, and complexity tradeoffs

My view on this domain

This domain is the reason I think the certification is genuinely interesting. It reflects a shift that is happening in the industry right now. Databases are no longer just passive systems underneath AI applications. They are becoming active parts of how those systems retrieve, filter, ground, and serve information.

The study resources I would actually use

I would start with the official Microsoft resources and then move into hands on documentation that maps directly to the objectives. Here is the stack I would use.

Start here

Core database development and delivery

T SQL and query depth

JSON and semi structured data

AI features in SQL platforms

Platform context that helps

A four week study plan that I think actually works

Week one

Build the map

Read the official study guide closely and create your own notes under the three domains. Mark each topic as strong, medium, or weak. Do not study everything with equal intensity. This exam has a very clear weighting, so your time should follow it.

Your goal in week one is not mastery. Your goal is clarity.

You should finish the week knowing exactly where your risk areas are.

Week two

Go deep on database design and programmability

Spend the week on schema design, specialized tables, JSON, views, procedures, functions, transactions, and query logic. Build one small realistic schema and keep improving it as you learn.

Ask yourself practical questions such as these.

Is this design flexible enough for application change
Where would semi structured data belong here
Which constraints actually protect integrity
What query patterns will this application need most often

Week three

Go deep on delivery security and performance

Focus on SQL database projects, CI CD, source control, schema drift, permissions, authentication, indexing, and plan awareness. This is the week where you move from being someone who can write database code to someone who can ship it well.

If you skip this part, you risk underestimating a huge portion of the exam.

Week four

Add the AI layer and consolidate

Now focus on embeddings, vector search, intelligent application patterns, RAG style thinking, and hybrid retrieval logic. Connect these back to the database fundamentals you already studied.

The key question for this final week is not simply how an AI feature works. It is why that feature belongs in this database design and what tradeoffs come with it.

A practical weekend sprint if you are short on time

If all you have is a weekend, I would still avoid random study.

Day one

Read the certification page and study guide end to end
Focus first on domain one and domain two
Review schema design, SQL projects, CI CD, permissions, indexing, and deployment flow
Build one page notes for design, one page notes for deployment, and one page notes for tuning

Day two

Focus on domain three
Review intelligent applications, embeddings, vector search, and SQL centric AI patterns
Walk through at least one official sample and one or two docs pages in detail
End by explaining each domain out loud in your own words

That last step sounds simple, but it is one of the best checks of whether you understand the role or are just recognizing keywords.

The notes format I would use

I would keep notes in a format that forces decision making.

Feature or concept

What it is
Why it exists
When I would choose it
What tradeoffs it creates
What could go wrong in production

That format is much more useful than copying definitions into a document you never read again.

What I would not waste time on

Blind memorization of isolated syntax without application context
Treating AI features as separate from core database design
Ignoring CI CD because it feels less technical than query writing
Overfocusing on rare edge features before mastering the heavily weighted fundamentals
Reading only summaries without opening the real documentation and samples

Final 48 hours

In the last two days, I would stop collecting new material and start tightening judgment.

Review the three domains and their weightings again
Revisit schema design, query logic, deployment, permissions, and performance tuning
Revisit embeddings, vector search, and intelligent application patterns
Review where SQL is the right home for AI style retrieval and where another service may be stronger
Sleep properly and keep your thinking sharp

Because the practice assessment is not currently available in beta, your ability to explain decisions clearly becomes even more important. This is one of those exams where maturity of reasoning is likely to matter more than pattern memorization.

My closing take

If I had to summarize this beta in one sentence, I would say it is testing whether you can think like a modern SQL application engineer whose database is part of the intelligence layer, not just the persistence layer.

That is why I think this certification is worth paying attention to. It still respects strong database fundamentals, but it also points toward where the role is heading. For me, that is what makes a beta worth studying. It is not just an exam. It is an early signal.

If you are preparing for this one, I would study it like a role that sits at the intersection of database engineering, delivery discipline, and AI aware application design. That is the mindset this guide is built to support.

Building a Document Processing Pipeline with S3, Textract, Step Functions and EventBridge

Renaldi — Sun, 05 Apr 2026 23:00:00 +0000

This is one of my favorite AWS patterns to demo because it is both visually compelling and production-relevant. It shows event-driven architecture, orchestration, asynchronous AI/ML service integration, scale-out processing, human-in-the-loop review, and operational discipline in one workflow.

In this post, I will walk through an end-to-end implementation of a document processing pipeline built with:

Amazon S3 for document ingress and result storage
Amazon Textract for OCR and structured extraction
AWS Step Functions for orchestration (including Distributed Map for batch scale)
Amazon EventBridge for event routing and downstream integration

I will also cover:

Async Textract orchestration
Batch scaling with Distributed Map
Result storage and audit trail
Human review step
Cost and throughput tuning
Architecture and code walkthrough

Why this pattern is so effective

In real teams, document processing is rarely just “OCR a file and store JSON.” We usually need to handle:

Multi-page PDFs and asynchronous processing
Bursty uploads (for example, end-of-day batch drops)
Traceability for compliance and audits
Human review for low-confidence or ambiguous fields
Clean integration points for downstream systems

This architecture solves those concerns in a way that is easy to demonstrate and scale.

What we are building

I am designing the pipeline around two operating modes:

Single-document event-driven processing
- A file lands in S3
- EventBridge triggers the workflow
- The workflow runs end-to-end for that document
Batch processing mode
- A manifest (JSON/CSV/list of document keys) is provided
- Step Functions uses Distributed Map to fan out child workflows
- Each child workflow processes one document independently

This keeps the core processing logic consistent while letting me scale from demos to production.

Architecture Overview

At a high level, the flow is:

A document is uploaded to an S3 input bucket
S3 emits an event to EventBridge
EventBridge starts a Step Functions parent workflow
The parent workflow prepares a manifest (single item or batch list)
A Distributed Map launches child workflows for each document
Each child workflow:
- validates input
- starts async Textract
- waits/polls for completion
- retrieves and normalizes results
- stores raw + normalized outputs
- writes audit records
- routes low-confidence cases to human review
- emits a processed event to EventBridge
Downstream systems consume the processed event

End-to-End Walkthrough

1) Document ingress with S3 + EventBridge

I use S3 as the system of record for inbound documents. The upload path usually looks like:

s3://my-docs-incoming/raw/YYYY/MM/DD/<tenant>/<document>.pdf

I prefer a structured key naming convention because it helps with:

lifecycle policies
tenant scoping
troubleshooting
cost attribution
replay operations

S3 can publish object events to EventBridge, and I use an EventBridge rule to filter only the prefixes and file types that should trigger processing (for example, PDFs in raw/).

Example EventBridge rule pattern (S3 object created)

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["my-docs-incoming"]
    },
    "object": {
      "key": [{ "prefix": "raw/" }]
    }
  }
}

From there, EventBridge starts the Step Functions parent state machine.

2) Parent orchestration with Step Functions

I use a Standard workflow for the parent because:

document jobs can run for a while
I may have retries and waits
I may pause for human review (callback pattern)
I want richer execution history

The parent workflow does a few things:

Parses the incoming event (or batch request)
Creates a manifest for processing (even if only one document)
Sets concurrency controls (important for Textract/Lambda quotas)
Launches a Distributed Map

Why Distributed Map matters

For real batches, a normal Map state can become limiting because of concurrency and history size. Distributed Map gives me child workflow executions, better scaling, and cleaner observability for each document.

3) Child workflow per document (core pipeline)

Each child workflow processes a single document from start to finish.

Core steps in the child workflow

Validate input
- file type
- bucket/key exists
- metadata (tenant, doc type, correlation ID)
- idempotency key check
Start Textract asynchronously
- call StartDocumentAnalysis
- capture JobId
- store correlation in audit trail
Wait for completion
- polling loop (simple and explicit for Step Functions)
- or callback/SNS-based completion (production optimization)
Retrieve paginated results
- call GetDocumentAnalysis until NextToken is exhausted
Normalize + enrich
- convert Textract blocks into domain JSON
- score confidence thresholds
- apply business rules
Store outputs
- raw output
- normalized output
- processing summary
- audit entries
Human review (if needed)
- pause with task token callback
- reviewer approves/edits/rejects
Emit completion event
- publish a custom EventBridge event
- downstream systems subscribe independently

4) Async Textract orchestration (practical implementation)

Textract async processing is the correct choice for multi-page documents and larger workloads. The key point is that I do not try to synchronously block a Lambda while waiting for Textract.

Instead, I let Step Functions own the waiting and retry logic.

Two patterns I use in practice

Pattern A (implemented in this post): Step Functions polling loop

This is the simplest pattern to explain and demo.

StartDocumentAnalysis
Wait
GetDocumentAnalysis (status check)
loop until SUCCEEDED / FAILED

Pros:

Easy to understand
No extra callback plumbing
Great for demos and many production cases

Cons:

More state transitions
Not the most efficient if jobs are very long

Pattern B (production optimization): SNS/SQS + callback

Textract async operations notify completion via SNS. I can correlate JobId, then call SendTaskSuccess to resume the Step Functions execution.

Pros:

Fewer polling transitions
More event-driven

Cons:

More moving parts (SNS/SQS/Lambda/token correlation)

For this article, I am implementing Pattern A for clarity, and I will still preserve auditability and scalability.

5) Result storage and audit trail (what I store and why)

This is where many demos stop too early. I do not want a pipeline that only prints extraction results to logs.

I store four classes of artifacts:

A. Raw input (immutable)

Original document in S3 (raw/)

B. Raw Textract result snapshot

Either Textract output JSON (if using OutputConfig)
Or consolidated raw blocks written by my retrieval Lambda

C. Normalized business result

A clean JSON model that downstream systems can actually use, for example:

{
  "documentId": "INV-2026-000123",
  "documentType": "invoice",
  "vendorName": "Acme Pty Ltd",
  "invoiceDate": "2026-02-25",
  "totalAmount": 1285.40,
  "currency": "AUD",
  "fields": [
    {
      "name": "invoice_number",
      "value": "INV-2026-000123",
      "confidence": 98.7,
      "source": "textract"
    }
  ],
  "review": {
    "required": false,
    "reason": null
  }
}

D. Audit trail (DynamoDB + events)

I keep an audit table with entries like:

correlation ID
S3 bucket/key/version
execution ARN
Textract JobId
timestamps (started/completed)
status transitions
retry count
reviewer ID + decision (if reviewed)

This makes troubleshooting and compliance conversations much easier.

6) Human review step (callback pattern)

I like to add a human review step because it turns the demo into a real workflow.

When I trigger human review

I send documents for human review if:

confidence is below threshold for required fields
required fields are missing
document type classification is ambiguous
business validations fail (for example, total does not match line items)

How the callback works

Step Functions reaches HumanReviewRequired
It invokes a Lambda using waitForTaskToken
That Lambda stores a review task (with the task token) in DynamoDB and optionally emits a notification event
A reviewer UI or ops tool loads the pending task
Reviewer approves/edits/rejects
API Gateway + Lambda calls:
- SendTaskSuccess (approve / approved edits)
- SendTaskFailure (reject / unrecoverable issue)

This lets me keep the workflow paused cleanly without polling the UI.

7) EventBridge for decoupled downstream integration

I use EventBridge in two places:

Ingress routing

S3 object-created events trigger the parent workflow

Egress publishing

The child workflow publishes custom events like:
- document.processed
- document.review.required
- document.failed

This is a huge win because downstream systems can subscribe independently:

search indexing
analytics pipelines
case management
notifications
data quality dashboards

No tight coupling to the core document processing workflow.

Implementation Discussion

Below is a concrete implementation outline with code.

State machine design (parent + child)

I prefer splitting the logic into:

Parent workflow: batching and fan-out
Child workflow: per-document processing

This keeps each workflow easier to reason about and test.

Parent workflow (ASL fragment with Distributed Map)

{
  "Comment": "Parent workflow for document batch orchestration",
  "StartAt": "BuildManifest",
  "States": {
    "BuildManifest": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${BuildManifestFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Next": "ProcessDocumentsDistributed"
    },
    "ProcessDocumentsDistributed": {
      "Type": "Map",
      "Label": "ProcessDocuments",
      "ItemsPath": "$.documents",
      "MaxConcurrencyPath": "$.maxConcurrency",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "StartChildExecution",
        "States": {
          "StartChildExecution": {
            "Type": "Task",
            "Resource": "arn:aws:states:::states:startExecution.sync:2",
            "Arguments": {
              "StateMachineArn": "${ChildStateMachineArn}",
              "Input": {
                "document.$": "$$.Map.Item.Value",
                "batchContext.$": "$.batchContext"
              }
            },
            "End": true
          }
        }
      },
      "Next": "BatchSummary"
    },
    "BatchSummary": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${BatchSummaryFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "End": true
    }
  }
}

Notes on this design

MaxConcurrencyPath lets me tune concurrency dynamically per batch.
I can set lower concurrency for production if Textract quotas are tighter than my desired fan-out.
Child executions give me better isolation and clearer failure analysis.

Child workflow (ASL fragment for async Textract + human review + EventBridge)

{
  "Comment": "Child workflow for processing one document",
  "StartAt": "ValidateDocument",
  "States": {
    "ValidateDocument": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${ValidateDocumentFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Next": "StartTextract"
    },
    "StartTextract": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:textract:startDocumentAnalysis",
      "Arguments": {
        "DocumentLocation": {
          "S3Object": {
            "Bucket.$": "$.document.bucket",
            "Name.$": "$.document.key"
          }
        },
        "FeatureTypes": ["FORMS", "TABLES"],
        "ClientRequestToken.$": "$.idempotencyKey",
        "JobTag.$": "$.jobTag"
      },
      "ResultPath": "$.textractStart",
      "Next": "WaitBeforeStatusCheck",
      "Retry": [
        {
          "ErrorEquals": [
            "Textract.ThrottlingException",
            "Textract.ProvisionedThroughputExceededException",
            "States.TaskFailed"
          ],
          "IntervalSeconds": 2,
          "BackoffRate": 2.0,
          "MaxAttempts": 5
        }
      ]
    },
    "WaitBeforeStatusCheck": {
      "Type": "Wait",
      "Seconds": 10,
      "Next": "CheckTextractStatus"
    },
    "CheckTextractStatus": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:textract:getDocumentAnalysis",
      "Arguments": {
        "JobId.$": "$.textractStart.JobId",
        "MaxResults": 1
      },
      "ResultPath": "$.textractStatus",
      "Next": "TextractComplete?"
    },
    "TextractComplete?": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.textractStatus.JobStatus",
          "StringEquals": "SUCCEEDED",
          "Next": "CollectTextractPages"
        },
        {
          "Variable": "$.textractStatus.JobStatus",
          "StringEquals": "FAILED",
          "Next": "MarkFailed"
        },
        {
          "Variable": "$.textractStatus.JobStatus",
          "StringEquals": "PARTIAL_SUCCESS",
          "Next": "CollectTextractPages"
        }
      ],
      "Default": "WaitBeforeStatusCheck"
    },
    "CollectTextractPages": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${CollectTextractPagesFnArn}",
        "Payload": {
          "jobId.$": "$.textractStart.JobId",
          "document.$": "$.document",
          "executionArn.$": "$$.Execution.Id"
        }
      },
      "OutputPath": "$.Payload",
      "Next": "NormalizeAndScore"
    },
    "NormalizeAndScore": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${NormalizeAndScoreFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Next": "NeedsHumanReview?"
    },
    "NeedsHumanReview?": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.review.required",
          "BooleanEquals": true,
          "Next": "CreateHumanReviewTask"
        }
      ],
      "Default": "PublishProcessedEvent"
    },
    "CreateHumanReviewTask": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Arguments": {
        "FunctionName": "${CreateHumanReviewTaskFnArn}",
        "Payload": {
          "taskToken.$": "$$.Task.Token",
          "document.$": "$.document",
          "result.$": "$",
          "executionArn.$": "$$.Execution.Id"
        }
      },
      "TimeoutSeconds": 604800,
      "ResultPath": "$.humanReviewDecision",
      "Next": "ApplyHumanReviewDecision"
    },
    "ApplyHumanReviewDecision": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${ApplyHumanReviewDecisionFnArn}",
        "Payload": {
          "pipelineResult.$": "$",
          "reviewDecision.$": "$.humanReviewDecision.Payload"
        }
      },
      "OutputPath": "$.Payload",
      "Next": "PublishProcessedEvent"
    },
    "PublishProcessedEvent": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Arguments": {
        "Entries": [
          {
            "Source": "com.example.documents",
            "DetailType": "document.processed",
            "EventBusName": "${EventBusName}",
            "Detail": {
              "documentId.$": "$.document.id",
              "bucket.$": "$.document.bucket",
              "key.$": "$.document.key",
              "status": "PROCESSED",
              "review.$": "$.review",
              "output.$": "$.output"
            }
          }
        ]
      },
      "Next": "FinalizeAudit"
    },
    "FinalizeAudit": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${FinalizeAuditFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "End": true
    },
    "MarkFailed": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "${MarkFailedFnArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "End": true
    }
  }
}

Lambda: Collect paginated Textract results and store raw output (Python)

This Lambda consolidates paginated GetDocumentAnalysis responses, writes a raw artifact to S3, and returns a pointer for downstream normalization.

import json
import os
from datetime import datetime, timezone

import boto3

textract = boto3.client("textract")
s3 = boto3.client("s3")

RAW_RESULTS_BUCKET = os.environ["RAW_RESULTS_BUCKET"]
RAW_RESULTS_PREFIX = os.environ.get("RAW_RESULTS_PREFIX", "textract-raw")


def lambda_handler(event, context):
    job_id = event["jobId"]
    document = event["document"]
    execution_arn = event["executionArn"]

    blocks = []
    next_token = None
    first_page_metadata = None
    final_status = None
    warnings = []

    while True:
        kwargs = {"JobId": job_id, "MaxResults": 1000}
        if next_token:
            kwargs["NextToken"] = next_token

        resp = textract.get_document_analysis(**kwargs)
        final_status = resp.get("JobStatus", final_status)

        if first_page_metadata is None:
            first_page_metadata = {
                "DocumentMetadata": resp.get("DocumentMetadata", {}),
                "AnalyzeDocumentModelVersion": resp.get("AnalyzeDocumentModelVersion"),
            }

        blocks.extend(resp.get("Blocks", []))
        warnings.extend(resp.get("Warnings", []))

        next_token = resp.get("NextToken")
        if not next_token:
            break

    now = datetime.now(timezone.utc).isoformat()
    raw_key = (
        f"{RAW_RESULTS_PREFIX}/"
        f"{document.get('tenant', 'unknown')}/"
        f"{document['id']}/"
        f"{job_id}.json"
    )

    payload = {
        "jobId": job_id,
        "document": document,
        "executionArn": execution_arn,
        "collectedAt": now,
        "jobStatus": final_status,
        "metadata": first_page_metadata,
        "warnings": warnings,
        "blocks": blocks,
    }

    s3.put_object(
        Bucket=RAW_RESULTS_BUCKET,
        Key=raw_key,
        Body=json.dumps(payload).encode("utf-8"),
        ContentType="application/json",
    )

    return {
        "document": document,
        "textract": {
            "jobId": job_id,
            "jobStatus": final_status,
            "rawResultsS3": {
                "bucket": RAW_RESULTS_BUCKET,
                "key": raw_key,
            },
            "blockCount": len(blocks),
            "warningsCount": len(warnings),
        },
    }

Lambda: Normalize and score extracted data (Python)

This is where I transform Textract’s generic block model into a domain result that downstream systems can consume.

import json
import os
from decimal import Decimal

import boto3

s3 = boto3.client("s3")

NORMALIZED_BUCKET = os.environ["NORMALIZED_BUCKET"]
NORMALIZED_PREFIX = os.environ.get("NORMALIZED_PREFIX", "normalized")


def _load_json_from_s3(bucket: str, key: str) -> dict:
    obj = s3.get_object(Bucket=bucket, Key=key)
    return json.loads(obj["Body"].read())


def _extract_lines_and_words(blocks):
    # Minimal example; in production I usually build maps by block Id and relationships.
    lines = []
    for b in blocks:
        if b.get("BlockType") == "LINE":
            lines.append({
                "text": b.get("Text", ""),
                "confidence": b.get("Confidence", 0.0)
            })
    return lines


def _simple_invoice_heuristics(lines):
    # Demo heuristics only. Replace with rule engine / ML classifier as needed.
    text_blob = "\n".join([l["text"] for l in lines])
    avg_conf = sum(l["confidence"] for l in lines) / max(len(lines), 1)

    result = {
        "documentType": "unknown",
        "fields": [],
        "review": {
            "required": False,
            "reason": None
        }
    }

    if "invoice" in text_blob.lower():
        result["documentType"] = "invoice"

    # Example field extraction heuristic placeholder
    # In production, I'd parse key-value pairs and tables from FORM/TABLE blocks
    result["fields"].append({
        "name": "avg_line_confidence",
        "value": round(avg_conf, 2),
        "confidence": round(avg_conf, 2),
        "source": "textract"
    })

    # Review policy example
    if avg_conf < 90:
        result["review"]["required"] = True
        result["review"]["reason"] = "Average OCR confidence below threshold"

    return result


def lambda_handler(event, context):
    document = event["document"]
    textract_ptr = event["textract"]["rawResultsS3"]

    raw = _load_json_from_s3(textract_ptr["bucket"], textract_ptr["key"])
    blocks = raw["blocks"]

    lines = _extract_lines_and_words(blocks)
    derived = _simple_invoice_heuristics(lines)

    normalized = {
        "document": document,
        "textract": {
            "jobId": raw["jobId"],
            "jobStatus": raw["jobStatus"],
            "warnings": raw.get("warnings", []),
            "blockCount": len(blocks)
        },
        "classification": {
            "documentType": derived["documentType"]
        },
        "fields": derived["fields"],
        "review": derived["review"],
        "output": {
            "rawResultsS3": textract_ptr
        }
    }

    out_key = f"{NORMALIZED_PREFIX}/{document.get('tenant', 'unknown')}/{document['id']}.json"
    s3.put_object(
        Bucket=NORMALIZED_BUCKET,
        Key=out_key,
        Body=json.dumps(normalized).encode("utf-8"),
        ContentType="application/json"
    )

    normalized["output"]["normalizedS3"] = {
        "bucket": NORMALIZED_BUCKET,
        "key": out_key
    }

    return normalized

Lambda: Create human review task (callback token storage) (Python)

This Lambda stores the task token so a reviewer can resume the workflow later.

import json
import os
import time
from datetime import datetime, timezone

import boto3

ddb = boto3.resource("dynamodb")
events = boto3.client("events")

REVIEW_TABLE = ddb.Table(os.environ["REVIEW_TABLE"])
EVENT_BUS_NAME = os.environ["EVENT_BUS_NAME"]


def lambda_handler(event, context):
    task_token = event["taskToken"]
    document = event["document"]
    result = event["result"]
    execution_arn = event["executionArn"]

    review_id = f"{document['id']}#{int(time.time())}"
    now = datetime.now(timezone.utc).isoformat()

    item = {
        "reviewId": review_id,
        "status": "PENDING",
        "createdAt": now,
        "documentId": document["id"],
        "executionArn": execution_arn,
        "taskToken": task_token,
        "document": document,
        "resultSummary": {
            "reason": result["review"]["reason"],
            "documentType": result.get("classification", {}).get("documentType", "unknown"),
            "normalizedOutput": result["output"].get("normalizedS3")
        }
    }

    REVIEW_TABLE.put_item(Item=item)

    events.put_events(
        Entries=[
            {
                "Source": "com.example.documents",
                "DetailType": "document.review.required",
                "EventBusName": EVENT_BUS_NAME,
                "Detail": json.dumps({
                    "reviewId": review_id,
                    "documentId": document["id"],
                    "reason": result["review"]["reason"]
                })
            }
        ]
    )

    # For waitForTaskToken, Lambda can return immediately after storing task metadata.
    # The workflow stays paused until SendTaskSuccess/SendTaskFailure is called.
    return {
        "reviewId": review_id,
        "status": "PENDING"
    }

Lambda: Reviewer callback endpoint (API Gateway -> Lambda -> Step Functions) (Python)

This Lambda receives the human decision and resumes the waiting execution.

import json
import os
from datetime import datetime, timezone

import boto3
from botocore.exceptions import ClientError

ddb = boto3.resource("dynamodb")
sfn = boto3.client("stepfunctions")

REVIEW_TABLE = ddb.Table(os.environ["REVIEW_TABLE"])


def _response(status_code, body):
    return {
        "statusCode": status_code,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps(body)
    }


def lambda_handler(event, context):
    try:
        body = json.loads(event.get("body") or "{}")
        review_id = body["reviewId"]
        action = body["action"]  # approve | reject
        reviewer = body.get("reviewer", "unknown")
        corrected_fields = body.get("correctedFields", {})

        item_resp = REVIEW_TABLE.get_item(Key={"reviewId": review_id})
        item = item_resp.get("Item")
        if not item:
            return _response(404, {"message": "Review task not found"})

        if item["status"] != "PENDING":
            return _response(409, {"message": f"Review task already {item['status']}"})

        task_token = item["taskToken"]
        now = datetime.now(timezone.utc).isoformat()

        if action == "approve":
            sfn.send_task_success(
                taskToken=task_token,
                output=json.dumps({
                    "reviewId": review_id,
                    "decision": "APPROVED",
                    "reviewer": reviewer,
                    "reviewedAt": now,
                    "correctedFields": corrected_fields
                })
            )
            new_status = "APPROVED"
        elif action == "reject":
            sfn.send_task_failure(
                taskToken=task_token,
                error="HumanReviewRejected",
                cause=json.dumps({
                    "reviewId": review_id,
                    "reviewer": reviewer,
                    "reviewedAt": now
                })
            )
            new_status = "REJECTED"
        else:
            return _response(400, {"message": "Invalid action"})

        REVIEW_TABLE.update_item(
            Key={"reviewId": review_id},
            UpdateExpression="SET #s = :s, reviewer = :r, reviewedAt = :t",
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={
                ":s": new_status,
                ":r": reviewer,
                ":t": now
            }
        )

        return _response(200, {"message": "Review decision recorded", "status": new_status})

    except KeyError as e:
        return _response(400, {"message": f"Missing field: {e.args[0]}"})
    except ClientError as e:
        return _response(500, {"message": "AWS error", "detail": str(e)})
    except Exception as e:
        return _response(500, {"message": "Unexpected error", "detail": str(e)})

CDK example (Python) for S3 -> EventBridge -> Step Functions start

This is a simplified snippet to show the core wiring. In a production stack, I would break this into constructs (ingress, orchestration, review, storage, observability).

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_events as events,
    aws_events_targets as targets,
    aws_stepfunctions as sfn,
    aws_iam as iam,
)
from constructs import Construct


class DocumentPipelineIngressStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs):
        super().__init__(scope, construct_id, **kwargs)

        input_bucket = s3.Bucket(
            self,
            "InputBucket",
            event_bridge_enabled=True
        )

        parent_state_machine = sfn.StateMachine.from_state_machine_arn(
            self,
            "ParentStateMachine",
            "arn:aws:states:ap-southeast-2:123456789012:stateMachine:doc-parent"
        )

        rule = events.Rule(
            self,
            "S3ObjectCreatedRule",
            event_pattern=events.EventPattern(
                source=["aws.s3"],
                detail_type=["Object Created"],
                detail={
                    "bucket": {"name": [input_bucket.bucket_name]},
                    "object": {"key": [{"prefix": "raw/"}]}
                }
            )
        )

        # Start execution on matching events
        rule.add_target(targets.SfnStateMachine(parent_state_machine))

In a real deployment, I also add:

explicit execution input transformation

IAM least privilege policies

DLQs / retry policies on targets where appropriate

environment-based prefix filtering (dev/test/prod)

Data model and audit design (recommended)

I strongly recommend treating auditability as a first-class feature.

DynamoDB audit table (example keys)

PK: DOC#<documentId>

SK: EVENT#<timestamp>#<eventType>

Example audit events

INGESTED
WORKFLOW_STARTED
TEXTRACT_STARTED
TEXTRACT_SUCCEEDED
NORMALIZED
HUMAN_REVIEW_REQUIRED
HUMAN_REVIEW_APPROVED
PUBLISHED
FAILED

This pattern gives me an append-only lineage timeline per document.

Error handling and retries (what I do on purpose)

This pipeline is asynchronous and distributed, so failures are normal. I design for them.

Where I retry

Textract throttling / throughput exceptions
transient Lambda errors
EventBridge PutEvents partial failures (Step Functions can fail the task and retry)
downstream S3 write failures

Where I do not blindly retry

invalid file format
missing required metadata
corrupted documents
deterministic business-rule violations

Failure routing

I emit a document.failed event and write a terminal audit entry with:

failure type
step name
error code
correlation IDs

That makes operations and replay much easier.

Cost and Throughput Tuning

This is where the architecture becomes production-grade.

1) Choose Step Functions workflow type intentionally

I use Standard for the main orchestration because:

async waits
human callback pauses
richer history and control

If I want to optimize cost for very high-volume short preprocessing tasks, I may split out a lightweight Express workflow in front, but I keep the long-running document path in Standard.

2) Tune Distributed Map concurrency to downstream capacity

Distributed Map can scale very high, but the bottleneck is usually downstream service quotas (Textract, Lambda concurrency, DynamoDB write capacity, etc.).

What I do:

make concurrency configurable (MaxConcurrencyPath)
start conservatively in production
load test with realistic document sizes and page counts
request quota increases before large launches

3) Reduce unnecessary state transitions

Polling is simple, but excessive polling increases state transitions and cost.

Practical tuning:

increase wait interval after the first few polls (progressive backoff)
use doc-size-aware polling intervals (longer waits for large PDFs)
consider SNS/SQS callback for long-running jobs

4) Store outputs once, reuse many times

I avoid re-running Textract when:

a replay is only needed for normalization logic changes
a downstream consumer fails
a human review needs to re-open a prior result

By storing raw output + normalized output + audit events, I can replay downstream steps without paying for re-extraction.

5) Separate hot path and review path

I do not force all documents through human review.
Most documents should:

extract
validate
publish
finish

Only low-confidence or exception cases enter the review path. This keeps cost and latency predictable.

6) Add lifecycle policies

For S3 cost management, I usually apply different retention windows:

raw uploads: longer retention (compliance dependent)
raw Textract artifacts: medium retention
normalized results: longer (business value)
debug-only artifacts: shorter

Security, Compliance, and Operational Notes

IAM least privilege

Give each component only what it needs:

Step Functions can call specific Lambdas / Textract / EventBridge
Lambdas can read/write only the intended buckets/prefixes
Reviewer API can only access review table + Step Functions callback APIs

Encryption

I enable encryption at rest for:

S3 buckets
DynamoDB tables
logs (when applicable)

If needed, I also use KMS for tighter key control and auditability.

PII considerations

Document pipelines often contain sensitive data. I plan for:

access controls by tenant
minimal logging (no full document contents in logs)
review UI redaction where needed
retention policies aligned to policy/legal requirements

Observability

I instrument:

Step Functions execution failure alarms
Lambda error/throttle alarms
custom metrics (documents processed, review rate, average pages/doc, latency)
audit events for business-level tracing

Demo Tips (if you are presenting this live)

This pattern demos extremely well if you show all three views:

Upload a document
- drag-and-drop into S3 (or app UI)
Workflow progress
- Step Functions execution graph
- Distributed Map child runs for batches
Outputs
- raw Textract JSON pointer
- normalized JSON in S3
- audit trail item(s)
- human review queue (triggered by a low-confidence sample)

If I want a smooth live demo, I prepare:

one “clean” invoice (no review needed)
one noisy/scanned doc (human review path)
one small batch manifest (3 to 10 docs) to show Distributed Map fan-out

Extensions I would add next

Once this base pipeline is running, it becomes a great platform for extensions:

Document classification before extraction (route to different extraction templates)
Schema validation per document type
OpenSearch indexing for searchable archives
Comprehend / Bedrock post-processing for summaries and entity normalization
Tenant-specific extraction configs (thresholds, required fields, routing)
Replay tooling for partial reprocessing from audit trail checkpoints

Closing Thoughts

I like this architecture because it is not just a toy OCR demo. It demonstrates a proper event-driven, auditable, scalable workflow that teams can actually build on.

The combination of:

S3 for durable ingress
Textract for extraction
Step Functions for orchestration
Distributed Map for batch scale
EventBridge for decoupling
callback-based human review for exceptions

gives a strong foundation for production document intelligence systems.

If I were publishing this as a developer advocacy post, I would also include a sample repository with:

CDK deployment
test documents
reviewer UI mock
load test script
dashboards/alarms

That turns the article into something readers can run and evolve.

References

Amazon Textract async APIs and async processing overview (start/get patterns, async job model, supported file types, SNS notification model for async operations)
GetDocumentAnalysis pagination (MaxResults, NextToken) and paginated retrieval behavior
Textract output storage and OutputConfig (custom S3 output and optional KMS for output)
Textract quotas and Service Quotas guidance
S3 -> EventBridge integration and S3 EventBridge event structure
Step Functions Distributed Map (when to use it, high concurrency, child workflows, map runs, input/history limits)
Step Functions callback/task token pattern and long waits for external/human steps
Step Functions optimized EventBridge integration (events:putEvents) and failure handling behavior
Step Functions pricing model (state transitions for Standard workflows)

My Study Guide for the Microsoft Certified Machine Learning Operations MLOps Engineer Associate Beta Exam

Renaldi — Fri, 03 Apr 2026 22:00:00 +0000

When I study beta exams, I do not treat them like ordinary certification exams; I treat them like signals. They usually show where Microsoft thinks the role is heading next, what skills are becoming core, and which responsibilities are moving from specialist territory into mainstream expectations. That is exactly how I approached this one.

What stood out to me most is that this exam is not only about shipping a model. It is about owning the full operational system around machine learning and generative AI on Azure. That means infrastructure, repeatability, deployment, monitoring, observability, evaluation, safety, retrieval quality, versioning, and optimization. If you study this as a narrow model training exam, I think you will undershoot what the role is really asking for.

Who I think this exam is really for

I think this beta makes the most sense for people who already have some machine learning familiarity and now want to prove they can think operationally. If you have trained models before but have not spent enough time on automation, deployment, rollout strategy, monitoring, or generative AI quality assurance, this exam will probably stretch you in exactly the right areas.

I would also say this guide is useful for people who sit somewhere between data science and platform engineering. That feels like the real audience. Not purely academic machine learning practitioners. Not purely infrastructure engineers. People who want to understand how AI systems actually become production systems.

What I believe the exam is really testing

The current official study guide breaks the exam into five major domains. I would not memorize the domain names only. I would translate them into practical questions.

Domain 1

Design and implement an MLOps infrastructure

This is really asking whether you can build the operational foundation for machine learning on Azure in a way that a team could actually reuse.

Domain 2

Implement machine learning model lifecycle and operations

This is really asking whether you understand that the model is only one piece of the job, and that lifecycle discipline matters just as much as model quality.

Domain 3

Design and implement a GenAIOps infrastructure

This is really asking whether you can treat generative AI systems with the same seriousness that strong teams already apply to software delivery and MLOps.

Domain 4

Implement generative AI quality assurance and observability

This is really asking whether you know how to judge if a generative AI system is good, safe, stable, and supportable after release.

Domain 5

Optimize generative AI systems and model performance

This is really asking whether you know how to improve a system after the first version works.

My overall study philosophy

My rule for this beta is simple. Study every topic from three different angles.

What the service or capability does
Why a team would choose it in real production work
What can go wrong after deployment

That third point is where I think a lot of people fall short. It is easy to say you know how to create a workspace, deploy a model, or run an evaluation. It is much harder to explain how you would operate that system safely over time, what you would monitor, what you would version, what you would lock down, and how you would know that something is getting worse.

Domain 1 in depth

Design and implement an MLOps infrastructure

This domain covers the platform foundations for traditional machine learning on Azure. I see it as the domain that separates casual Azure Machine Learning usage from real MLOps thinking.

What I would focus on

Azure Machine Learning workspace design
Datastores and data assets
Compute targets and their purpose
Environments and reusable components
Registries and cross workspace reuse
Identity and access control
Private networking and secure access
Git integration
Azure CLI and Bicep for repeatable setup
GitHub Actions for automation

What I think matters most here

The most important mindset shift is to stop thinking about a workspace as a place where you click around in the portal and start thinking about it as a controlled platform foundation. The exam focus here feels relevant because this is where operational maturity begins. A lot of AI teams still treat infrastructure as something they sort out later. This exam clearly does not.

What I would actually practice

Create a workspace and understand the role of related resources
Compare different compute options and know why you would choose each one
Create data assets and environments that can be reused
Understand when a registry helps with scale and team collaboration
Read Bicep and Azure CLI examples until they stop feeling unfamiliar
Set up a simple GitHub Actions workflow that touches the machine learning lifecycle
Review a secure networking setup and be able to explain why it matters

Red flags during study

Only learning portal steps
Ignoring identity and networking
Treating source control as optional
Memorizing syntax without understanding the operational reason behind it

Domain 2 in depth

Implement machine learning model lifecycle and operations

This is the domain I would spend the most time on. It is also the one that feels closest to the day to day reality of MLOps work.

What I would focus on

Experiment tracking with MLflow
Automated machine learning
Notebooks and jobs
Hyperparameter tuning
Distributed training
Training pipelines
Model registration and lifecycle management
Responsible AI checks
Batch inference and real time inference
Rollout strategy and rollback strategy
Drift detection and response
Monitoring and retraining triggers

What I think matters most here

To me, this domain tests whether you understand that training a model is just the beginning. Strong candidates should be able to think in versions, runs, promotion paths, failure handling, deployment strategy, and operational feedback loops.

I also think this is where many people who call themselves familiar with MLOps get exposed. A lot of people really mean that they have deployed a model once. That is not the same as managing the lifecycle of a model in production. This domain seems much more aligned with the real role.

What I would actually practice

Track experiments in MLflow and compare runs with intention
Run a simple sweep job and understand what problem it solves
Register a model and think carefully about version control
Compare batch inference and online inference through real scenarios
Study what safe rollout and rollback mean for business risk
Review how drift shows up and what an operational response should look like
Think through what should trigger retraining and when retraining should not be automatic

Questions I would ask myself

How do I know which model version is live
What would I monitor after deployment
When would I prefer batch over online
What would make me roll back quickly
What signals tell me performance is degrading

Domain 3 in depth

Design and implement a GenAIOps infrastructure

This domain is one of the clearest signs that Microsoft is treating generative AI operations as a first class engineering discipline, not just an experimental side topic.

What I would focus on

Microsoft Foundry setup and project structure
Managed identity and access control
Network security
Azure CLI and Bicep for deployment
Model selection strategy
Serverless deployment options
Managed compute options
Prompt design as an engineering asset
Prompt variants and versioning
Source control for prompts and application logic
Capacity and throughput planning

What I think matters most here

I really like that this domain does not reduce generative AI work to clever prompting. That would have made the exam feel shallow. Instead, the domain focus suggests that Microsoft wants candidates to think about generative AI as an operational system with infrastructure, security, versioning, and delivery discipline.

That feels much more useful and much closer to what strong teams are actually trying to build.

What I would actually practice

Create a small Foundry project and inspect how it is organized
Compare serverless access with managed compute and explain the tradeoff
Practice thinking through model choice based on latency, cost, scale, and task fit
Store prompt variants in Git and treat them as production assets
Compare prompt versions with a repeatable test set instead of instinct alone
Review how access, networking, and deployment decisions affect operational safety

Common mistake I would avoid

Do not study generative AI operations as if it is just prompt engineering plus model selection. The exam focus appears broader than that. I would expect the stronger candidates to understand the surrounding engineering system as well.

Domain 4 in depth

Implement generative AI quality assurance and observability

This domain is smaller by weighting but huge in practical value. In real production work, this is where weak systems often reveal themselves.

What I would focus on

Test datasets for evaluation
Quality metrics such as groundedness relevance coherence and fluency
Safety evaluation
Harmful content monitoring
Automated evaluations
Continuous monitoring in Foundry
Latency and throughput tracking
Token and cost visibility
Logging tracing and debugging

What I think matters most here

Generative AI systems do not fail in only one way. They can become ungrounded, irrelevant, too expensive, too slow, inconsistent, or unsafe. That means quality assurance for generative AI has to be broader than a simple accuracy mindset.

This domain feels very relevant because it pushes candidates toward that broader view. I would not treat it as a side topic just because the weighting is lower. In real projects, a surprising amount of pain starts here.

What I would actually practice

Build a small evaluation set for a realistic use case
Compare outputs against several quality dimensions
Think about safety and harmful output as operational concerns
Review token consumption and latency as first class engineering metrics
Explore tracing and debugging ideas for multi step AI application flows
Practice describing what good monitoring should actually tell you

A practical mindset that helps here

When I study this domain, I try to move beyond asking whether an answer looks good. I ask whether the system is measurable, debuggable, supportable, and safe enough to own over time.

Domain 5 in depth

Optimize generative AI systems and model performance

This domain is about moving from a working prototype to a stronger production system.

What I would focus on

Retrieval augmented generation tuning
Chunking strategy
Similarity thresholds
Embedding model choice
Hybrid retrieval
Relevance evaluation
A B style testing
Fine tuning strategy
Synthetic data management
Monitoring customized models in production

What I think matters most here

I find this domain especially realistic because the hard part of generative AI is often not making something work once. The hard part is making it work better in a controlled and measurable way.

The focus here suggests that Microsoft expects candidates to understand optimization as an ongoing engineering process. That feels right to me.

What I would actually practice

Compare different chunk sizes and review how retrieval quality changes
Think about what happens when similarity thresholds are too strict or too loose
Compare semantic retrieval and hybrid retrieval for a realistic use case
Practice explaining when fine tuning is justified and when retrieval or prompt work is enough
Think about how you would promote a customized model into production safely
Review relevance tradeoffs alongside latency and cost tradeoffs

The resources I would use first

I would not study from random blogs first. I would start with the Microsoft material that best matches the beta and then layer my own hands on practice on top.

Official exam page

Use this first to confirm the role focus and linked preparation material.

learn.microsoft.com/en-us/credentials/certifications/operationalizing-machine-learning-and-generative-ai-solutions/

Official study guide for AI 300

This is the single most important resource for mapping your preparation. I would keep coming back to it while studying.

learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-300

Official course for operationalizing machine learning and generative AI solutions

This is a very good structured backbone if you want guided preparation.

learn.microsoft.com/en-us/training/courses/ai-300t00

Learning path for operationalize machine learning models

This is especially useful for the traditional MLOps parts of the exam such as Azure Machine Learning workflows, pipelines, GitHub Actions, environments, and deployment.

learn.microsoft.com/en-us/training/paths/build-first-machine-operations-workflow/

GitHub Actions for Azure Machine Learning

I would study this if CI and CD for machine learning still feels a bit fuzzy.

learn.microsoft.com/en-us/azure/machine-learning/how-to-github-actions-machine-learning

Set up MLOps with GitHub and Azure Machine Learning

This is useful if you want a more end to end sample oriented view.

learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-github-azure-ml

Deploy machine learning models to online endpoints

A key resource for real time inference preparation.

learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints

Deploy MLflow models to online endpoints

Important if you want your MLflow understanding to connect directly to Azure deployment patterns.

learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints

Deploy MLflow models in batch deployments

Helpful for understanding batch inference patterns and where they fit.

learn.microsoft.com/en-us/azure/machine-learning/how-to-mlflow-batch

Microsoft Foundry documentation

This is the starting point for the generative AI side of the exam.

learn.microsoft.com/en-us/azure/foundry/

Foundry observability

Very important for the quality assurance and monitoring domains.

learn.microsoft.com/en-us/azure/foundry/concepts/observability

Evaluate generative AI applications in Foundry

A strong resource for learning how evaluations actually work in practice.

learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app

Evaluate AI agents in Foundry

Worth reviewing if you want to understand how Microsoft is framing agent evaluation in operational terms.

learn.microsoft.com/en-us/azure/foundry/observability/how-to/evaluate-agent

My recommended study sequence

If I were starting from scratch, this is the order I would follow.

Step 1

Build the map

Read the official study guide carefully and create your own notes under the five domains. Mark every topic as strong medium or weak.

Step 2

Go deep on core MLOps

Spend focused time on experiments pipelines MLflow model registration endpoints rollout rollback and drift. This is where I think the exam has the most practical depth.

Step 3

Add the GenAIOps layer

Move into Foundry setup model choice prompt versioning evaluations observability and optimization. Do not treat this as a separate world. Connect it back to operational discipline.

Step 4

Practice decision making

The exam is likely to reward people who can choose between approaches, not just define features. Practice reasoning through tradeoffs.

Step 5

Revisit weak areas with hands on reinforcement

When a topic feels vague, do not just reread it. Build a tiny example, sketch a workflow, or explain it aloud.

A four week plan that I think is realistic

Week 1

Build your domain map

Read the exam page and study guide slowly
Build notes for all five domains
Mark weak areas honestly
Start with the official course or learning path

Week 2

Go deep on machine learning lifecycle work

Focus on experiments MLflow sweeps pipelines registration deployment and monitoring
Practice thinking through real scenarios
Learn to describe rollout rollback and drift clearly

Week 3

Go deep on generative AI operations

Focus on Foundry setup model choice prompt versioning evaluations observability and optimization
Build a small comparison mindset for prompts and models
Think in terms of quality safety latency and cost together

Week 4

Consolidate and rehearse

Revisit all weak areas
Rebuild key workflows from memory
Review official resources again
Practice explaining why one approach is better than another

If you only have a weekend

If time is short, I would not try to cover everything equally.

Day 1

Read the study guide carefully
Focus first on Domain 2 and Domain 3
Build one page notes for traditional MLOps and GenAIOps
Review lifecycle thinking more than portal details

Day 2

Cover Domain 1 Domain 4 and Domain 5
Review security automation observability evaluation and optimization
End by testing whether you can explain each domain in your own words

What I would avoid during preparation

Passive reading without building a domain map
Studying only portal click paths
Ignoring GitHub Actions and infrastructure setup
Treating prompt work as separate from engineering discipline
Overfocusing on one familiar area and neglecting the rest
Memorizing terms without understanding why they matter in production

My final take

If I had to sum up the beta in one sentence, I would say this exam is trying to validate whether you can think like an operator of AI systems on Azure, not just a user of AI tools.

That is why I find it interesting. It feels like a certification for people who want to move from experimentation into ownership.

If you are preparing for it, I would study it like a role, not like a quiz. That mindset alone will make your preparation much stronger.

Serverless ETL/ELT Architecture with S3, EventBridge, Lambda, Step Functions, and Glue

Renaldi — Sun, 29 Mar 2026 22:00:00 +0000

In this post, I will walk through a production-style serverless ETL/ELT architecture on AWS using Amazon S3, Amazon EventBridge, AWS Lambda, AWS Step Functions, and AWS Glue. I will cover the full flow from event-driven ingestion to validation, quarantine handling, orchestration, schema drift handling, data quality checks, and replay.

I am intentionally designing this as a pattern that can support both ETL and ELT:

ETL when I perform transformations in Glue before landing curated outputs
ELT when I land validated/raw data first and defer transformation to downstream query engines or warehouse jobs

This architecture is a strong fit for data lake ingestion pipelines where I want:

event-driven automation
low operational overhead
clear failure handling
replayability
observability
and enough flexibility to survive real-world data messiness

A few implementation choices in this post are deliberate:

I use Step Functions Standard (not Express) because I want durable, auditable executions and because the common Glue .sync integration pattern is not supported in Express workflows. AWS documents the Standard vs Express execution semantics and notes that Express does not support .sync job-run service integration patterns. (docs.aws.amazon.com)
I use S3 Event Notifications via EventBridge for flexible routing and filtering. AWS also notes that after enabling EventBridge delivery on an S3 bucket, it can take around five minutes for changes to take effect, which is worth remembering during testing. (docs.aws.amazon.com)

Why this architecture works well in practice

When I build data ingestion pipelines, the biggest problems are usually not the happy path. They are:

malformed files
duplicate deliveries
late files
schema drift
partial failures
silently bad data
and the inability to replay safely

This pattern addresses those operational concerns directly:

S3 + EventBridge gives me flexible event routing
Lambda handles fast validation/classification
Step Functions gives me explicit orchestration, branching, retries, and auditability
Glue handles scalable transformation and schema normalization
Glue Data Quality (or equivalent checks) enforces quality gates before data becomes trusted

AWS Glue Data Quality is a managed, serverless capability built on DeeQu and uses DQDL (Data Quality Definition Language), which is useful when I want rules-as-code for quality checks. (docs.aws.amazon.com)

Architecture Overview

At a high level, I split the data lake into zones and make each stage explicit:

raw-ingest bucket: original landing location (immutable input)
validated bucket: files that passed lightweight ingest validation
quarantine bucket: invalid files, bad schema changes, or failed quality checks
curated bucket: transformed and query-ready outputs
dq-results bucket: data quality evaluation outputs and reports
replay-manifest bucket: manifests used to reprocess files safely

The flow is:

A producer drops a file in raw-ingest
S3 emits an event to EventBridge
EventBridge triggers a validation Lambda
Lambda validates and stages valid files into validated
Lambda starts Step Functions with a normalized payload
Step Functions runs Glue transformation
Step Functions runs data quality checks
Pass -> publish success / expose curated data
Fail -> quarantine and notify
Replay can be triggered later using replay manifests

End-to-End Walkthrough

1) Event-driven ingest (S3 -> EventBridge)

I start by enabling EventBridge notifications on the raw ingest bucket. This gives me centralized event filtering/routing in EventBridge rather than hardwiring everything as direct S3 notifications.

AWS documents S3 EventBridge integration and bucket-level enablement for EventBridge event delivery. (docs.aws.amazon.com)

Example: enable EventBridge on the raw ingest bucket

aws s3api put-bucket-notification-configuration \
  --bucket my-raw-ingest-bucket \
  --notification-configuration '{
    "EventBridgeConfiguration": {}
  }'

2) Event filtering in EventBridge

I usually create one or more EventBridge rules to target datasets by path prefix and file type. This lets me route different datasets into different workflows without changing the producers.

Example EventBridge rule pattern (illustrative)

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["my-raw-ingest-bucket"]
    },
    "object": {
      "key": [
        { "prefix": "landing/orders/" }
      ]
    }
  }
}

From here, I send the event to a Validation Lambda (or directly to Step Functions if I want validation inside the state machine).

3) Validation and quarantine buckets

I like to separate lightweight ingest validation from heavy transforms:

file naming / suffix checks
minimum size checks
header checks (CSV/JSON)
partition key extraction from path
duplicate detection / idempotency signal generation
metadata enrichment (dataset, source, arrival timestamp, run ID)

If validation passes, I stage the object into a validated bucket.
If it fails, I route it to quarantine with a reason code.

This gives me two important benefits:

I preserve the original raw object for traceability
I avoid sending obviously bad data into expensive Glue runs

Example validation Lambda (Python)

import json
import os
import uuid
import urllib.parse
from datetime import datetime, timezone

import boto3

s3 = boto3.client("s3")
sfn = boto3.client("stepfunctions")

VALIDATED_BUCKET = os.environ["VALIDATED_BUCKET"]
STATE_MACHINE_ARN = os.environ["STATE_MACHINE_ARN"]

ALLOWED_SUFFIXES = (".csv", ".json", ".jsonl")
MIN_BYTES = 10  # example threshold


def _extract_dataset(key: str) -> str:
    # Example key: landing/orders/2026/02/25/orders_001.csv
    parts = key.split("/")
    if len(parts) < 2 or parts[0] != "landing":
        raise ValueError("invalid_key_prefix")
    return parts[1]


def _validate_object(bucket: str, key: str) -> dict:
    head = s3.head_object(Bucket=bucket, Key=key)
    size = head["ContentLength"]

    if not key.lower().endswith(ALLOWED_SUFFIXES):
        return {"valid": False, "reason": "unsupported_extension"}

    if size < MIN_BYTES:
        return {"valid": False, "reason": "file_too_small"}

    dataset = _extract_dataset(key)

    # Optional: inspect a small header/sample for CSV/JSON sanity
    sample = s3.get_object(Bucket=bucket, Key=key, Range="bytes=0-1023")["Body"].read()
    if not sample:
        return {"valid": False, "reason": "empty_sample"}

    return {
        "valid": True,
        "dataset": dataset,
        "size_bytes": size,
        "content_type": head.get("ContentType", "application/octet-stream"),
        "etag": head.get("ETag", "").strip('"'),
    }


def _stage_validated_copy(src_bucket: str, src_key: str, dataset: str, run_id: str) -> str:
    # I usually add a run/date prefix for traceability
    now = datetime.now(timezone.utc)
    dst_key = (
        f"validated/{dataset}/"
        f"ingest_date={now:%Y-%m-%d}/"
        f"run_id={run_id}/"
        f"{src_key.split('/')[-1]}"
    )

    s3.copy_object(
        Bucket=VALIDATED_BUCKET,
        Key=dst_key,
        CopySource={"Bucket": src_bucket, "Key": src_key},
        MetadataDirective="REPLACE",
        Metadata={
            "source_bucket": src_bucket,
            "source_key": src_key,
            "run_id": run_id,
            "dataset": dataset,
            "validated_at": now.isoformat()
        }
    )
    return dst_key


def lambda_handler(event, context):
    """
    Supports direct EventBridge S3 events.
    """
    detail = event["detail"]
    src_bucket = detail["bucket"]["name"]
    src_key = urllib.parse.unquote_plus(detail["object"]["key"])

    run_id = str(uuid.uuid4())
    validation = _validate_object(src_bucket, src_key)

    payload = {
        "runId": run_id,
        "source": {
            "bucket": src_bucket,
            "key": src_key
        },
        "validation": validation,
        "receivedAt": datetime.now(timezone.utc).isoformat()
    }

    if validation["valid"]:
        dst_key = _stage_validated_copy(src_bucket, src_key, validation["dataset"], run_id)
        payload["validated"] = {
            "bucket": VALIDATED_BUCKET,
            "key": dst_key,
            "s3Uri": f"s3://{VALIDATED_BUCKET}/{dst_key}"
        }

    # Start orchestration (Step Functions Standard)
    sfn.start_execution(
        stateMachineArn=STATE_MACHINE_ARN,
        name=run_id,
        input=json.dumps(payload)
    )

    return {"statusCode": 202, "body": json.dumps({"runId": run_id})}

Implementation notes for validation

A few things I recommend in production:

Do not rely on ETag as a universal checksum for multipart uploads
Keep validation Lambda fast and deterministic
Record a consistent run ID and propagate it through every stage
Treat the raw bucket as immutable evidence
Add bucket lifecycle policies for quarantine and intermediate zones

4) Orchestration for transforms (Step Functions)

This is where the pipeline becomes operationally manageable.

I use Step Functions to:

branch on validation status
invoke quarantine logic on failure
invoke Glue for transform
run data quality checks
branch on DQ result
emit success/failure events/notifications
maintain an auditable execution trail

AWS documents the Step Functions Glue integration for starting Glue jobs, which is exactly what I use here. (docs.aws.amazon.com)

Example Step Functions state machine (ASL JSON, simplified)

{
  "Comment": "Serverless ETL/ELT pipeline with validation, transform, DQ, and quarantine",
  "StartAt": "ValidationPassed?",
  "States": {
    "ValidationPassed?": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.validation.valid",
          "BooleanEquals": true,
          "Next": "PersistMetadata"
        }
      ],
      "Default": "QuarantineInvalidInput"
    },

    "PersistMetadata": {
      "Type": "Pass",
      "Parameters": {
        "runId.$": "$.runId",
        "source.$": "$.source",
        "validated.$": "$.validated",
        "validation.$": "$.validation",
        "receivedAt.$": "$.receivedAt"
      },
      "Next": "RunGlueTransform"
    },

    "RunGlueTransform": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "orders-curation-job",
        "Arguments": {
          "--run_id.$": "$.runId",
          "--input_s3_uri.$": "$.validated.s3Uri",
          "--output_s3_uri": "s3://my-curated-bucket/curated/orders/",
          "--dataset": "orders"
        }
      },
      "Retry": [
        {
          "ErrorEquals": ["Glue.ConcurrentRunsExceededException", "States.TaskFailed"],
          "IntervalSeconds": 10,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "QuarantineTransformFailure"
        }
      ],
      "Next": "RunDataQualityChecks"
    },

    "RunDataQualityChecks": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "dq-evaluator-lambda",
        "Payload": {
          "runId.$": "$.runId",
          "dataset": "orders",
          "curatedS3Prefix": "s3://my-curated-bucket/curated/orders/"
        }
      },
      "ResultPath": "$.dqInvoke",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "QuarantineDQFailure"
        }
      ],
      "Next": "DQPassed?"
    },

    "DQPassed?": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.dqInvoke.Payload.pass",
          "BooleanEquals": true,
          "Next": "PublishSuccess"
        }
      ],
      "Default": "QuarantineDQFailure"
    },

    "PublishSuccess": {
      "Type": "Pass",
      "Result": {
        "status": "SUCCEEDED"
      },
      "End": true
    },

    "QuarantineInvalidInput": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "quarantine-lambda",
        "Payload": {
          "runId.$": "$.runId",
          "source.$": "$.source",
          "reason.$": "$.validation.reason",
          "stage": "INGEST_VALIDATION"
        }
      },
      "End": true
    },

    "QuarantineTransformFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "quarantine-lambda",
        "Payload": {
          "runId.$": "$.runId",
          "source.$": "$.source",
          "reason": "GLUE_TRANSFORM_FAILURE",
          "error.$": "$.error",
          "stage": "TRANSFORM"
        }
      },
      "End": true
    },

    "QuarantineDQFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "quarantine-lambda",
        "Payload": {
          "runId.$": "$.runId",
          "source.$": "$.source",
          "reason": "DATA_QUALITY_FAILURE",
          "dqResult.$": "$.dqInvoke.Payload",
          "stage": "DATA_QUALITY"
        }
      },
      "End": true
    }
  }
}

Why I prefer Standard over Express here

This pattern tends to include:

Glue jobs
multiple branches
retries
audit needs
potential human investigation/replay

Step Functions Standard gives me durability and auditability, and AWS documents that Standard and Express have different execution semantics, durations, and integration pattern support (including the .sync limitation in Express). (docs.aws.amazon.com)

Glue transformation implementation (ETL core)

For the transformation layer, I use AWS Glue because it scales well and integrates naturally with S3 and the Data Catalog.

AWS Glue’s DynamicFrame is especially useful when incoming schemas are not cleanly stable because records are self-describing and schema inconsistencies can be represented as choice/union types. AWS also documents resolveChoice() as a recommended approach to handle multi-typed fields. (docs.aws.amazon.com)

Example Glue job script (PySpark, simplified)

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql import functions as F

args = getResolvedOptions(sys.argv, [
    "JOB_NAME",
    "run_id",
    "input_s3_uri",
    "output_s3_uri",
    "dataset"
])

sc = SparkContext()
glue_context = GlueContext(sc)
spark = glue_context.spark_session
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

run_id = args["run_id"]
input_s3_uri = args["input_s3_uri"]
output_s3_uri = args["output_s3_uri"]
dataset = args["dataset"]

# Example assumes CSV; in production I usually parameterize format/options
raw_dyf = glue_context.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={"paths": [input_s3_uri], "recurse": True},
    format="csv",
    format_options={
        "withHeader": True,
        "separator": ",",
        "quoteChar": "\""
    },
    transformation_ctx="raw_dyf"
)

# Handle schema drift / ambiguous types
# Example: amount might arrive as string in some files and numeric in others
resolved_dyf = raw_dyf.resolveChoice(specs=[
    ("amount", "cast:double"),
    ("order_id", "cast:string"),
    ("customer_id", "cast:string")
])

# Convert to Spark DataFrame for richer transforms
df = resolved_dyf.toDF()

# Add operational metadata
df = (
    df
    .withColumn("pipeline_run_id", F.lit(run_id))
    .withColumn("dataset_name", F.lit(dataset))
    .withColumn("processed_at_utc", F.current_timestamp())
)

# Example normalization
if "currency" in df.columns:
    df = df.withColumn("currency", F.upper(F.col("currency")))

if "event_ts" in df.columns:
    df = df.withColumn("event_ts", F.to_timestamp("event_ts"))

# Example partition column
df = df.withColumn("ingest_date", F.to_date(F.col("processed_at_utc")))

# Optional: enforce target schema / select only known columns (ELT-friendly alternative is to keep extras)
target_columns = [c for c in [
    "order_id", "customer_id", "amount", "currency", "event_ts",
    "pipeline_run_id", "dataset_name", "processed_at_utc", "ingest_date"
] if c in df.columns]

df_out = df.select(*target_columns)

# Write curated data
(
    df_out.write
    .mode("append")
    .format("parquet")
    .partitionBy("ingest_date")
    .save(output_s3_uri)
)

job.commit()

Schema drift handling (the part that usually breaks first)

Schema drift is not one problem. It is several different problems:

additive drift (new columns appear)
type drift (same column changes type)
structural drift (nested shape changes)
semantic drift (same field name, different meaning)
contract drift (breaking changes without notice)

I recommend treating them differently.

My practical schema drift policy

1) Additive columns (usually safe)

If a producer adds a new column and my downstream consumers can tolerate it, I usually:

allow the file
retain the new column in a bronze/validated or semi-curated layer
update downstream mappings later

2) Type drift (common and dangerous)

Example:

amount was numeric yesterday
today it arrives as "12.34" string
next week it arrives as "N/A"

This is where Glue DynamicFrame + resolveChoice() helps because I can explicitly coerce or split logic for ambiguous types. AWS documents DynamicFrame and resolveChoice() for schema inconsistencies and choice types. (docs.aws.amazon.com)

My preference:

cast when safe
preserve raw/original field if coercion may lose information
quarantine only when the drift breaks contractual expectations for trusted datasets

3) Breaking structural changes (often quarantine-worthy)

If a file violates the expected envelope (for example, missing required keys, wrong delimiter, broken nesting), I quarantine it and emit an alert.

4) Catalog and schema tracking

I typically track schema evolution using one or both:

Glue Data Catalog + crawler/table versioning
a dataset contract file (JSON/YAML) in a config repo or S3

For higher maturity pipelines, I define a per-dataset schema policy:

allow_additive_columns: true
allow_type_coercion: limited
strict_required_columns: ["order_id", "event_ts"]
quarantine_on_breaking_change: true

Data quality checks (quality gates before trust)

Schema validity is not data quality.

A file can be structurally valid and still be unusable:

duplicate IDs
null business keys
stale data
impossible values
code list mismatches
row counts far outside expected ranges

I usually place data quality checks after transformation and before I publish the data as trusted/curated.

AWS Glue Data Quality supports rules via DQDL and can be used in Glue ETL jobs or against Data Catalog datasets. (docs.aws.amazon.com)

Example DQDL ruleset (illustrative)

Rules = [
  RowCount > 0,
  IsComplete "order_id",
  IsUnique "order_id",
  ColumnExists "event_ts",
  ColumnValues "currency" in ["AUD","USD","EUR"],
  ColumnDataType "amount" = "DOUBLE"
]

Example DQ evaluator Lambda (simple custom approach)

In some environments, I start with a lightweight custom Lambda check before moving to full Glue DQ. This is especially useful for small curated datasets or when I want fast control over business-specific rules.

import json
import os
import re

import boto3
import pyarrow.parquet as pq
import s3fs

s3 = boto3.client("s3")

def _parse_s3_uri(uri: str):
    m = re.match(r"^s3://([^/]+)/?(.*)$", uri)
    if not m:
        raise ValueError("Invalid S3 URI")
    return m.group(1), m.group(2)

def lambda_handler(event, context):
    run_id = event["runId"]
    curated_prefix = event["curatedS3Prefix"]

    bucket, prefix = _parse_s3_uri(curated_prefix)
    resp = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=1000)

    objects = [o["Key"] for o in resp.get("Contents", []) if o["Key"].endswith(".parquet")]
    if not objects:
        return {"pass": False, "reason": "NO_PARQUET_OUTPUT", "runId": run_id}

    # Example: check first file quickly (for large jobs, use Glue DQ or Athena-based checks)
    fs = s3fs.S3FileSystem()
    first_uri = f"{bucket}/{objects[0]}"
    table = pq.read_table(first_uri, filesystem=fs)
    df = table.to_pandas()

    required_cols = ["order_id", "amount"]
    missing = [c for c in required_cols if c not in df.columns]
    if missing:
        return {"pass": False, "reason": "MISSING_COLUMNS", "missing": missing, "runId": run_id}

    if df["order_id"].isnull().any():
        return {"pass": False, "reason": "NULL_ORDER_ID", "runId": run_id}

    if (df["amount"].astype(float) < 0).any():
        return {"pass": False, "reason": "NEGATIVE_AMOUNT", "runId": run_id}

    return {"pass": True, "runId": run_id, "checkedFiles": len(objects)}

My recommendation in production

Start with a small ruleset that reflects actual business criticality
Separate hard-fail rules (must quarantine) from warn rules (monitor only)
Persist DQ results to an S3 prefix and emit metrics (pass rate, failed rules, affected rows)

Replay design (safe reprocessing without chaos)

Replay is one of the most under-designed parts of data pipelines.

If I cannot replay safely, operations become manual and risky very quickly.

What I want from replay

deterministic input selection
idempotent processing
clear audit trail
no accidental duplicate publishing
support for single-file and bulk replays

Replay pattern used here

I use a replay manifest dropped into a replay bucket. That manifest triggers the replay dispatcher Lambda (via EventBridge), which then starts the same Step Functions state machine with replay metadata.

This gives me:

a consistent path for normal and replay runs
a clear replay artifact I can review/version
easy automation from scripts or ops tooling

Example replay manifest

{
  "replayId": "replay-2026-02-25-orders-fix-01",
  "reason": "Reprocess files after schema mapping fix",
  "dataset": "orders",
  "files": [
    {
      "bucket": "my-raw-ingest-bucket",
      "key": "landing/orders/2026/02/24/orders_001.csv"
    },
    {
      "bucket": "my-raw-ingest-bucket",
      "key": "landing/orders/2026/02/24/orders_002.csv"
    }
  ],
  "options": {
    "force": false,
    "dryRun": false
  }
}

Example replay dispatcher Lambda (conceptual)

import json
import os
import uuid
import urllib.parse

import boto3

s3 = boto3.client("s3")
sfn = boto3.client("stepfunctions")

STATE_MACHINE_ARN = os.environ["STATE_MACHINE_ARN"]

def lambda_handler(event, context):
    detail = event["detail"]
    bucket = detail["bucket"]["name"]
    key = urllib.parse.unquote_plus(detail["object"]["key"])

    obj = s3.get_object(Bucket=bucket, Key=key)
    manifest = json.loads(obj["Body"].read())

    replay_id = manifest["replayId"]

    for i, f in enumerate(manifest["files"]):
        run_id = f"{replay_id}-{i:04d}"
        payload = {
            "runId": run_id,
            "source": {
                "bucket": f["bucket"],
                "key": f["key"]
            },
            "replay": {
                "replayId": replay_id,
                "reason": manifest.get("reason", ""),
                "options": manifest.get("options", {})
            },
            "validation": {
                "valid": True,
                "reason": None
            }
            # In a fuller implementation, I would re-run validation or call the same validator Lambda
        }

        sfn.start_execution(
            stateMachineArn=STATE_MACHINE_ARN,
            name=str(uuid.uuid4()),
            input=json.dumps(payload)
        )

    return {"statusCode": 202, "started": len(manifest["files"])}

Replay best practices I strongly recommend

Use a business idempotency key (for example, source file URI + dataset + version)
Tag replay outputs with replay metadata
Keep replay behavior explicit (force, dryRun, skipIfPublished)
Never mutate raw inputs during replay
Record replay requests and outcomes in a durable audit log (DynamoDB or S3 append logs)

ETL vs ELT: how I support both in the same architecture

This pattern is intentionally hybrid.

ETL mode (transform before publish)

I use this when:

downstream consumers need consistent schema fast
I want centralized transformation logic
I need quality gates before data is widely consumed

Flow:
validated -> Glue transform -> DQ -> curated

ELT mode (load first, transform later)

I use this when:

I need rapid ingestion of many datasets
transforms are evolving quickly
downstream teams own transformations (Athena/warehouse/dbt/etc.)

Flow:
validated -> catalog/register -> downstream SQL transforms
with the same validation/quarantine/replay capabilities still in place.

I can even run both:

land curated “v1” in Glue
also preserve validated data for future ELT workloads

Implementation discussion (what matters beyond the happy path)

1) Idempotency and duplicates

At-least-once delivery can show up in many places (producer retries, EventBridge fan-out, replays, manual operations). I assume duplicates can happen.

My approach:

derive a stable source_object_id (bucket + key + versionId if enabled)
persist processing status (optional DynamoDB table)
make publishing steps idempotent
partition outputs carefully to avoid duplicate row amplification

If your producers can overwrite keys, enable S3 versioning and include versionId in your identity model.

2) Observability

I treat observability as part of the pipeline design, not an add-on.

I emit:

ValidationPassed, ValidationFailed
GlueJobSucceeded, GlueJobFailed
DQPassed, DQFailed
Quarantined
ReplayStarted, ReplayCompleted

I also propagate:

runId
dataset
source bucket/key
replayId (if applicable)

This makes CloudWatch Logs searches and alarms much more useful.

3) Security and access control

At minimum, I lock down:

S3 bucket policies to only the required principals
KMS encryption for all buckets (SSE-KMS)
Least-privilege IAM roles for Lambda, Step Functions, and Glue
Explicit separation between raw, validated, curated, and quarantine write permissions

I also recommend:

object ownership settings to avoid ACL surprises
S3 lifecycle policies for quarantine retention
VPC configuration for Glue/Lambda only when needed (avoid unnecessary complexity)

4) Cost and performance tuning

This serverless pattern scales well, but costs can drift if I do not tune it.

Where I watch cost first

too many tiny files causing too many executions
over-frequent Glue job runs for tiny payloads
verbose DQ checks on every micro-batch
repeated replay runs without dedupe

Practical optimizations

batch small files by dataset/time window before heavy transforms
use prefix routing to isolate noisy datasets
right-size Glue workers and concurrency
use Step Functions Express only for truly short, idempotent, high-volume orchestration (not this .sync Glue-driven path)

5) Deployment strategy

I normally deploy this stack with AWS CDK, SAM, or Terraform.

A clean decomposition is:

Foundation stack
- S3 buckets
- KMS keys
- IAM roles
- EventBridge bus/rules
Compute stack
- Lambda functions
- Glue jobs
- Step Functions
Data contracts stack/config
- dataset configs
- DQ rulesets
- schema policies

This reduces blast radius when I update Glue scripts or state machine logic.

6) Testing strategy (worth doing early)

I test this pipeline at three levels:

Unit tests

Lambda validation logic
path parsing
schema policy decisions
quarantine payload generation

Integration tests

sample files dropped into raw bucket
expected state machine transitions
curated output creation
quarantine routing and reason codes

Replay tests

replay one file
replay a batch
replay a previously quarantined file after a code fix
verify no duplicate publish side effects

I strongly recommend keeping a small set of golden input files:

valid file
missing required column
type drift file
malformed file
duplicate file
late-arriving file

Common failure modes and how I handle them

Failure mode: “Pipeline is green but data is wrong”

This usually means schema validation exists but business DQ does not.
Fix: add DQ gates and track DQ outcomes as first-class metrics.

Failure mode: “Replays create duplicate records”

This is almost always an idempotency design issue.
Fix: make output writes and publish steps idempotent using stable keys and checkpointing.

Failure mode: “Glue job keeps failing after producer change”

This is often schema drift or delimiter/header changes.
Fix: detect earlier in validation, classify drift type, and quarantine with explicit reason.

Failure mode: “Debugging is painful”

This happens when correlation IDs are not propagated.
Fix: carry runId across EventBridge -> Lambda -> Step Functions -> Glue -> DQ -> Notifications.

Closing thoughts

This architecture works well because it is not just a data flow. It is an operational flow.

The key idea is simple: I do not let “data landed in S3” mean “data is trusted.” I explicitly separate:

landing
validation
transformation
quality enforcement
quarantine
replay

That separation gives me reliability, observability, and a much better developer experience when the inevitable producer-side surprises happen.

If I were extending this next, I would add:

a dataset contract registry (versioned)
a metadata/status table (DynamoDB)
automated replay tooling for quarantined objects
a producer scorecard (quality and schema stability over time)

References

Amazon S3 EventBridge integration and enablement (S3 User Guide) (docs.aws.amazon.com)
Step Functions workflow type selection (Standard vs Express semantics, durations, integration pattern differences) (docs.aws.amazon.com)
Step Functions integration with AWS Glue (startJobRun / Glue service integration) (docs.aws.amazon.com)
AWS Glue Data Quality (DQDL, DeeQu-based managed service) (docs.aws.amazon.com)
AWS Glue DynamicFrame and schema inconsistency handling (docs.aws.amazon.com)
AWS Glue resolveChoice() guidance for ambiguous types in DynamicFrame (docs.aws.amazon.com)

Idempotency Architecture for Lambda-Driven Systems on AWS

Renaldi — Sun, 22 Mar 2026 22:00:00 +0000

Duplicate processing is one of those problems that looks small in a diagram and very expensive in production.

I have seen teams build clean event-driven and Lambda-based systems, only to run into duplicate charges, duplicated emails, repeated downstream writes, or inconsistent state once retries and redrives start happening. The tricky part is that the system is often behaving as designed. AWS services are doing what they should do: retrying, buffering, redriving, and favoring delivery durability.

This is exactly why I consider idempotency architecture one of the most important and most underexplained topics in serverless engineering.

In this post, I will walk through how I design idempotency for Lambda-driven systems on AWS, including:

the exactly-once myth vs the at-least-once reality
how to choose idempotency keys and dedupe windows
a DynamoDB-based idempotency store design
using AWS Lambda Powertools idempotency utility
handling retries from API Gateway, SQS, and EventBridge
an end-to-end implementation walkthrough with code

I will keep this practical and architecture-heavy, so you can adapt it to real workloads instead of only toy examples.

The core idea in one sentence

Idempotency means I can safely process the same logical request more than once and still end up with the same intended outcome.

That does not mean the system only receives the request once. It means my system is resilient when it receives it multiple times.

Exactly-once is usually not the right mental model

A lot of production mistakes start with the assumption that a serverless flow will process each request exactly once.

In practice, in distributed systems, what I usually get is:

at-least-once delivery (common with queues/events)
retries at multiple layers (client SDKs, AWS services, Lambda, Step Functions, etc.)
timeouts and ambiguous outcomes (did the function finish but the caller timed out?)
redrives / replay (DLQ reprocessing, archive replay, manual reruns)
duplicate submissions from clients (double-click, refresh, mobile reconnect)

So instead of trying to force an “exactly-once” guarantee everywhere, I design for:

at-least-once delivery
idempotent handlers
safe retries
observability around duplicates

That mental shift makes the architecture much more robust.

Where duplicates come from in AWS Lambda-driven systems

Before I show the solution, I like to make the duplicate paths explicit.

API Gateway -> Lambda

Duplicates can happen when:

the client retries after a timeout
the network drops after the backend succeeded
the user taps “Submit” multiple times
an upstream reverse proxy retries

SQS -> Lambda

Duplicates can happen when:

Lambda fails and the message becomes visible again
processing exceeds visibility timeout
partial batch failures cause some records to be retried
DLQ redrive sends records back later
standard queues deliver the same message more than once

EventBridge -> Lambda

Duplicates can happen when:

target invocation is retried
a publisher emits semantically duplicate events
archive/replay is used
consumers reprocess historical events intentionally

That is why I architect idempotency at the business operation level, not at just one transport layer.

What a good idempotency architecture looks like

At a high level, I want:

a stable idempotency key for each logical operation
a dedupe window (TTL) appropriate for the business
a persistence store to track processing status and cached results
conditional writes to prevent concurrent duplicate execution
response replay for safe duplicate requests when appropriate
clear behavior for mismatched payloads using the same key

Architecture at a glance

End-to-end walkthrough (the scenario I will implement)

To make this concrete, I will use a common example:

“Create payment intent / order processing” style operation.

Flow

A client sends POST /payments with an Idempotency-Key header.
API Gateway invokes Lambda.
Lambda checks DynamoDB idempotency table.
If the key is new, Lambda acquires an IN_PROGRESS lock and processes the request.
Lambda writes the business result (for example, a payment record) and stores the response in the idempotency table with status COMPLETED.
If the same request is retried, Lambda returns the cached response instead of processing again.

Then I will extend the same pattern to:

SQS worker retries
EventBridge consumer retries/replay

Designing the idempotency key

This is where a lot of teams accidentally introduce bugs.

A good idempotency key should identify the logical operation, not just the transport envelope.

Good key examples

payment:{merchant_id}:{client_request_id}
order-create:{tenant_id}:{cart_checkout_id}
invoice-email:{invoice_id}:{template_version}
event:{event_id} (if the publisher guarantees a stable event ID)

Risky key choices

raw timestamp
Lambda aws_request_id (changes every invocation)
SQS receiptHandle (changes on delivery)
entire payload serialized without normalization (field order / formatting issues)
keys that are too broad (cause false dedupe)
keys that are too narrow (miss duplicates)

My rule of thumb

I choose a key from business identity + operation intent, and I define it explicitly in the contract.

For APIs, that often means:

require an Idempotency-Key header from the client, and
validate that it maps to a stable request intent.

For asynchronous consumers, that often means:

use the publisher’s stable eventId, or
derive a deterministic business key (for example order_id + action).

Dedupe windows (TTL): how long should I remember a key?

There is no universal value. I set the dedupe window based on business and retry patterns.

What affects the dedupe window

expected client retry duration
SQS redrive timing and replay operations
EventBridge replay windows / operational reruns
whether duplicates after long periods are still harmful
cost of storing idempotency records

Practical examples

API payment creation: 24 hours to 7 days
Webhook ingestion: 1 to 7 days (depends on provider retry policy)
Batch event processing: hours to days
high-volume telemetry: maybe minutes to hours (if duplicate impact is low)

Important nuance

TTL in DynamoDB is eventually applied, not immediate deletion. I design my logic so:

expiry_timestamp is authoritative in code, and
TTL is the cleanup mechanism.

In other words, I do not depend on the item disappearing exactly at expiry time.

DynamoDB-based idempotency store design (recommended pattern)

I prefer DynamoDB for idempotency state in Lambda workloads because it gives me:

low-latency key lookups
conditional writes
TTL support
simple scaling
good fit for stateless Lambda functions

Table design (single-purpose table)

A dedicated table keeps the pattern easy to reason about.

Partition key

id (string): the idempotency key

Recommended attributes

status (IN_PROGRESS, COMPLETED, optionally EXPIRED)
expiryTimestamp (epoch seconds for dedupe window)
inProgressExpiryTimestamp (shorter lock expiry to recover from crashed executions)
payloadHash (optional but highly recommended)
responseData (optional, cached result or safe response envelope)
createdAt
updatedAt
source (api / sqs / eventbridge)
functionName (optional for ops visibility)

Why `payloadHash` matters

If a client reuses the same idempotency key with a different payload, I want to detect that and reject it. Otherwise I can accidentally return a cached response for the wrong request.

This is a subtle but critical best practice.

State transitions I use

Here is the lifecycle I generally implement:

No record exists
- Try conditional write -> create IN_PROGRESS
IN_PROGRESS exists
- Another invocation is already processing (or crashed recently)
- Return a retryable outcome or fail fast depending on source
COMPLETED exists and not expired
- Return cached result (or safe ack)
Expired
- Treat as a new request (depending on business policy)

That gives me concurrency safety and retry safety.

Infrastructure example (SAM / CloudFormation snippets)

Below is a minimal setup for:

Lambda function
DynamoDB idempotency table
IAM permissions
env vars for configuration

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Idempotent Lambda API example

Resources:
  PaymentsFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.12
      Handler: app.lambda_handler
      CodeUri: src/
      MemorySize: 512
      Timeout: 29
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - dynamodb:GetItem
                - dynamodb:PutItem
                - dynamodb:UpdateItem
                - dynamodb:DeleteItem
              Resource: !GetAtt IdempotencyTable.Arn
      Environment:
        Variables:
          IDEMPOTENCY_TABLE_NAME: !Ref IdempotencyTable
          IDEMPOTENCY_EXPIRES_SECONDS: "86400" # 24h
      Events:
        CreatePaymentApi:
          Type: Api
          Properties:
            Path: /payments
            Method: post

  IdempotencyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      TimeToLiveSpecification:
        AttributeName: expiration
        Enabled: true
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      SSESpecification:
        SSEEnabled: true

Outputs:
  ApiUrl:
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/payments"

Notes on the table schema vs Powertools

AWS Lambda Powertools idempotency utility has its own default attribute names and record model. If I use the library, I usually let it manage the item shape and only customize when I truly need to.

That said, I still think through the conceptual schema above so the team understands what is being stored and why.

Implementation option 1 (recommended): AWS Lambda Powertools idempotency utility

If I am using Python Lambda functions, AWS Lambda Powertools is my default choice. It saves me from reimplementing concurrency locks, conditional checks, and record state transitions from scratch.

Install

pip install aws-lambda-powertools[boto3]

API Gateway example with idempotency (Python)

This example assumes the client sends:

Idempotency-Key header
JSON body containing payment details

I use:

event_key_jmespath to extract the idempotency key from headers
payload_validation_jmespath to validate that key reuse with different payloads is detected
a DynamoDB persistence layer

import json
import os
from typing import Any, Dict

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent,
)
from aws_lambda_powertools.utilities.idempotency.exceptions import (
    IdempotencyValidationError,
)

logger = Logger(service="payments-api")

TABLE_NAME = os.environ["IDEMPOTENCY_TABLE_NAME"]
EXPIRES_SECONDS = int(os.getenv("IDEMPOTENCY_EXPIRES_SECONDS", "86400"))

persistence_layer = DynamoDBPersistenceLayer(table_name=TABLE_NAME)

config = IdempotencyConfig(
    # API Gateway/Lambda proxy event header lookup (normalize to lower-case in code if needed)
    event_key_jmespath="headers.idempotency-key",
    # Payload fields that should remain consistent when reusing the same key
    payload_validation_jmespath="powertools_json(body).[customerId, amount, currency]",
    expires_after_seconds=EXPIRES_SECONDS,
    use_local_cache=True,
)

def create_payment_intent(request: Dict[str, Any]) -> Dict[str, Any]:
    # Replace this with your real business logic / external payment call.
    # The critical point is that this function is wrapped with idempotency.
    payment_id = f"pay_{request['customerId']}_{request['amount']}_{request['currency']}"
    return {
        "paymentId": payment_id,
        "status": "AUTHORIZED",
        "amount": request["amount"],
        "currency": request["currency"],
    }

@idempotent(config=config, persistence_store=persistence_layer)
def process_request(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
    body = json.loads(event.get("body") or "{}")
    required = ["customerId", "amount", "currency"]
    missing = [k for k in required if k not in body]
    if missing:
        return {
            "statusCode": 400,
            "body": json.dumps({"message": f"Missing required fields: {missing}"})
        }

    result = create_payment_intent(body)

    # Powertools can persist and return this response for duplicate calls
    return {
        "statusCode": 200,
        "body": json.dumps(result),
        "headers": {"Content-Type": "application/json"},
    }

@logger.inject_lambda_context(log_event=False)
def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
    # Normalize headers so event_key_jmespath is predictable
    headers = event.get("headers") or {}
    event["headers"] = {str(k).lower(): v for k, v in headers.items()}

    try:
        return process_request(event, context)
    except IdempotencyValidationError:
        return {
            "statusCode": 409,
            "body": json.dumps({
                "message": "Idempotency key was reused with a different request payload"
            }),
            "headers": {"Content-Type": "application/json"},
        }

Why this pattern works well

If the same request is retried, Powertools returns the stored response.
If the same key is reused with a different body, I return 409 Conflict.
I avoid duplicate payment authorization for simple retries.

API client contract (important and often skipped)

Idempotency works much better when the contract is explicit.

For API producers/clients, I document:

Idempotency-Key is required for mutating operations (POST, sometimes PATCH)
same key + same intent/payload -> safe retry
same key + different payload -> 409 Conflict
dedupe window (for example, 24h)
response replay behavior (cached result may be returned)

That avoids ambiguity across frontend, mobile, and backend teams.

Handling retries from SQS (Lambda event source mapping)

SQS is one of the most common places where teams need idempotency but only discover it after duplicates happen.

What changes for SQS?

Lambda receives a batch of messages.
Some messages may succeed while others fail.
I should use partial batch response so only failed records are retried.
Each message should still be processed idempotently.

Key design for SQS consumers

I avoid using transient delivery metadata. I prefer:

a business key in the message body (for example orderId)
or an upstream event ID included in the message

Example key:

order-paid:{orderId}
inventory-reservation:{reservationId}

SQS consumer example (Powertools Batch + per-record idempotency)

Below is a simplified Python example using:

Powertools Batch utility for partial batch handling
Powertools idempotency on the per-record processing function

import json
import os
from typing import Any, Dict

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response
from aws_lambda_powertools.utilities.batch.types import PartialItemFailureResponse
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function,
)

logger = Logger(service="orders-sqs-worker")

TABLE_NAME = os.environ["IDEMPOTENCY_TABLE_NAME"]
persistence_layer = DynamoDBPersistenceLayer(table_name=TABLE_NAME)

# Here we build the key from the function argument "data"
# (Powertools hashes the configured subset to create an idempotency key)
idempotency_config = IdempotencyConfig(
    event_key_jmespath="orderId",
    payload_validation_jmespath="[orderId, action, version]",
    expires_after_seconds=3 * 24 * 60 * 60,  # 3 days
)

processor = BatchProcessor(event_type=EventType.SQS)

@idempotent_function(data_keyword_argument="data", config=idempotency_config, persistence_store=persistence_layer)
def process_order_event(*, data: Dict[str, Any]) -> Dict[str, Any]:
    # Business logic here (must be safe to retry / replay via idempotency)
    # Example: reserve inventory, update status, publish follow-up event, etc.
    logger.info("Processing order event", extra={"orderId": data["orderId"], "action": data["action"]})
    return {"ok": True, "orderId": data["orderId"], "action": data["action"]}

def record_handler(record: SQSRecord) -> Dict[str, Any]:
    payload = json.loads(record.body)
    return process_order_event(data=payload)

def lambda_handler(event, context) -> PartialItemFailureResponse:
    return process_partial_response(
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context,
    )

Best practices I apply with SQS + idempotency

Use partial batch response to avoid retrying already-successful records.
Set visibility timeout longer than worst-case processing time (or heartbeat/extend strategy).
Keep the idempotency key in the message payload, not delivery metadata.
Use a dedupe window long enough to cover retries, DLQ redrive, and operational replay.

Handling retries from EventBridge targets

EventBridge makes event-driven architecture clean, but duplicate-safe consumers are still my responsibility.

EventBridge-specific considerations

EventBridge may retry target delivery.
Archive/replay can intentionally re-send events.
Different publishers may emit semantically duplicate events unless the contract is strict.

Key strategy for EventBridge consumers

If the event has a stable id or domain event ID, I use it. If not, I derive one from:

detail-type
source/domain identifier
business entity ID
action/version

Example:

eventbridge:{source}:{detailType}:{detail.orderId}:{detail.version}

EventBridge consumer example (Python Lambda + Powertools idempotency)

import os
from typing import Any, Dict

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent,
)

logger = Logger(service="eventbridge-consumer")

TABLE_NAME = os.environ["IDEMPOTENCY_TABLE_NAME"]
persistence_layer = DynamoDBPersistenceLayer(table_name=TABLE_NAME)

config = IdempotencyConfig(
    # Prefer a producer-defined unique event ID if available in detail
    event_key_jmespath="detail.eventId || id",
    payload_validation_jmespath="[source, detail-type, detail.orderId, detail.version]",
    expires_after_seconds=7 * 24 * 60 * 60,
)

@idempotent(config=config, persistence_store=persistence_layer)
def handle_event(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
    detail = event["detail"]
    logger.info(
        "Handling EventBridge event",
        extra={"eventId": detail.get("eventId", event.get("id")), "orderId": detail.get("orderId")}
    )

    # Execute your domain logic here
    # Example: update read model, trigger notification, call downstream API, etc.
    return {"handled": True, "orderId": detail.get("orderId")}

def lambda_handler(event, context):
    return handle_event(event, context)

When I implement idempotency manually (without Powertools)

Powertools is my default, but sometimes I implement manually when:

I need a custom record schema shared across services/languages
I need custom conflict behavior beyond the library defaults
I am in a language/runtime where I am standardizing a platform abstraction
I want explicit control over lock and result update semantics

The key principle stays the same: use DynamoDB conditional writes.

Manual DynamoDB idempotency pattern (Python + boto3)

The pattern below shows the core idea:

Try to write IN_PROGRESS with attribute_not_exists(id)
If successful, execute business logic
Update item to COMPLETED and store result
If conditional check fails, inspect existing item and decide

import json
import time
import hashlib
import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.client("dynamodb")
TABLE_NAME = "IdempotencyTable"

def sha256_text(value: str) -> str:
    return hashlib.sha256(value.encode("utf-8")).hexdigest()

def acquire_lock(idempotency_key: str, payload_hash: str, ttl_seconds: int = 86400, lock_seconds: int = 120):
    now = int(time.time())
    item = {
        "id": {"S": idempotency_key},
        "status": {"S": "IN_PROGRESS"},
        "payloadHash": {"S": payload_hash},
        "expiration": {"N": str(now + ttl_seconds)},
        "inProgressExpiryTimestamp": {"N": str(now + lock_seconds)},
        "createdAt": {"N": str(now)},
        "updatedAt": {"N": str(now)},
    }
    try:
        dynamodb.put_item(
            TableName=TABLE_NAME,
            Item=item,
            ConditionExpression="attribute_not_exists(id)"
        )
        return {"acquired": True}
    except ClientError as e:
        if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise
        return {"acquired": False}

def get_record(idempotency_key: str):
    resp = dynamodb.get_item(
        TableName=TABLE_NAME,
        Key={"id": {"S": idempotency_key}},
        ConsistentRead=True,
    )
    return resp.get("Item")

def mark_completed(idempotency_key: str, response_obj: dict):
    now = int(time.time())
    dynamodb.update_item(
        TableName=TABLE_NAME,
        Key={"id": {"S": idempotency_key}},
        UpdateExpression="SET #s = :completed, responseData = :resp, updatedAt = :now",
        ExpressionAttributeNames={"#s": "status"},
        ExpressionAttributeValues={
            ":completed": {"S": "COMPLETED"},
            ":resp": {"S": json.dumps(response_obj)},
            ":now": {"N": str(now)},
        },
    )

def handler(event, context):
    body = json.loads(event["body"])
    idem_key = event["headers"]["idempotency-key"]
    payload_hash = sha256_text(json.dumps({
        "customerId": body["customerId"],
        "amount": body["amount"],
        "currency": body["currency"],
    }, sort_keys=True))

    lock = acquire_lock(idem_key, payload_hash)
    if not lock["acquired"]:
        existing = get_record(idem_key)
        if not existing:
            # Rare race / eventual state issue; safe to retry
            raise Exception("Retry request")
        if existing.get("payloadHash", {}).get("S") != payload_hash:
            return {"statusCode": 409, "body": json.dumps({"message": "Idempotency key payload mismatch"})}
        status = existing["status"]["S"]
        if status == "COMPLETED":
            return {"statusCode": 200, "body": existing["responseData"]["S"]}
        if status == "IN_PROGRESS":
            # For API workflows you might return 409 or 425-style retry signal (implementation-specific)
            return {"statusCode": 409, "body": json.dumps({"message": "Request is already in progress"})}

    # Execute business logic after lock acquired
    result = {"paymentId": "pay_123", "status": "AUTHORIZED"}
    mark_completed(idem_key, result)
    return {"statusCode": 200, "body": json.dumps(result)}

Manual pattern caveats

If I go manual, I also need to think about:

expired IN_PROGRESS lock recovery
exception handling and safe cleanup
serialization of cached responses
metrics for duplicate hits vs fresh requests
consistent behavior across all event sources

That is exactly why Powertools is usually the better default.

End-to-end implementation discussion (how I wire this in production)

This is the part I care about most in architecture reviews: not just the code, but where idempotency sits in the overall system.

1) Idempotency belongs close to the handler boundary

I usually apply idempotency at the Lambda entry point (or record handler for batches), before business side effects occur.

Why:

it prevents duplicate external calls
it keeps the protection broad
it makes retries safe by default

2) I still design downstream writes carefully

Idempotency at the Lambda layer is great, but if the function can partially succeed before crashing, I also check downstream safety:

unique constraints in relational DBs
conditional writes in DynamoDB
provider-side idempotency support for payment APIs or webhooks

Think in layers, not in a single magic switch.

3) I define duplicate behavior per source

The response to a duplicate is not always the same.

API Gateway: return cached success response (best UX)
SQS: ack success for already-processed message, avoid poison-loop
EventBridge: safely no-op or return success after dedupe

4) I separate “idempotency key” and “correlation ID”

They are related but not identical.

Correlation ID -> tracing/observability
Idempotency key -> duplicate suppression for a specific operation

Sometimes they can be the same, but I do not assume that.

Handling edge cases and failure modes

Edge case 1: Same key, different payload

This should be treated as a contract violation.

Best practice: return 409 Conflict (or equivalent domain error) and log it loudly.

Why I do this:

protects clients from accidental misuse
prevents serving incorrect cached results
surfaces integration bugs early

Edge case 2: Function times out after making a side effect

This is the classic ambiguous outcome problem.

Idempotency helps, but only if:

the side effect can be detected or safely repeated, and/or
the result gets persisted before timeout

Best practices:

keep timeouts realistic
use downstream idempotency where available
break long operations into Step Functions when needed
persist progress checkpoints for multi-step work

Edge case 3: `IN_PROGRESS` records stuck after crashes

If a function crashes after acquiring the lock, duplicates may keep seeing IN_PROGRESS.

Best practices:

use an in-progress lock expiry
make retries back off
alert on sustained IN_PROGRESS accumulation
evaluate whether the operation is safe to re-attempt after lock expiry

Edge case 4: Replay and backfill

Operational replay is common and healthy. I design for it intentionally.

Best practices:

choose a dedupe window that matches replay expectations
if replay should re-run effects, use a different idempotency namespace/version
document replay semantics for ops teams

Example:

normal key: invoice-email:{invoiceId}
forced replay key namespace: invoice-email:replay:{jobId}:{invoiceId}

Observability: what I monitor for idempotency

Idempotency is not just a code concern. I want to know how often it is being exercised and whether it is hiding a deeper issue.

Metrics I like to emit

IdempotencyFreshRequests
IdempotencyDuplicateHits
IdempotencyInProgressConflicts
IdempotencyValidationConflicts (same key, different payload)
IdempotencyStoreErrors
handler latency split by fresh vs duplicate

Logs I always include

idempotency key (or redacted/hash if sensitive)
source (api, sqs, eventbridge)
dedupe outcome (fresh, duplicate_completed, duplicate_in_progress, validation_mismatch)
business identifier (orderId, paymentId, etc.)

This makes incident triage much faster.

Practical best practices checklist (the part I use in reviews)

Key selection

[ ] Key represents a business operation, not transport metadata
[ ] Key is stable across retries/replays
[ ] Key cardinality is high enough to avoid false collisions
[ ] Payload mismatch with same key is detected

Dedupe window

[ ] TTL matches retry + redrive + replay realities
[ ] Expiry is checked in code (not only by DynamoDB TTL cleanup)
[ ] Window is documented in API/event contract

Store design

[ ] DynamoDB conditional write used for first writer wins
[ ] IN_PROGRESS and COMPLETED states are handled explicitly
[ ] Cached response/ack strategy is defined
[ ] Encryption, backups/PITR, and least privilege are configured

Source-specific behavior

[ ] API duplicates return deterministic response
[ ] SQS uses partial batch response
[ ] EventBridge consumer supports replay safely
[ ] Redrive/replay semantics are documented for ops

Operations

[ ] Metrics and alarms exist for duplicate spikes and store failures
[ ] Logs include dedupe outcomes and business IDs
[ ] Runbooks cover stale IN_PROGRESS records and replay scenarios

Common mistakes I see (and how I avoid them)

Mistake 1: Using Lambda `requestId` as the idempotency key

That only identifies the invocation, not the logical request.

Fix: use business operation identity or client-provided idempotency key.

Mistake 2: Assuming FIFO queue means I do not need idempotency

FIFO helps with ordering and deduplication windows, but it does not replace end-to-end idempotency for all side effects and replay paths.

Fix: still make the consumer idempotent.

Mistake 3: Dedupe only at the API layer

Then an async worker downstream duplicates the side effect anyway.

Fix: apply idempotency where side effects happen, especially in SQS/EventBridge consumers.

Mistake 4: No payload validation on key reuse

This can return the wrong cached response and create hidden data integrity issues.

Fix: validate a stable subset of the payload with the idempotency key.

Mistake 5: Too-short TTL

The key expires before retries/redrives finish, so duplicates sneak through.

Fix: pick TTL based on actual operational timelines, not guesswork.

Final thoughts

If I had to summarize production-grade idempotency architecture in one line, it would be this:

Design for duplicate delivery as normal behavior, then make your Lambda handlers safe, deterministic, and observable.

AWS gives us excellent building blocks for this:

Lambda
SQS / EventBridge / API Gateway
DynamoDB conditional writes
AWS Lambda Powertools idempotency utility

When I combine them intentionally, retries stop being scary and start being a reliability feature instead of a data integrity risk.

If you are building Lambda-driven systems that write to money, inventory, notifications, or customer state, idempotency is not optional. It is a core part of the architecture.

That placement helps readers understand the flow before diving into implementation details.

References

AWS Lambda Powertools (Python) documentation
AWS Lambda developer guide
Amazon SQS developer guide (Lambda event source mappings / retries / partial batch response)
Amazon EventBridge documentation (retries, targets, replay/archive)
Amazon DynamoDB documentation (conditional writes, TTL, PITR)

Building a Serverless Command Bus / Workflow Backbone for Microservices

Renaldi — Sun, 15 Mar 2026 22:00:00 +0000

When I design microservice platforms on AWS, one of the recurring problems I see is not “how do I trigger a Lambda,” but rather “how do I keep a growing set of services coordinated, observable, and governed without turning everything into tight coupling?”

That is where a serverless command bus plus workflow backbone becomes a powerful pattern.

In this post, I will walk through how I build this in practice using Amazon EventBridge and AWS Step Functions, and I will cover:

Commands vs events
EventBridge vs direct invoke
Step Functions for orchestrated flows
Auditability and tracing
Contract evolution
Governance and domain boundaries

I will also include an end-to-end implementation walkthrough, code samples, and architecture guidance so this is not just conceptual.

Why this pattern is worth building

As systems grow, teams usually end up with some mixture of:

synchronous service-to-service calls
ad hoc retries
duplicated orchestration logic
inconsistent logging/correlation IDs
unclear ownership of contracts
event naming drift
“mystery failures” across multiple services

A command bus/workflow backbone gives you a more intentional architecture:

Commands are routed to the right handler or orchestrator
Workflows coordinate multi-step business actions
Events are published as business facts for downstream consumers
Tracing/audit become standard instead of optional
Contract governance is enforced at boundaries rather than after incidents

This is especially effective for domains like:

order fulfillment
onboarding/KYC
claims processing
content moderation pipelines
payment settlement
inventory reservation
fulfillment and notification fan-out

What I mean by “Command Bus” in a serverless context

In classic application architecture, a command bus is often an in-process dispatching mechanism. In distributed systems, I use the term a bit differently:

A command represents an intent to perform a business action
The command bus is the routing layer that accepts commands and sends them to the correct handler/orchestrator
A workflow backbone is the orchestration mechanism that executes multi-step business flows and emits events

On AWS, a practical implementation is:

API Gateway + Lambda for ingress
EventBridge as the command routing fabric
Step Functions as the workflow engine
EventBridge (domain event bus) for business events
DynamoDB / S3 / CloudWatch / X-Ray / OTEL for auditability and observability

Commands vs Events (and why this distinction matters)

This is the first thing I standardize with teams, because it affects naming, routing, retries, and ownership.

Commands

A command is an instruction or request to do something.

Examples:

CreateOrder
ApproveLoanApplication
ReserveInventory
InitiateRefund

Commands are typically:

intent-driven
often addressed to one owning domain/service
can be accepted/rejected
may need validation and idempotency
often require stronger governance at the boundary

A command implies someone is asking for work to be done.

Events

An event is a fact that something already happened.

Examples:

OrderCreated
InventoryReserved
PaymentAuthorized
RefundInitiated

Events are typically:

past-tense facts
can have many subscribers
should be treated as immutable
support decoupled reactions
are excellent for read models, analytics, notifications, integrations

An event should not be phrased as an instruction.

A rule I use with teams

If the payload is saying, “please do this,” it is a command.

If the payload is saying, “this happened,” it is an event.

That sounds simple, but it prevents a lot of design drift.

EventBridge vs direct invoke (when to use each)

I do not recommend forcing everything through a bus. Some interactions should remain direct and synchronous.

Use EventBridge when you want

loose coupling between producers and consumers
routing based on message metadata
asynchronous execution
fan-out to multiple consumers
cross-account integration patterns
central governance on message flow
easy addition of new downstream consumers later

Use direct invoke (HTTP/gRPC/Lambda invoke) when you need

immediate response to the caller
low latency request/response semantics
tight transactional dependencies (within reason)
user-facing UX that cannot tolerate async polling/callbacks
simple service composition where orchestration overhead is not justified

My practical guidance

I usually use a hybrid model:

Direct invoke for request/response reads and tightly-coupled synchronous operations
Command bus + Step Functions for business workflows and long-running actions
Event bus for domain events and downstream side effects

That gives you the best of both worlds without ideological overreach.

Reference Architecture

Architecture overview

At a high level, the flow is:

Client sends a command (e.g., CreateOrder)
API layer validates, stamps correlation metadata, applies idempotency
Command is published to a Command Bus (EventBridge)
Rule routes command to a Step Functions state machine
Workflow invokes domain steps (fraud, payment, inventory, order write)
Workflow emits domain events (e.g., OrderCreated or OrderRejected)
Event consumers react independently (notifications, analytics, CRM, projections)
Audit and tracing data is captured throughout

This separates:

request intent ingestion
workflow orchestration
event publication
consumer reactions
governance/observability

End-to-End Walkthrough: `CreateOrder` command

To make this concrete, I will use an order workflow.

Business scenario

A client submits an order. The system needs to:

Validate the request and check idempotency
Run a fraud check
Authorize payment
Reserve inventory
Persist the order
Publish the resulting business event(s)
Notify downstream systems

Step 1: Client submits a command

The client calls an API endpoint such as:

POST /orders

Internally, I treat this as a CreateOrderCommand.

The API layer does not directly orchestrate business logic. Its job is to:

authenticate/authorize
validate payload shape
assign commandId and correlationId
apply idempotency safeguards
publish the command to the bus

Step 2: Command is routed on the command bus

The command lands on an EventBridge bus (logical “command bus”).

A rule matches:

source
detail-type
possibly domain/version metadata

That rule starts a Step Functions state machine for CreateOrder.

This gives me routing flexibility without wiring the producer directly to the orchestrator.

Step 3: Step Functions orchestrates the business flow

The state machine runs each step with retries and error handling.

Typical flow:

FraudCheck
AuthorizePayment
ReserveInventory
PersistOrder
PublishOrderCreatedEvent

If any step fails:

compensate if needed (e.g., release reservation / void payment)
emit failure event (e.g., OrderRejected, PaymentFailed)
write audit trail

Step 4: Domain events are published

Once the workflow succeeds or fails, the workflow emits a domain event to an EventBridge event bus (logical “domain event bus”).

Examples:

OrderCreated
OrderRejected
PaymentFailed

These events can fan out to multiple consumers without changing the workflow code.

Step 5: Consumers react independently

Downstream services subscribe and act:

Notifications send email/SMS/push
Analytics ingests event into a lake/warehouse
CRM updates lifecycle state
Projection service updates a query/read model

None of these consumers need to be in the critical path of order creation.

Implementation discussion (practical, not just theoretical)

Below is a concrete implementation shape I use and recommend.

Message contract design (command and event envelopes)

I strongly recommend standardizing a message envelope early. It pays off in observability, replay, governance, and contract evolution.

Command envelope example

{
  "meta": {
    "messageType": "command",
    "commandName": "CreateOrder",
    "schemaVersion": "1.0",
    "commandId": "cmd_01HSYJQF3A6M0R6G0J7VY7A1J9",
    "correlationId": "corr_2f1d3a07-3d89-4a62-9f6c-9f29f4e96f7e",
    "causationId": null,
    "occurredAt": "2026-02-25T09:20:00Z",
    "tenantId": "tenant-001",
    "initiator": {
      "type": "user",
      "id": "user-123"
    }
  },
  "routing": {
    "domain": "orders",
    "target": "orders.workflow",
    "priority": "normal"
  },
  "data": {
    "orderId": "ORD-100045",
    "customerId": "CUST-9001",
    "currency": "AUD",
    "items": [
      { "sku": "SKU-123", "qty": 2, "unitPrice": 49.95 },
      { "sku": "SKU-999", "qty": 1, "unitPrice": 19.95 }
    ],
    "paymentMethodToken": "tok_abc123"
  }
}

Event envelope example

{
  "meta": {
    "messageType": "event",
    "eventName": "OrderCreated",
    "schemaVersion": "1.1",
    "eventId": "evt_01HSYK2GQKCNZC3Q0Z6R3T9M7K",
    "correlationId": "corr_2f1d3a07-3d89-4a62-9f6c-9f29f4e96f7e",
    "causationId": "cmd_01HSYJQF3A6M0R6G0J7VY7A1J9",
    "occurredAt": "2026-02-25T09:20:03Z",
    "producer": "orders.workflow"
  },
  "subject": {
    "domain": "orders",
    "id": "ORD-100045"
  },
  "data": {
    "orderId": "ORD-100045",
    "status": "CREATED",
    "paymentStatus": "AUTHORIZED",
    "inventoryStatus": "RESERVED",
    "totalAmount": 119.85,
    "currency": "AUD"
  }
}

Why I standardize envelopes

This makes it much easier to implement:

trace propagation
replay tooling
audit logging
schema validation
version migration
tenant-aware governance

API ingress (Lambda) for command intake

This example shows a command-ingress Lambda that validates input, applies idempotency, and publishes to EventBridge.

// src/handlers/create-order-command.ts
import { APIGatewayProxyEventV2, APIGatewayProxyStructuredResultV2 } from "aws-lambda";
import { EventBridgeClient, PutEventsCommand } from "@aws-sdk/client-eventbridge";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";
import { randomUUID } from "crypto";

const eb = new EventBridgeClient({});
const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));

const COMMAND_BUS_NAME = process.env.COMMAND_BUS_NAME!;
const IDEMPOTENCY_TABLE = process.env.IDEMPOTENCY_TABLE!;
const COMMAND_SOURCE = "com.example.orders.api";

export const handler = async (
  event: APIGatewayProxyEventV2
): Promise<APIGatewayProxyStructuredResultV2> => {
  const correlationId = event.headers["x-correlation-id"] ?? `corr_${randomUUID()}`;
  const idempotencyKey = event.headers["idempotency-key"] ?? `idem_${randomUUID()}`;

  if (!event.body) {
    return { statusCode: 400, body: JSON.stringify({ message: "Missing request body" }) };
  }

  const body = JSON.parse(event.body);

  // Minimal shape validation (use JSON Schema in production)
  if (!body.customerId || !Array.isArray(body.items) || body.items.length === 0) {
    return { statusCode: 400, body: JSON.stringify({ message: "Invalid order payload" }) };
  }

  // Idempotency write (fail if key already exists)
  try {
    await ddb.send(
      new PutCommand({
        TableName: IDEMPOTENCY_TABLE,
        Item: {
          pk: idempotencyKey,
          createdAt: new Date().toISOString(),
          correlationId
        },
        ConditionExpression: "attribute_not_exists(pk)"
      })
    );
  } catch (err: any) {
    if (err.name === "ConditionalCheckFailedException") {
      return {
        statusCode: 409,
        body: JSON.stringify({ message: "Duplicate request", correlationId, idempotencyKey })
      };
    }
    throw err;
  }

  const commandId = `cmd_${randomUUID()}`;
  const now = new Date().toISOString();

  const commandEnvelope = {
    meta: {
      messageType: "command",
      commandName: "CreateOrder",
      schemaVersion: "1.0",
      commandId,
      correlationId,
      causationId: null,
      occurredAt: now,
      tenantId: "tenant-001",
      initiator: { type: "user", id: "user-123" }
    },
    routing: {
      domain: "orders",
      target: "orders.workflow",
      priority: "normal"
    },
    data: body
  };

  await eb.send(
    new PutEventsCommand({
      Entries: [
        {
          EventBusName: COMMAND_BUS_NAME,
          Source: COMMAND_SOURCE,
          DetailType: "CreateOrderCommand",
          Time: new Date(now),
          Detail: JSON.stringify(commandEnvelope)
        }
      ]
    })
  );

  return {
    statusCode: 202,
    headers: {
      "content-type": "application/json",
      "x-correlation-id": correlationId
    },
    body: JSON.stringify({
      message: "Command accepted",
      commandId,
      correlationId
    })
  };
};

A note on API response semantics

I usually return 202 Accepted for async workflows. That is clearer than pretending the business operation is already complete.

EventBridge routing for the command bus

I use EventBridge rules to map command types to orchestrators or handlers.

Routing rule concept

source = com.example.orders.api
detail-type = CreateOrderCommand
target = Step Functions state machine

This is a clean handoff between ingress and orchestration.

Step Functions for orchestrated flows

This is the backbone of the pattern for multi-step business actions.

Step Functions gives you:

retries/backoff
explicit error branches
timeouts
compensation flows
state tracking
execution history
integration with Lambda and AWS services

Example state machine (Amazon States Language)

{
  "Comment": "CreateOrder workflow",
  "StartAt": "FraudCheck",
  "States": {
    "FraudCheck": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${FraudCheckFnArn}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "fraud.$": "$.Payload"
      },
      "ResultPath": "$.fraudResult",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException", "States.TaskFailed"],
          "IntervalSeconds": 2,
          "BackoffRate": 2.0,
          "MaxAttempts": 3
        }
      ],
      "Next": "FraudDecision"
    },
    "FraudDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.fraudResult.fraud.approved",
          "BooleanEquals": true,
          "Next": "AuthorizePayment"
        }
      ],
      "Default": "PublishOrderRejected"
    },
    "AuthorizePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${AuthorizePaymentFnArn}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "payment.$": "$.Payload"
      },
      "ResultPath": "$.paymentResult",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "PublishPaymentFailed"
        }
      ],
      "Next": "ReserveInventory"
    },
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${ReserveInventoryFnArn}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "inventory.$": "$.Payload"
      },
      "ResultPath": "$.inventoryResult",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "PublishOrderRejected"
        }
      ],
      "Next": "PersistOrder"
    },
    "PersistOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${PersistOrderFnArn}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "order.$": "$.Payload"
      },
      "ResultPath": "$.orderResult",
      "Next": "PublishOrderCreated"
    },
    "PublishOrderCreated": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "EventBusName": "${DomainEventBusName}",
            "Source": "com.example.orders.workflow",
            "DetailType": "OrderCreated",
            "Detail.$": "States.JsonToString($.orderResult.order.eventEnvelope)"
          }
        ]
      },
      "End": true
    },
    "PublishPaymentFailed": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "EventBusName": "${DomainEventBusName}",
            "Source": "com.example.orders.workflow",
            "DetailType": "PaymentFailed",
            "Detail.$": "States.JsonToString($.paymentResult.paymentFailureEvent)"
          }
        ]
      },
      "End": true
    },
    "PublishOrderRejected": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "EventBusName": "${DomainEventBusName}",
            "Source": "com.example.orders.workflow",
            "DetailType": "OrderRejected",
            "Detail.$": "States.JsonToString($.fraudResult.fraud.rejectionEvent)"
          }
        ]
      },
      "End": true
    }
  }
}

Why I like this approach

The orchestration is visible and reviewable. You can reason about:

what happens on success
what retries are safe
how failures are surfaced
which events are emitted
which steps need compensation

That is much harder when orchestration is hidden across multiple services.

Domain task implementation example (Lambda)

Here is a simplified payment authorization step.

// src/handlers/authorize-payment.ts
import { Context } from "aws-lambda";

type WorkflowInput = {
  meta: {
    correlationId: string;
    commandId?: string;
  };
  data: {
    orderId: string;
    customerId: string;
    currency: string;
    items: Array<{ sku: string; qty: number; unitPrice: number }>;
    paymentMethodToken: string;
  };
};

export const handler = async (input: WorkflowInput, context: Context) => {
  const correlationId = input.meta?.correlationId ?? "unknown";
  const totalAmount = input.data.items.reduce((sum, i) => sum + i.qty * i.unitPrice, 0);

  // Simulated payment auth
  const paymentAuthorized = true;

  console.log(JSON.stringify({
    level: "INFO",
    message: "Authorizing payment",
    correlationId,
    awsRequestId: context.awsRequestId,
    orderId: input.data.orderId,
    totalAmount
  }));

  if (!paymentAuthorized) {
    throw new Error("Payment authorization failed");
  }

  return {
    authorized: true,
    authorizationId: `auth_${Date.now()}`,
    amount: Number(totalAmount.toFixed(2)),
    currency: input.data.currency
  };
};

CDK example (wiring the core pieces)

Below is a simplified AWS CDK (TypeScript) example to show how the pieces connect.

// lib/command-bus-stack.ts
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as events from "aws-cdk-lib/aws-events";
import * as targets from "aws-cdk-lib/aws-events-targets";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import * as sfn from "aws-cdk-lib/aws-stepfunctions";

export class CommandBusStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const commandBus = new events.EventBus(this, "CommandBus", {
      eventBusName: "command-bus"
    });

    const domainEventBus = new events.EventBus(this, "DomainEventBus", {
      eventBusName: "domain-event-bus"
    });

    const idempotencyTable = new dynamodb.Table(this, "IdempotencyTable", {
      partitionKey: { name: "pk", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      timeToLiveAttribute: "ttl"
    });

    const createOrderCommandFn = new lambda.Function(this, "CreateOrderCommandFn", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "create-order-command.handler",
      code: lambda.Code.fromAsset("dist"),
      environment: {
        COMMAND_BUS_NAME: commandBus.eventBusName,
        IDEMPOTENCY_TABLE: idempotencyTable.tableName
      }
    });

    commandBus.grantPutEventsTo(createOrderCommandFn);
    idempotencyTable.grantReadWriteData(createOrderCommandFn);

    // Placeholder state machine - replace with full definition
    const createOrderStateMachine = new sfn.StateMachine(this, "CreateOrderStateMachine", {
      definitionBody: sfn.DefinitionBody.fromString(JSON.stringify({
        StartAt: "Done",
        States: { Done: { Type: "Succeed" } }
      }))
    });

    new events.Rule(this, "RouteCreateOrderCommand", {
      eventBus: commandBus,
      eventPattern: {
        source: ["com.example.orders.api"],
        detailType: ["CreateOrderCommand"]
      },
      targets: [new targets.SfnStateMachine(createOrderStateMachine)]
    });

    new cdk.CfnOutput(this, "CommandBusName", { value: commandBus.eventBusName });
    new cdk.CfnOutput(this, "DomainEventBusName", { value: domainEventBus.eventBusName });
  }
}

CDK notes from real projects

In production, I also add:

DLQs where supported
alarms on failed executions and throttles
log retention policies
encryption settings
resource policies for cross-account publishing
least-privilege IAM per state/task
tagging and cost allocation labels

Auditability and tracing (this is where the pattern really pays off)

A lot of teams build async systems and then struggle to answer a simple question:

“What happened to this request?”

I design for that from day one.

What I want to trace across the platform

At minimum:

correlationId (request-level trace)
commandId / eventId
causationId (what produced this)
workflow execution ID / ARN
domain entity ID (e.g., orderId)
tenant ID (if multi-tenant)
timestamps and status transitions

Where I record this

I typically combine:

structured logs (CloudWatch Logs)
execution history (Step Functions)
traces (X-Ray / OpenTelemetry where applicable)
audit store (DynamoDB or S3, depending on retention/query needs)

Audit log strategy (practical approach)

I usually write a normalized audit record for each important transition:

command accepted
workflow started
step completed/failed
event published
consumer processed/failed

This can be as simple as a DynamoDB table keyed by correlationId and timestamp, or an append-only S3 log for long-term retention.

Important distinction: audit vs debug logs

Debug logs help developers troubleshoot
Audit records help operators, compliance, and support teams reconstruct what happened

Do not assume one replaces the other.

Contract evolution (how to avoid breaking consumers)

This is one of the biggest long-term risks in event-driven systems.

My default rule: evolve, do not break

For events, I prefer:

additive changes
optional fields
versioned schemas
compatibility checks in CI
deprecation windows

For commands, I am stricter because commands are business API boundaries.

Practical contract evolution patterns

1) Envelope version + payload schema version

I keep envelope metadata stable and version the business schema independently.

{
  "meta": {
    "schemaVersion": "1.1"
  }
}

2) Add fields, do not rename/remove casually

Safer changes:

add optional field
add new event type (if semantics change)
add new metadata

Riskier changes:

renaming fields
changing field meaning
changing enum semantics
changing required/optional behavior

3) Prefer new event types when semantics change materially

If OrderCreated starts meaning something different, I usually create a new event type rather than mutating semantics.

4) Validate contracts in CI

I recommend automated checks for:

JSON schema validation
compatibility rules (backward/forward depending on consumer strategy)
sample payload fixtures
consumer contract tests

Command vs event evolution nuance

Commands are usually controlled at ingress and can be validated more strictly
Events may have many consumers and require more conservative change management

That is another reason to keep commands and events conceptually separate.

Governance and domain boundaries (the part teams often postpone)

A command bus is not just a technical component. It is a governance surface.

If you skip governance, you will eventually get:

inconsistent naming
ambiguous ownership
wildcard subscriptions everywhere
unbounded fan-out
accidental data exposure
“shared bus chaos”

Governance principles I use

1) Explicit ownership by domain

Each command/event type should have a clear owner.

Examples:

Orders owns CreateOrder, OrderCreated, OrderRejected
Inventory owns ReserveInventory (if exposed), InventoryReserved
Payments owns PaymentAuthorized, PaymentFailed

2) Naming conventions

I standardize at least:

source values (e.g., com.example.orders.workflow)
detail-type values (e.g., OrderCreated)
envelope metadata keys
versioning conventions

This prevents drift and simplifies routing/analytics.

3) Separate buses by concern (when warranted)

I often separate:

command bus (internal command routing)
domain event bus (business events)
optional integration bus (external partner/integration events)

In larger organizations, I may also separate by:

environment (dev, test, prod)
business domain
account boundaries
compliance/data residency needs

4) Least privilege on publishers and subscribers

Producers should not be able to publish arbitrary event types to every bus. IAM and EventBridge resource policies should reflect domain boundaries.

5) Data classification rules

Do not let sensitive payloads flow “because it is convenient.” Define:

PII handling rules
masking/redaction requirements
retention policies
replay restrictions
external egress controls

Failure handling and resilience (what makes this production-grade)

Async architecture is easy to demo and hard to run well unless failure paths are designed intentionally.

Areas I design explicitly

Idempotency

Commands can be retried by clients, gateways, or operators. Idempotency is non-negotiable for business writes.

Use:

client-provided idempotency key (preferred)
server-generated fallback only as backup
TTL on idempotency records to manage table growth

Retries and backoff

Retries should be defined per step, not copy-pasted blindly.

Questions I ask:

Is the step retry-safe?
Is the downstream side effect idempotent?
What is the blast radius if retried?
Do I need jitter/backoff?

Dead-letter and replay

Some failures need operator intervention, not endless retries.

I usually implement:

DLQ for failed async processing where supported
replay tooling with filters (by date, event type, correlation ID)
safe replay mode (e.g., dry-run validation before reprocessing)

Compensation

Not every workflow can be truly atomic. For distributed business flows, compensation matters.

Examples:

if inventory reservation succeeds but payment later fails, release reservation
if payment succeeds but order persistence fails, void/mark payment for manual review

Step Functions makes these paths explicit and testable.

Testing strategy (what I actually test)

1) Contract tests

validate command/event envelopes against schemas
enforce required metadata fields
check backward compatibility rules

2) Workflow tests

success path
step-specific failure paths
retries/timeouts
compensation branches

3) Consumer isolation tests

consumers should handle unknown fields
consumers should tolerate duplicate events
consumers should reject malformed payloads safely

4) Observability assertions

logs include correlation IDs
emitted events include causation/correlation metadata
alarms trigger on failed workflows

I treat observability metadata as part of the contract, not a nice-to-have.

When not to use this pattern

I like this pattern a lot, but I would not use it everywhere.

I avoid it when:

the flow is very simple and synchronous
latency is critical and async orchestration adds unnecessary overhead
the team is not ready to operate event-driven systems yet
there is no real need for decoupling/auditability
a direct service call is clearer and sufficient

Architecture should reduce complexity, not relocate it.

A practical rollout plan (if you are introducing this incrementally)

If I were introducing this into an existing microservice platform, I would phase it in:

Phase 1: Standardize envelopes and correlation IDs

Keep current calls, but enforce consistent metadata.

Phase 2: Introduce EventBridge for non-critical domain events

Start with notifications/analytics fan-out.

Phase 3: Move one business workflow to Step Functions

Choose a flow with clear business value and observable pain today.

Phase 4: Add command routing and idempotent ingress

Introduce a command bus for selected workflows.

Phase 5: Formalize governance and contract checks

Naming, ownership, CI validation, replay procedures, IAM boundaries.

This sequence reduces organizational friction while building platform confidence.

Final thoughts

The biggest advantage of a serverless command bus/workflow backbone is not just that it is “event-driven” or “serverless.” It is that it gives your microservices ecosystem a clear execution model:

commands express intent
workflows coordinate business actions
events publish facts
consumers react independently
audit and tracing are built in
governance is enforceable at boundaries

That combination is what helps a platform scale across both services and teams.

If I am building for long-term maintainability, this is one of the patterns I reach for early, especially when I know business workflows will grow in complexity and scrutiny.

References

AWS EventBridge documentation
AWS Step Functions documentation
AWS Lambda developer guide
AWS API Gateway developer guide
AWS CDK documentation
AWS X-Ray and observability documentation
AWS Well-Architected Framework (especially reliability and operational excellence guidance)
Event-driven architecture patterns and domain-driven design references (commands/events, bounded contexts, integration events)

Building with API Gateway, Lambda and DynamoDB Single-Table Design for Multi-Tenant SaaS

Renaldi — Sun, 08 Mar 2026 22:00:00 +0000

When I build multi-tenant SaaS on AWS, one of my favorite combinations is API Gateway + Lambda + DynamoDB. It gives me a clean serverless control plane, fast iteration, and a lot of flexibility in how I model data and enforce tenancy.

In this post, I will walk through an end-to-end implementation pattern for a multi-tenant SaaS API using:

Amazon API Gateway for API ingress
AWS Lambda for business logic
Amazon DynamoDB with a single-table design
Amazon Cognito + JWT for authentication
Per-tenant throttling and quotas using API Gateway usage plans (with a Lambda authorizer pattern)

I will focus on the architectural and data modeling decisions that matter in production:

Tenant isolation patterns
Partition key design
Hot partition mitigation
GSIs for access patterns
Auth context propagation (Cognito/JWT)
Per-tenant throttling and quotas

I will also include code and an implementation walkthrough so you can adapt the pattern to your own SaaS.

Why this stack works so well for multi-tenant SaaS

I like this stack because it lets me combine application-level tenancy controls with infrastructure-level scalability:

API Gateway gives me request routing, auth integration, throttling, and observability.
Lambda gives me a stateless execution layer where I can consistently apply tenant-aware logic.
DynamoDB gives me low-latency reads/writes and a data model that can be shaped around access patterns, which is exactly what multi-tenant SaaS needs.

The biggest challenge is not deploying these services. It is designing them so that one tenant cannot interfere with another tenant, and so that one noisy tenant does not dominate throughput.

That is where the design details matter.

What I am building in this walkthrough

To make the examples concrete, I will use a simple B2B issue tracking SaaS:

Each tenant is a customer organization
Each tenant has:
- users
- projects
- tickets
- comments
Users authenticate with Cognito
API calls are authorized via JWT and mapped to a tenant context
Data lives in one DynamoDB table (single-table design)

Example endpoints:

POST /projects/{projectId}/tickets
GET /projects/{projectId}/tickets?status=OPEN
GET /me/tickets
POST /tickets/{ticketId}/comments

Important design principle:

I do not trust the client to tell me the tenant ID in the path/body for authorization purposes. I derive tenancy from the validated JWT, then enforce it in my Lambda logic and DynamoDB key design.

Architecture Overview

This architecture uses:

Cognito to issue JWTs
API Gateway (REST API) as the front door
Lambda Authorizer to validate JWT and inject normalized auth context
Lambda handlers for business operations
DynamoDB single table for all entities
API Gateway usage plans for per-tenant throttling/quota (via authorizer metering key)

I am intentionally using a Lambda authorizer (instead of only a native Cognito authorizer) because it lets me:

normalize claims (tenantId, roles, plan)
enforce custom auth checks
return a usageIdentifierKey to support per-tenant usage plans and quotas in API Gateway REST API

End-to-end request flow (from login to DynamoDB write)

Let’s walk through a POST /projects/{projectId}/tickets call.

User signs in via Cognito
- Cognito issues a JWT (ID token or access token, depending your claim strategy)
- The token contains user identity and tenant-related claims (for example tenant_id, plan, roles)
Client calls API Gateway
- Sends Authorization: Bearer <JWT>
API Gateway invokes Lambda Authorizer
- Validates the JWT signature and issuer
- Extracts/normalizes claims
- Returns:
  - IAM policy (Allow)
  - context (tenant ID, user ID, roles, plan)
  - usageIdentifierKey (tenant-specific metering key for usage plan enforcement)
API Gateway invokes Lambda handler
- The handler reads the authorizer context from event.requestContext.authorizer
Lambda writes to DynamoDB
- Uses tenant-prefixed keys and tenant-aware access patterns
- Applies conditional expressions as needed
- Writes base item + GSI attributes
CloudWatch metrics/logs capture telemetry
- API Gateway request count/throttles
- Lambda duration/errors
- DynamoDB consumed capacity/throttles

This pattern keeps tenancy and throttling context consistent across the request path.

Tenant isolation patterns (and which one I am using)

Before writing a single key schema, I decide what kind of tenancy model I need.

1) Silo model (strongest isolation)

Each tenant gets isolated infrastructure (separate table / account / stack).

Pros

Strong isolation
Easier compliance stories for some customers
No noisy neighbor at data layer

Cons

Higher operational overhead
Harder fleet-wide schema changes
More expensive for small tenants

2) Pool model (shared infrastructure, tenant-aware app logic)

All tenants share the same infrastructure (same API, same Lambda, same table), but every request and every item is tenant-scoped.

Pros

Cost-efficient
Operationally simpler for most SaaS products
Scales well if keys are designed correctly

Cons

You must be disciplined about isolation in code and data model
Hot partitions/noisy neighbors need mitigation

3) Bridge model (hybrid)

Some tenants are pooled, some are siloed (for enterprise/compliance tiers).

Pros

Flexible business model
Lets you upsell isolation

Cons

More deployment and routing complexity

What I am using here

For this post, I am using the pool model with strong tenant-aware design:

Tenant ID is always derived from auth context
DynamoDB keys are tenant-prefixed
Access patterns are tenant-scoped
API Gateway usage plans enforce per-tenant limits
Lambda code refuses cross-tenant access even if a user manipulates IDs

This is a very practical pattern for SaaS teams that want speed and scale without overbuilding day one.

Single-table design mindset for multi-tenant SaaS

A lot of DynamoDB pain comes from designing tables like relational schemas.

I do the opposite: I start with access patterns, then design keys around them.

Access patterns I need

For this SaaS, I care about these reads/writes:

Create a ticket in a project
List tickets by project (optionally filtered by status)
List tickets assigned to the current user
Get a ticket by ID
Add/list comments for a ticket
List projects for a tenant
Look up user membership / role in a tenant

These access patterns will drive:

primary key design (PK, SK)
GSI design
where I shard to avoid hot partitions

Partition key design (tenant-aware and access-pattern-first)

This is the part that usually makes or breaks multi-tenant DynamoDB.

A common mistake

A very common first attempt is:

PK = TENANT#<tenantId>
SK = <everything else>

That can work for small tenants, but it becomes risky when a tenant is large or very active because all writes for that tenant concentrate on a small number of partitions.

My design approach

I still make keys tenant-aware, but I distribute writes across tenant sub-collections.

I use patterns like:

PK = TENANT#<tenantId> for tenant metadata and low-volume tenant-level items
PK = TENANT#<tenantId>#PROJECT#<projectId> for project and ticket collections
PK = TENANT#<tenantId>#TICKET#<ticketId> for ticket root + comments (if comment-heavy)
Sharded GSIs (or write-sharded partitions) for high-volume list patterns

This gives me tenant isolation without forcing every high-volume write into one tenant partition.

Example item model (single table)

I usually keep a few shared attributes on every item:

PK, SK
entityType
tenantId
timestamps (createdAt, updatedAt)
optional GSI attributes (GSI1PK, GSI1SK, GSI2PK, GSI2SK)

Example items

Tenant metadata

{
  "PK": "TENANT#t_123",
  "SK": "META",
  "entityType": "TENANT",
  "tenantId": "t_123",
  "name": "Acme Corp",
  "plan": "pro",
  "createdAt": "2026-02-25T00:00:00Z"
}

Project

{
  "PK": "TENANT#t_123",
  "SK": "PROJECT#p_001",
  "entityType": "PROJECT",
  "tenantId": "t_123",
  "projectId": "p_001",
  "name": "Platform",
  "createdAt": "2026-02-25T00:00:00Z"
}

Ticket (project-scoped partition)

{
  "PK": "TENANT#t_123#PROJECT#p_001",
  "SK": "TICKET#tk_9001",
  "entityType": "TICKET",
  "tenantId": "t_123",
  "projectId": "p_001",
  "ticketId": "tk_9001",
  "title": "API returns 500 on retry",
  "status": "OPEN",
  "priority": "HIGH",
  "assigneeUserId": "u_77",
  "createdByUserId": "u_12",
  "createdAt": "2026-02-25T01:00:00Z",
  "updatedAt": "2026-02-25T01:00:00Z",

  "GSI1PK": "TENANT#t_123#PROJECT#p_001#STATUS#OPEN",
  "GSI1SK": "UPDATED#2026-02-25T01:00:00Z#TICKET#tk_9001",

  "GSI2PK": "TENANT#t_123#ASSIGNEE#u_77",
  "GSI2SK": "UPDATED#2026-02-25T01:00:00Z#TICKET#tk_9001"
}

Comment (ticket-scoped partition)

{
  "PK": "TENANT#t_123#TICKET#tk_9001",
  "SK": "COMMENT#2026-02-25T01:05:00Z#c_001",
  "entityType": "COMMENT",
  "tenantId": "t_123",
  "ticketId": "tk_9001",
  "commentId": "c_001",
  "authorUserId": "u_12",
  "body": "Investigating now",
  "createdAt": "2026-02-25T01:05:00Z"
}

Why this layout works

Tenant is embedded in every key path
Tickets are grouped by project for efficient project views
Comments are grouped by ticket for efficient comment threads
User-assignee lookup is handled by GSI
Status-based listing is handled by GSI

This is a good default shape for many SaaS workloads.

GSIs for access patterns (and how I decide them)

I try to make every GSI exist for a specific query, not “just in case”.

GSI1: List tickets by project + status

Use case: GET /projects/{projectId}/tickets?status=OPEN

GSI1PK = TENANT#<tenantId>#PROJECT#<projectId>#STATUS#<status>
GSI1SK = UPDATED#<timestamp>#TICKET#<ticketId>

This lets me list tickets by status and sort by most recent updates.

GSI2: List tickets assigned to a user

Use case: GET /me/tickets

GSI2PK = TENANT#<tenantId>#ASSIGNEE#<userId>
GSI2SK = UPDATED#<timestamp>#TICKET#<ticketId>

This gives me a tenant-safe “my work” view.

Optional GSI3: Lookup by public ticket reference (if needed)

If you expose a user-friendly ticket reference like ACME-194, I may add a sparse GSI:

GSI3PK = TENANT#<tenantId>#REF#ACME-194
GSI3SK = TICKET#<ticketId>

I do not add this unless I truly need it.

Hot partition mitigation (the part many examples skip)

A single-table design can be “correct” and still fail under traffic if hot partitions are ignored.

Here are the patterns I use.

1) Avoid concentrating all writes into one tenant root partition

If all writes use PK = TENANT#<tenantId>, a large tenant can get hot fast.

Mitigation: partition by natural sub-aggregates (project, ticket thread, etc.)

2) Use write sharding for high-volume list/index patterns

Suppose one tenant has a huge number of ticket updates, and your status GSI gets hot.

I can shard the GSI partition key:

GSI1PK = TENANT#t_123#PROJECT#p_001#STATUS#OPEN#SHARD#03

Then query multiple shards and merge results in Lambda.

How do I pick the shard?

Hash ticketId or (ticketId + status) into N buckets
Start small (for example 4 or 8 shards)
Increase if traffic grows

Example helper:

function shardFor(value: string, shardCount = 8): string {
  let hash = 0;
  for (let i = 0; i < value.length; i++) {
    hash = (hash * 31 + value.charCodeAt(i)) >>> 0;
  }
  const shard = hash % shardCount;
  return shard.toString().padStart(2, "0");
}

3) Separate write-heavy entities from read-heavy aggregations

If comments are very high volume, I often avoid mixing them in the same partition as ticket metadata.

That is why in the example above I used:

ticket record under TENANT#...#PROJECT#...
comments under TENANT#...#TICKET#...

4) Use on-demand capacity early, then tune if needed

For newer SaaS products, I often start with on-demand capacity because it reduces operational tuning.

As traffic patterns stabilize, I decide whether provisioned + autoscaling is worth it.

5) Watch metrics by access pattern, not just table-wide

I monitor:

Throttled requests
Consumed read/write capacity
Latency by endpoint
GSI-specific pressure

If one endpoint is slow, I inspect whether the underlying key pattern is too concentrated.

Auth context propagation (Cognito/JWT -> API Gateway -> Lambda)

This is the glue that keeps tenant isolation consistent.

What I want from auth context

By the time my business Lambda runs, I want a normalized auth context like this:

tenantId
userId
roles
plan
(optional) scopes, email, orgId, isSupportUser

I do not want each Lambda handler parsing arbitrary JWT claims differently.

Cognito claim strategy

You have a few options:

Use Cognito ID token claims directly (simpler if tenant custom attributes are on the user profile)
Use Pre Token Generation trigger to inject normalized tenant claims into the token you use for APIs
Use groups + custom claims for role/plan mapping

For this post, I will assume the token contains:

sub
custom:tenant_id (or tenant_id)
custom:plan (or plan)
cognito:groups

Lambda Authorizer example (TypeScript)

This authorizer validates the Cognito JWT, extracts tenant context, and returns:

an Allow policy
authorizer context
usageIdentifierKey (for per-tenant API Gateway usage plan metering)

Note: In API Gateway REST APIs, usageIdentifierKey should be the API key value you want metered. In production, I usually map tenantId -> meteringKey and return the metering key, not the raw tenant ID.

// authorizer.ts
import type {
  APIGatewayTokenAuthorizerEvent,
  APIGatewayAuthorizerResult,
} from "aws-lambda";
import { CognitoJwtVerifier } from "aws-jwt-verify";

const userPoolId = process.env.USER_POOL_ID!;
const clientId = process.env.APP_CLIENT_ID!;

const verifier = CognitoJwtVerifier.create({
  userPoolId,
  tokenUse: "id", // or "access" if your claims are injected there
  clientId,
});

function buildPolicy(principalId: string, effect: "Allow" | "Deny", resource: string) {
  return {
    principalId,
    policyDocument: {
      Version: "2012-10-17",
      Statement: [
        {
          Action: "execute-api:Invoke",
          Effect: effect,
          Resource: resource,
        },
      ],
    },
  };
}

// Replace with a real lookup (DynamoDB / Secrets Manager / config service)
async function lookupMeteringKey(tenantId: string): Promise<string> {
  // Example only. In production, return the API Gateway API key *value* assigned to that tenant.
  return `meter-${tenantId}`;
}

export const handler = async (
  event: APIGatewayTokenAuthorizerEvent
): Promise<APIGatewayAuthorizerResult> => {
  try {
    const raw = event.authorizationToken || "";
    const token = raw.replace(/^Bearer\s+/i, "");

    const claims = await verifier.verify(token);

    const tenantId =
      (claims["custom:tenant_id"] as string | undefined) ??
      (claims["tenant_id"] as string | undefined);

    if (!tenantId) {
      throw new Error("Missing tenant claim");
    }

    const userId = String(claims.sub);
    const plan =
      String(
        (claims["custom:plan"] as string | undefined) ??
        (claims["plan"] as string | undefined) ??
        "free"
      );

    const groups = (claims["cognito:groups"] as string[] | undefined) ?? [];
    const meteringKey = await lookupMeteringKey(tenantId);

    const policy = buildPolicy(userId, "Allow", event.methodArn);

    return {
      ...policy,
      context: {
        tenantId,
        userId,
        plan,
        rolesJson: JSON.stringify(groups),
      },
      usageIdentifierKey: meteringKey,
    };
  } catch (err) {
    // API Gateway treats "Unauthorized" specially
    throw new Error("Unauthorized");
  }
};

Why I normalize here instead of in every Lambda

This avoids duplicated auth parsing and reduces inconsistent authorization logic across handlers.

It also gives me one place to:

map plans
map roles
support internal support-user impersonation rules (if needed)
connect metering/throttling to tenancy

Business Lambda: reading auth context and enforcing tenancy

Now let’s create a ticket.

This handler:

reads tenant context from the authorizer
ignores any tenant ID in the request body
writes a ticket item to DynamoDB using a tenant-prefixed key
populates GSIs for list access patterns

// create-ticket.ts
import type { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";
import { randomUUID } from "crypto";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE_NAME = process.env.TABLE_NAME!;

type AuthorizerCtx = {
  tenantId: string;
  userId: string;
  plan: string;
  rolesJson?: string;
};

function response(statusCode: number, body: unknown): APIGatewayProxyResult {
  return {
    statusCode,
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body),
  };
}

function getAuthContext(event: APIGatewayProxyEvent): AuthorizerCtx {
  const auth = (event.requestContext.authorizer ?? {}) as Record<string, unknown>;
  const tenantId = String(auth.tenantId ?? "");
  const userId = String(auth.userId ?? "");
  const plan = String(auth.plan ?? "free");
  const rolesJson = auth.rolesJson ? String(auth.rolesJson) : "[]";

  if (!tenantId || !userId) {
    throw new Error("Missing auth context");
  }

  return { tenantId, userId, plan, rolesJson };
}

function shardFor(value: string, shardCount = 8): string {
  let hash = 0;
  for (let i = 0; i < value.length; i++) {
    hash = (hash * 31 + value.charCodeAt(i)) >>> 0;
  }
  return String(hash % shardCount).padStart(2, "0");
}

export const handler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  try {
    const { tenantId, userId } = getAuthContext(event);
    const projectId = event.pathParameters?.projectId;
    if (!projectId) return response(400, { message: "projectId is required" });

    const body = event.body ? JSON.parse(event.body) : {};
    const title = String(body.title ?? "").trim();
    const assigneeUserId = body.assigneeUserId ? String(body.assigneeUserId) : undefined;
    const priority = body.priority ? String(body.priority) : "MEDIUM";

    if (!title) return response(400, { message: "title is required" });

    const ticketId = `tk_${randomUUID()}`;
    const now = new Date().toISOString();
    const status = "OPEN";

    // Optional GSI sharding for high-volume status lists
    const statusShard = shardFor(ticketId, 8);

    const item = {
      PK: `TENANT#${tenantId}#PROJECT#${projectId}`,
      SK: `TICKET#${ticketId}`,
      entityType: "TICKET",

      tenantId,
      projectId,
      ticketId,
      title,
      status,
      priority,
      assigneeUserId,
      createdByUserId: userId,
      createdAt: now,
      updatedAt: now,

      // List tickets by project + status
      GSI1PK: `TENANT#${tenantId}#PROJECT#${projectId}#STATUS#${status}#SHARD#${statusShard}`,
      GSI1SK: `UPDATED#${now}#TICKET#${ticketId}`,

      // List tickets by assignee
      ...(assigneeUserId && {
        GSI2PK: `TENANT#${tenantId}#ASSIGNEE#${assigneeUserId}`,
        GSI2SK: `UPDATED#${now}#TICKET#${ticketId}`,
      }),
    };

    await ddb.send(
      new PutCommand({
        TableName: TABLE_NAME,
        Item: item,
        ConditionExpression: "attribute_not_exists(PK) AND attribute_not_exists(SK)",
      })
    );

    return response(201, {
      ticketId,
      status,
      createdAt: now,
    });
  } catch (err) {
    console.error("create-ticket failed", err);
    return response(500, { message: "Internal server error" });
  }
};

Isolation note

The handler never accepts tenantId from the request body for authorization purposes.

Even if a client sends a different tenant ID, it does not matter because the handler uses tenantId from the validated authorizer context.

Querying with GSIs (list tickets by project + status)

Now let’s implement GET /projects/{projectId}/tickets?status=OPEN.

Because I sharded GSI1PK, I may need to query multiple shards and merge the results.

// list-project-tickets.ts
import type { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE_NAME = process.env.TABLE_NAME!;

function response(statusCode: number, body: unknown): APIGatewayProxyResult {
  return {
    statusCode,
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body),
  };
}

function getTenantId(event: APIGatewayProxyEvent): string {
  const auth = (event.requestContext.authorizer ?? {}) as Record<string, unknown>;
  const tenantId = String(auth.tenantId ?? "");
  if (!tenantId) throw new Error("Missing tenantId");
  return tenantId;
}

export const handler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  try {
    const tenantId = getTenantId(event);
    const projectId = event.pathParameters?.projectId;
    const status = String(event.queryStringParameters?.status ?? "OPEN");

    if (!projectId) return response(400, { message: "projectId is required" });

    const shardCount = 8;
    const queries = Array.from({ length: shardCount }, (_, n) => {
      const shard = String(n).padStart(2, "0");
      return ddb.send(
        new QueryCommand({
          TableName: TABLE_NAME,
          IndexName: "GSI1",
          KeyConditionExpression: "GSI1PK = :pk",
          ExpressionAttributeValues: {
            ":pk": `TENANT#${tenantId}#PROJECT#${projectId}#STATUS#${status}#SHARD#${shard}`,
          },
          Limit: 25, // tune based on your UX pagination model
          ScanIndexForward: false, // newest first
        })
      );
    });

    const results = await Promise.all(queries);
    const items = results.flatMap((r) => r.Items ?? []);

    // Merge and sort in-memory because we queried multiple shards
    items.sort((a, b) => String(b.updatedAt).localeCompare(String(a.updatedAt)));

    return response(200, {
      items,
      count: items.length,
    });
  } catch (err) {
    console.error("list-project-tickets failed", err);
    return response(500, { message: "Internal server error" });
  }
};

Tradeoff discussion

Sharding improves write scalability, but read logic is more complex because I need fan-out queries.

That is a normal DynamoDB tradeoff. I only add this complexity for access patterns that actually need it.

Per-tenant throttling and quotas (API Gateway + authorizer metering)

This is one of the most useful SaaS controls and one of the least discussed implementation details.

What I want

I want to enforce limits like:

Free tier tenant: lower RPS and daily quota
Pro tier tenant: higher RPS and quota
Enterprise tenant: custom limits

Why API Gateway usage plans help

API Gateway REST APIs support:

throttling
quotas
API keys
usage plans

With a Lambda authorizer, I can return a usageIdentifierKey so API Gateway meters/throttles by a tenant-specific key, even when the client authenticates with JWT.

Important implementation note

For REST APIs, set apiKeySourceType to AUTHORIZER so the usage key comes from the authorizer rather than an x-api-key header.

CDK sketch (TypeScript)

Below is a simplified CDK sketch to show the pattern. You will need to adapt it to your stack structure.

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as apigw from "aws-cdk-lib/aws-apigateway";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";

export class MultiTenantApiStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const table = new dynamodb.Table(this, "AppTable", {
      partitionKey: { name: "PK", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "SK", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      pointInTimeRecovery: true,
      removalPolicy: cdk.RemovalPolicy.RETAIN,
    });

    table.addGlobalSecondaryIndex({
      indexName: "GSI1",
      partitionKey: { name: "GSI1PK", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "GSI1SK", type: dynamodb.AttributeType.STRING },
      projectionType: dynamodb.ProjectionType.ALL,
    });

    table.addGlobalSecondaryIndex({
      indexName: "GSI2",
      partitionKey: { name: "GSI2PK", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "GSI2SK", type: dynamodb.AttributeType.STRING },
      projectionType: dynamodb.ProjectionType.ALL,
    });

    const authorizerFn = new lambda.Function(this, "AuthorizerFn", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "authorizer.handler",
      code: lambda.Code.fromAsset("dist/authorizer"),
      environment: {
        USER_POOL_ID: "your-user-pool-id",
        APP_CLIENT_ID: "your-app-client-id",
      },
    });

    const createTicketFn = new lambda.Function(this, "CreateTicketFn", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "create-ticket.handler",
      code: lambda.Code.fromAsset("dist/create-ticket"),
      environment: {
        TABLE_NAME: table.tableName,
      },
    });

    table.grantReadWriteData(createTicketFn);

    const api = new apigw.RestApi(this, "SaaSApi", {
      restApiName: "multi-tenant-saas-api",
      apiKeySourceType: apigw.ApiKeySourceType.AUTHORIZER, // key point for per-tenant metering
      deployOptions: {
        stageName: "prod",
        metricsEnabled: true,
        loggingLevel: apigw.MethodLoggingLevel.INFO,
        dataTraceEnabled: false,
        throttlingBurstLimit: 2000,
        throttlingRateLimit: 1000,
      },
    });

    const tokenAuthorizer = new apigw.TokenAuthorizer(this, "JwtAuthorizer", {
      handler: authorizerFn,
      resultsCacheTtl: cdk.Duration.minutes(5),
    });

    const projects = api.root.addResource("projects");
    const projectId = projects.addResource("{projectId}");
    const tickets = projectId.addResource("tickets");

    tickets.addMethod(
      "POST",
      new apigw.LambdaIntegration(createTicketFn),
      {
        authorizer: tokenAuthorizer,
        authorizationType: apigw.AuthorizationType.CUSTOM,
      }
    );

    // Example usage plan (create multiple plans for free/pro/enterprise)
    const freePlan = api.addUsagePlan("FreePlan", {
      name: "free-tier-plan",
      throttle: {
        rateLimit: 50,
        burstLimit: 100,
      },
      quota: {
        limit: 100000,
        period: apigw.Period.DAY,
      },
    });

    freePlan.addApiStage({
      api,
      stage: api.deploymentStage,
    });

    // API keys that represent tenant metering identities.
    // In production, create one per tenant and associate with the correct plan.
    const tenantApiKey = api.addApiKey("TenantApiKey_t_123", {
      apiKeyName: "tenant-t_123-metering-key",
      enabled: true,
      // value can be auto-generated or explicitly set
    });

    freePlan.addApiKey(tenantApiKey);
  }
}

How I usually operationalize this

Create usage plans by tier (free, pro, enterprise)
Create an API key per tenant (metering identity)
Store the key value (or a mapping) securely
Have the authorizer return the correct usageIdentifierKey for the tenant

This gives me a clean SaaS throttle/quota model without making clients manage API keys directly.

Authorization checks beyond JWT validation (critical for tenant safety)

JWT validation is not enough. I also enforce authorization in Lambda based on tenant and role.

Examples:

User can create tickets only in projects belonging to their tenant
User can comment only on tickets in their tenant
Support/admin actions require specific roles/scopes
Cross-tenant resource IDs are rejected even if guessed

Example guard pattern

function requireRole(rolesJson: string | undefined, allowed: string[]) {
  const roles = rolesJson ? JSON.parse(rolesJson) as string[] : [];
  const ok = roles.some((r) => allowed.includes(r));
  if (!ok) {
    const err = new Error("Forbidden");
    (err as any).statusCode = 403;
    throw err;
  }
}

I typically build a small internal library for:

auth context parsing
role checks
tenant-scoped key helpers
common error responses

That reduces drift across Lambda functions.

End-to-end implementation walkthrough

Here is a practical sequence I would follow to build this from scratch.

Step 1: Define your access patterns first

Before touching CloudFormation/CDK, write down:

reads
writes
sorting/filtering requirements
expected high-volume paths

This will drive your PK/SK and GSI decisions.

Step 2: Set up Cognito and define tenant claims

Create a user pool and app client. Decide how tenant context appears in JWTs.

I recommend normalizing these claims:

tenant_id
plan
roles or groups

If your current user profile attributes are inconsistent, fix that first. It will save a lot of pain later.

Step 3: Build the Lambda authorizer

Implement JWT validation and claim normalization.

Outputs should include:

tenantId
userId
rolesJson
plan
usageIdentifierKey (tenant metering key)

Step 4: Create DynamoDB single table + GSIs

Start with only the GSIs you need for your first release.

A good initial setup:

base table (PK, SK)
GSI1 for project+status ticket listing
GSI2 for assignee ticket listing

Step 5: Implement tenant-safe handlers

In every handler:

parse auth context from API Gateway authorizer context
derive keys from auth context
never trust client-supplied tenant IDs for authorization
use conditional expressions when appropriate

Step 6: Add per-tenant usage plans

Create:

usage plans by tier
API keys per tenant
tenant -> metering key mapping

Then return the metering key in the Lambda authorizer.

Step 7: Test with real tokens and tenant boundary tests

I always run tests for:

valid tenant access
invalid token
wrong tenant resource ID
quota exceeded / throttle behavior
large-tenant list/query traffic

Example request/response flow (local testing mindset)

Create ticket request

curl -X POST \
  "https://{apiId}.execute-api.{region}.amazonaws.com/prod/projects/p_001/tickets" \
  -H "Authorization: Bearer <JWT>" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "API returns 500 on retry",
    "priority": "HIGH",
    "assigneeUserId": "u_77"
  }'

Expected response

{
  "ticketId": "tk_6b4d8e3e-....",
  "status": "OPEN",
  "createdAt": "2026-02-25T01:00:00Z"
}

List tickets by status

curl \
  "https://{apiId}.execute-api.{region}.amazonaws.com/prod/projects/p_001/tickets?status=OPEN" \
  -H "Authorization: Bearer <JWT>"

Common mistakes I see in this architecture

1) Trusting `tenantId` from the client

This is the fastest way to create cross-tenant data leaks.

Fix: derive tenant context from validated JWT and enforce server-side.

2) Designing the table like a relational schema

DynamoDB single-table design is access-pattern driven, not entity-table driven.

Fix: start with queries and write paths first.

3) Overusing one tenant root partition key

This can create hot partitions for large tenants.

Fix: use tenant-aware sub-collections and shard high-volume patterns.

4) Adding too many GSIs early

Every GSI adds write amplification and operational cost.

Fix: add GSIs only for clear access patterns.

5) Treating JWT validation as full authorization

A valid token does not mean a valid action.

Fix: layer role checks and tenant-resource checks in Lambda.

6) Ignoring quota/throttling strategy until later

By the time you need it, retrofitting it can be messy.

Fix: establish a per-tenant metering pattern early (usage plans or app-level rate limiting).

When I would extend this design

As the SaaS grows, I often add:

DynamoDB Streams for async workflows (notifications, audit, search indexing)
SQS/EventBridge for background processing
Fine-grained authorization model (RBAC/ABAC) beyond simple groups
Tenant-level feature flags
Audit log table (append-only) for compliance
Support-user safe impersonation with strict audit trails
Silo/bridge routing for enterprise tenants needing dedicated infrastructure

The nice part is that the core pattern (API Gateway + Lambda + DynamoDB + normalized auth context) still holds.

Final thoughts

This stack is powerful because it lets me solve architecture and data modeling together.

The API layer, auth model, and DynamoDB key design are not separate decisions in a multi-tenant SaaS. They are one system:

Auth defines the tenant context
The API propagates and enforces it
The data model encodes it
Throttling/quotas protect the platform from noisy neighbors

If you get those four pieces aligned early, you can move fast without constantly revisiting foundational decisions.

If I were building a new SaaS MVP today on AWS, this would still be one of my first choices.

References

Amazon API Gateway REST API documentation
AWS Lambda developer guide
Amazon DynamoDB developer guide
DynamoDB single-table design patterns (AWS documentation and re:Invent talks)
Amazon Cognito user pools and JWT token documentation
API Gateway usage plans and API keys documentation
aws-jwt-verify library documentation
DynamoDB best practices for partition key design and adaptive capacity
AWS Well-Architected Framework (Serverless Lens)

Step Functions Distributed Map Best Practices for Large-Scale Batch Workloads

Renaldi — Sun, 01 Mar 2026 22:00:00 +0000

When I need to process a very large dataset in parallel on AWS without standing up a whole batch platform, AWS Step Functions Distributed Map is one of the first tools I reach for.

It gives me a clean orchestration layer for large-scale batch jobs while still letting me control concurrency, retries, failure thresholds, and output handling in a way that is production-friendly.

In this post, I will walk through how I approach Distributed Map for large-scale batch workloads, with a focus on practical best practices I would actually use in a real system.

When I choose Distributed Map (and when I do not)

I typically consider Distributed Map when one or more of these are true:

My dataset is too large to comfortably pass around as state input
I would exceed normal inline Map limits or create a huge execution history
I need high concurrency across many items
My input already lives in S3 (CSV, JSON, object lists, etc.)
I want a serverless orchestration layer and clear visibility into the run

I usually do not start with Distributed Map if:

The batch is small and an inline Map or a single Lambda is enough
I need heavy distributed compute semantics (for example, Spark-style transformations)
My processing requires very long-running per-item tasks that are better suited to another service design

The key is to treat Distributed Map as an orchestration primitive for massively parallel item or batch processing, not as a replacement for every data platform.

Architecture at a glance

Figure 1. Reference architecture for large-scale batch processing with Step Functions Distributed Map (S3 input, batched child workflows, ResultWriter, and observability).

Core best practices I use in production

1) Start with a lower `MaxConcurrency` and scale up deliberately

This is the biggest mistake I see: people let the workflow fan out too aggressively before validating downstream capacity.

Distributed Map can run a very high number of child executions in parallel (AWS documents up to 10,000 parallel child workflow executions in distributed mode), which is great, but your downstream systems (Lambda concurrency, databases, APIs, third-party services) are usually the real bottleneck.

What I do instead:

Start with a conservative MaxConcurrency
Load test with representative input
Watch downstream latency, throttling, and error rates
Increase concurrency gradually

I treat MaxConcurrency as a safety valve, not just a performance knob.

2) Use `ItemBatcher` for throughput and cost efficiency

If each item is tiny and the per-item work is short, processing one item per child execution can create unnecessary overhead.

In those cases, I use ItemBatcher to group items into batches and process each batch in one child execution.

Why this helps:

Fewer child executions
Fewer state transitions
Better Lambda efficiency (especially for small records)
Easier downstream write batching

The tradeoff is that my worker logic must handle Items[] input and return a sensible batch-level result.

3) Keep child outputs compact and use S3 for detailed results

A common anti-pattern is returning large payloads from every child execution. That creates pressure on Step Functions payload limits and makes debugging harder.

My default pattern is:

Child workflow returns a compact summary (counts, status, pointers)
Worker writes detailed outputs/errors to S3 (if needed)
Distributed Map uses ResultWriter to export consolidated execution results

This keeps the orchestration layer lightweight while preserving traceability.

4) Define explicit failure tolerance (`ToleratedFailure*`) for noisy datasets

Large batch workloads often contain some bad records. If I expect occasional malformed rows or downstream rejections, failing the entire Map Run on the first bad batch is usually too strict.

I set one of these based on business tolerance:

ToleratedFailurePercentage
ToleratedFailureCount

This lets me absorb known-noisy inputs while still failing fast when quality degrades beyond an acceptable threshold.

Important design point: these thresholds apply to failed/timed-out child workflow executions, so I make sure my child workflow raises failures intentionally when a batch truly cannot be processed.

5) Put retries and error handling inside the child workflow

I prefer retries at the point of failure (inside the ItemProcessor states), not only at the parent level.

For example:

Retry transient Lambda/service errors
Catch and classify non-retryable failures
Return a clear batch summary when possible
Fail the child execution when the batch should count toward failure tolerance

This gives me better control over what counts as a retryable blip versus a real batch failure.

6) Choose child execution type intentionally (`EXPRESS` vs `STANDARD`)

Distributed Map runs inside a Standard parent state machine, but each child workflow can be EXPRESS or STANDARD.

My practical rule of thumb:

Use EXPRESS child workflows for high-throughput, shorter-running item/batch tasks
Use STANDARD child workflows when I need longer-running orchestration or richer execution semantics

I decide this early because it affects cost, runtime behavior, and observability patterns.

7) Design the worker to be idempotent

At scale, retries and reprocessing happen. I assume they will happen.

That means my batch worker should be idempotent, for example by:

Using deterministic item IDs
Recording processed item keys/checkpoints
Using conditional writes (where supported)
Writing outputs with deterministic object keys (or versioned prefixes)

This makes retries safe and reduces cleanup work.

8) Treat quotas and throttling as part of the design, not an afterthought

With large-scale fan-out, service quotas and API throttling become part of normal engineering.

I usually validate:

Step Functions quotas relevant to Distributed Map and Map Runs
StartExecution / API throttling expectations
Lambda concurrency and reserved concurrency
Downstream service write capacity or request limits

This saves a lot of pain later.

Example state machine (Distributed Map with batching and ResultWriter)

Below is a JSONPath-based example that shows a production-style shape I like to start from.

What this example demonstrates:

S3-backed input via ItemReader
ItemBatcher to process multiple records per child execution
Controlled parallelism via MaxConcurrency
Failure tolerance via ToleratedFailurePercentage
ResultWriter to S3
Retries and a clear failure path inside the child workflow

{
  "Comment": "Large-scale batch processing with Distributed Map",
  "StartAt": "RunDistributedBatch",
  "States": {
    "RunDistributedBatch": {
      "Type": "Map",
      "Label": "BatchIngestV1",
      "ItemReader": {
        "ReaderConfig": {
          "InputType": "CSV",
          "CSVHeaderLocation": "FIRST_ROW"
        },
        "Resource": "arn:aws:states:::s3:getObject",
        "Parameters": {
          "Bucket.$": "$.input.bucket",
          "Key.$": "$.input.key"
        }
      },
      "ItemSelector": {
        "jobId.$": "$.jobId",
        "record.$": "$$.Map.Item.Value"
      },
      "ItemBatcher": {
        "MaxItemsPerBatch": 100
      },
      "MaxConcurrency": 300,
      "ToleratedFailurePercentage": 2,
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "EXPRESS"
        },
        "StartAt": "ProcessBatch",
        "States": {
          "ProcessBatch": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "OutputPath": "$.Payload",
            "Parameters": {
              "FunctionName": "arn:aws:lambda:ap-southeast-2:123456789012:function:batch-worker",
              "Payload.$": "$"
            },
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "States.TaskFailed"
                ],
                "IntervalSeconds": 2,
                "BackoffRate": 2.0,
                "MaxAttempts": 3
              }
            ],
            "Catch": [
              {
                "ErrorEquals": ["States.ALL"],
                "ResultPath": "$.error",
                "Next": "MarkBatchFailed"
              }
            ],
            "End": true
          },
          "MarkBatchFailed": {
            "Type": "Fail",
            "Cause": "Batch worker failed after retries"
          }
        }
      },
      "ResultWriter": {
        "WriterConfig": {
          "Transformation": "COMPACT",
          "OutputType": "JSONL"
        },
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {
          "Bucket.$": "$.resultWriter.bucket",
          "Prefix.$": "$.resultWriter.prefix"
        }
      },
      "End": true
    }
  }
}

Notes on this definition (why I like this shape)

I use ItemReader so the parent execution input stays small
I use ItemBatcher to reduce overhead for tiny records
I set MaxConcurrency explicitly so I do not accidentally overwhelm downstream systems
I use a small but non-zero failure tolerance only if the workload can tolerate some bad batches
I use ResultWriter so the Map Run does not need to return a giant in-memory result array

Example batch worker (Python Lambda)

This is a simplified worker pattern I use for batched processing. The main ideas are:

Accept Items[] from ItemBatcher
Process each item safely
Return a compact summary
Fail the batch only when appropriate (so failure tolerance behaves meaningfully)

import json
import hashlib
from typing import Dict, Any, List

def _stable_item_id(item: Dict[str, Any]) -> str:
    raw = json.dumps(item, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

def handler(event: Dict[str, Any], context) -> Dict[str, Any]:
    # When ItemBatcher is used, Step Functions passes batched input as {"Items": [...]}
    items: List[Dict[str, Any]] = event.get("Items", [])
    successes = 0
    failures = 0
    failed_items = []

    for item_wrapper in items:
        # In this example, ItemSelector wrapped the original row into {"jobId": ..., "record": ...}
        record = item_wrapper.get("record", {})
        item_id = _stable_item_id(record)

        try:
            # Example validation (replace with real business logic)
            if not record:
                raise ValueError("Empty record")

            # TODO: process the record (write to DynamoDB/S3/etc.)
            # Make downstream writes idempotent (conditional write / deterministic key)
            successes += 1

        except Exception as exc:
            failures += 1
            failed_items.append({
                "itemId": item_id,
                "error": str(exc)
            })

    total = len(items)
    failure_ratio = (failures / total) if total else 0.0

    # Business decision:
    # fail the child execution only when the batch quality is unacceptable.
    if total == 0 or failure_ratio > 0.20 or successes == 0:
        raise RuntimeError(
            f"Batch failed: successes={successes}, failures={failures}, total={total}"
        )

    return {
        "status": "PARTIAL_SUCCESS" if failures else "SUCCESS",
        "totalItems": total,
        "successes": successes,
        "failures": failures,
        "failedItemSample": failed_items[:10]
    }

Worker implementation tips I strongly recommend

Keep the response small
Write detailed failure artifacts to S3 if you need full diagnostics
Use deterministic keys / conditional writes for idempotency
Separate retryable vs non-retryable exceptions in your code (if possible)
Emit metrics (success count, failure count, latency) from the worker

One retry gotcha I always call out (important)

I am very careful with adding a Retry block on the Distributed Map state itself.

Why? Because AWS documents that if you define retriers on the Distributed Map state, the retry policy applies to all child workflow executions started by that Map state, not just the one that failed. In practice, that can create a lot of duplicate work and surprise costs if you are not expecting it.

My default approach:

Put retries inside the child workflow first (fine-grained retries)
Use Map-level Retry only when I intentionally want to rerun the whole Map state / Map Run behavior
Make sure idempotency is in place before enabling broader retries

Observability and operations checklist (what I watch)

When I productionize this pattern, I monitor the workflow at three levels:

A) Parent state machine

Execution failures/timeouts
End-to-end duration
Throttling indicators (if any)

B) Map Run / child workflow level

Child execution success/failure trends
Failure rate relative to my tolerated threshold
Backlog behavior when concurrency is constrained

C) Worker and downstream systems

Lambda concurrency and duration
Error rate and retries
Database/API throttling
Write latency and saturation

I also make sure the team has a runbook for:

What to do when failure tolerance is exceeded
How to inspect Map Run details and failing child executions
When and how to redrive after fixing data or IAM issues

Common mistakes I try to avoid

1) Leaving `MaxConcurrency` effectively unlimited

This can look great in testing and then melt a downstream dependency in production.

2) Returning huge outputs from child workflows

This adds payload pressure and makes the state machine harder to operate.

3) Using batch sizes without thinking about item size

Small records and large records behave very differently. Batch by both count and (if needed) input bytes based on your real data.

4) Treating all failures the same

Malformed data, throttling, and transient AWS service errors should not all be handled identically.

5) Skipping idempotency

At scale, retries are normal. Idempotency is what keeps retries safe.

A practical rollout strategy I use

If I am introducing Distributed Map into an existing system, I usually do it in phases:

Prototype with a small sample dataset
Enable explicit MaxConcurrency
Add ItemBatcher
Add ResultWriter and compact outputs
Load test with realistic record sizes
Tune retries, batch size, and failure thresholds
Document redrive/runbook steps

This sequence helps me avoid premature complexity while still landing on a production-grade design.

Final thoughts

Distributed Map is one of those features that becomes incredibly powerful once I stop thinking of it as “just a bigger map loop” and start treating it as a batch orchestration framework with explicit controls.

The best results usually come from combining:

S3-backed input
Deliberate concurrency limits
Batched child processing
Compact outputs + S3 result export
Clear failure semantics
Good observability and runbooks

That combination gives me something that scales well and is still operationally sane.

References

How to Monitor Event Delivery in Amazon EventBridge

Renaldi — Wed, 25 Feb 2026 05:06:19 +0000

When I first started using Amazon EventBridge more heavily, I realized something important pretty quickly: it is very easy to build event-driven flows, but much harder to know when delivery is degrading before things break.

This post is specifically about best practices for monitoring event delivery in EventBridge (not just target-side application monitoring). AWS actually has strong guidance here, and the key is to monitor a combination of retry, success, failure, DLQ, and latency metrics together instead of looking at a single number in isolation.

I’ll walk through:

what to monitor
what I alert on (and why)
an architecture pattern that works well
practical alarm design so you do not get spammed all day

Why this matters

AWS recommends monitoring EventBridge delivery behavior because underperforming or undersized targets can cause excessive retries, delivery delays, and permanent delivery failures. They also explicitly recommend combining multiple metrics and setting alarms/dashboards for early detection.

Also, by default, EventBridge retries failed target delivery for up to 24 hours and up to 185 times (with exponential backoff and jitter). If retries are exhausted, the event is dropped unless you configured a DLQ.

That default behavior is great for resilience, but it also means a problem can quietly turn into a backlog / latency issue if you are not watching the right signals.

The architecture I use for monitoring EventBridge delivery

Here is the monitoring architecture pattern I like for production workloads.

Best-practice notes in this architecture

I prefer a DLQ per rule target (or at least per critical target path), because AWS recommends configuring a dead-letter queue for each rule target to avoid losing undelivered events.
I keep CloudWatch alarms on EventBridge metrics and separate alarms on the target service (for example Lambda errors, API 5xx, Step Functions failures). EventBridge tells me about delivery behavior; target metrics tell me what is happening after delivery.
I treat the DLQ as a diagnostics feed, not just a parking lot.

The core EventBridge metrics I monitor (and why)

AWS’s EventBridge best-practices page specifically calls out a small set of metrics for delivery monitoring, and I use those as the foundation.

1) `InvocationAttempts`

This is the total number of times EventBridge attempted to invoke a target, including retries.

Why I care:

It gives me the denominator for delivery health.
It helps me see if retries are inflating traffic to a struggling target.

2) `SuccessfulInvocationAttempts`

This is the number of invocation attempts where EventBridge successfully delivered the event to the target.

Why I care:

It is the numerator for a success-rate metric.
On its own, it can look healthy even when retries are growing, so I always compare it with InvocationAttempts.

3) `RetryInvocationAttempts`

This is the number of invocation attempts that were retries. AWS explicitly says an increase here may be an early indication of an undersized target.

Why I care:

This is usually my earliest warning signal.
I can catch target stress before it becomes permanent failure.

4) `FailedInvocations`

This is the number of invocations that failed permanently (not retried, or not recovered after retry), with some exclusions noted in the metric definition.

Why I care:

This is a direct “something is broken” signal.
If this rises and there is no DLQ, I am at risk of event loss after retries are exhausted.

5) `InvocationsSentToDlq`

This is the number of invocations moved to a dead-letter queue.

Why I care:

This tells me failed deliveries are now becoming operational work.
Even if the system is “surviving,” this still means customers or downstream processes may be impacted.

6) `InvocationsFailedToBeSentToDlq`

This is the nightmare metric: EventBridge tried to move failed invocations to the DLQ but could not (for example due to permissions, unavailable resource, or size issues). AWS documents this metric and notes it is emitted only when non-zero.

Why I care:

This is a critical alarm for me.
It means my failure safety net is not working.

7) `IngestionToInvocationSuccessLatency`

AWS recommends this latency metric for detecting event delivery delays. It measures end-to-end latency from ingestion to successful delivery and can surface delays from retries, timeouts, or slow target responses. It also includes the target’s time to respond successfully.

Why I care:

This catches degraded delivery even when there are no obvious hard failures.
It is great for detecting “everything is technically working, but much slower than usual.”

The most useful derived metric: `SuccessfulInvocationRate`

AWS explicitly recommends combining metrics and even gives a metric math example:

SuccessfulInvocationRate = SuccessfulInvocationAttempts / InvocationAttempts
I strongly agree with this. In practice, I always graph and alert on a success rate, not just raw counts.

CloudWatch metric math expression (example)

SuccessfulInvocationRate = SuccessfulInvocationAttempts / InvocationAttempts

My practical implementation tip

I only evaluate this alarm when there is meaningful traffic. Otherwise, low-volume rules can produce noisy alerts (for example, 1 failed attempt on a rarely used rule).

A common pattern is:

a volume guard (enough attempts in the window)
plus a success-rate threshold

What I alert on (my starting best-practice alarm set)

Below is a practical alert set I use as a starting point. The exact thresholds should be tuned to your workload and business SLOs.

I treat these as starting thresholds, not universal truths.

Alert 1 (Critical): `InvocationsFailedToBeSentToDlq > 0`

Severity: P1 / Critical

Why: The DLQ safety net itself is failing. AWS notes this can happen due to permission issues, unavailable resources, or size limits.

What I do immediately

Check SQS queue existence and region
Check SQS resource policy / permissions
Check encryption / access configuration if applicable
Check message size / payload patterns

Alert 2 (High): `FailedInvocations > 0` on critical rules

Severity: P1 or P2 (depends on rule criticality)

Why: Permanent delivery failure is already happening. AWS documents FailedInvocations as permanently failed invocations.

Best practice

Scope alarms per rule for critical paths
Route alerts to the owning team, not one giant shared channel

Alert 3 (High): `InvocationsSentToDlq > 0`

Severity: P2 (or P1 for critical business flows)

Why: Failures are being captured safely, but delivery is still failing and backlog is building. AWS best-practice guidance specifically calls out watching FailedInvocations and InvocationsSentToDlq.

What I do

Triage DLQ message attributes (ERROR_CODE, ERROR_MESSAGE, RETRY_ATTEMPTS, etc.) to identify the root cause faster. EventBridge includes these attributes in DLQ messages.

Alert 4 (Warning): `RetryInvocationAttempts` rising above baseline

Severity: Warning / P3

Why: AWS says an increase may be an early indication of an undersized target.

This is one of my favorite early-warning alerts.

How I tune it

For steady traffic: alert on absolute count per period
For bursty traffic: alert on retry rate (retry attempts / invocation attempts)
Use a longer evaluation window to avoid false positives during short spikes

Alert 5 (Warning/High): `SuccessfulInvocationRate` below threshold

Severity: Warning or P2

Why: This gives a single health signal that accounts for retries and success together (exactly what AWS recommends via metric math).

Example starting thresholds

Warning: < 99.9% for 10 minutes (high-volume, sensitive systems)
High: < 99% for 5–10 minutes
Critical: < 95% for 5 minutes on critical paths

Again, tune these to your business tolerance.

Alert 6 (Warning): `IngestionToInvocationSuccessLatency` p95/p99 above baseline

Severity: Warning or P2

Why: AWS recommends using this metric to detect delays, and it surfaces retry effects and slow target responses.

My recommendation

Alert on p95 or p99 rather than average
Use a baseline per rule (or per target type), because latency expectations differ a lot

Alert 7 (Warning): `ThrottledRules > 0`

Severity: Warning / P2 if persistent

Why: AWS documents that throttled rule executions may delay invocations.

This is especially useful when traffic grows and quota pressure starts to show up.

My dashboard layout (simple but effective)

I try not to overcomplicate the dashboard. I keep one dashboard section per critical rule / target path, and I always put these together:

Panel group A: Delivery health

InvocationAttempts
SuccessfulInvocationAttempts
RetryInvocationAttempts
FailedInvocations
InvocationsSentToDlq

Panel group B: Delivery quality

SuccessfulInvocationRate (metric math)
IngestionToInvocationSuccessLatency (p50, p95, p99)

Panel group C: Protection / risk

InvocationsFailedToBeSentToDlq
ThrottledRules

This matches AWS’s “combine multiple metrics for a holistic overview” guidance, and in practice it makes troubleshooting much faster.

Practical runbook: how I interpret the pattern combinations

This is the part that saves the most time during incidents.

Pattern 1: Retries rising, success rate still high

Likely meaning: target is stressed but still mostly coping

Metrics pattern:

RetryInvocationAttempts up
SuccessfulInvocationRate mostly stable
latency rising (IngestionToInvocationSuccessLatency)

What I check

target concurrency / scaling
downstream dependency latency
throttling on target service

Pattern 2: Retries rising, success rate dropping, DLQ starts filling

Likely meaning: target is underprovisioned or misconfigured

Metrics pattern:

RetryInvocationAttempts up
SuccessfulInvocationRate down
InvocationsSentToDlq > 0

What I check

target capacity / rate limits
permission changes
endpoint availability / DNS / networking
recent deployments

Pattern 3: Failed invocations and DLQ-send failures

Likely meaning: primary path failing and safety net misconfigured

Metrics pattern:

FailedInvocations > 0
InvocationsFailedToBeSentToDlq > 0

What I do

page immediately
validate DLQ queue, region, permissions, and policy
verify EventBridge can SendMessage to the SQS DLQ

AWS’s DLQ guidance includes permission requirements and examples, and this is the first place I look when DLQ delivery fails.

Common mistakes I see (and try to avoid)

1) Only alerting on `FailedInvocations`

This catches issues late. By the time permanent failures show up, you may already have retries, delays, and customer impact.

Better: also alert on retries and latency.

2) No DLQ configured

AWS explicitly recommends configuring a DLQ for each rule target to avoid losing undelivered events.

3) No alarm on `InvocationsFailedToBeSentToDlq`

This is the “we thought we were safe, but we are not” blind spot.

4) One global alarm for all rules

This creates noisy alerts and slow triage.

Better: alarm per critical rule / target path, and map ownership.

5) Using static latency thresholds with no baseline thinking

Different targets behave differently. A Lambda target and an API destination will not share the same normal latency profile.

A practical starting checklist (copy this into your ops notes)

[ ] Configure DLQ for each critical EventBridge rule target
[ ] Alarm on InvocationsFailedToBeSentToDlq > 0 (critical)
[ ] Alarm on FailedInvocations > 0 for critical rules
[ ] Alarm on InvocationsSentToDlq > 0 for critical rules
[ ] Track RetryInvocationAttempts trend (early warning)
[ ] Create SuccessfulInvocationRate metric math and alert on sustained dips
[ ] Track IngestionToInvocationSuccessLatency p95/p99 and alert on sustained increase
[ ] Add ThrottledRules alert if traffic growth is expected
[ ] Build per-rule dashboards for critical flows
[ ] Document a DLQ triage + replay runbook

Final thoughts

If I had to summarize the best practice in one sentence, it would be this:

Do not monitor EventBridge delivery with a single failure metric. Monitor retries, success rate, DLQ activity, and delivery latency together.

That combination gives me an early warning signal, a confirmed-failure signal, and a customer-impact signal, which is exactly what I want in production.

References

AWS EventBridge monitoring best practices for event delivery (recommended metrics, success-rate metric math, retries/DLQ guidance). (AWS Documentation)
AWS EventBridge monitoring metrics reference (AWS/Events metrics and definitions). (AWS Documentation)
AWS EventBridge retry behavior (default retry duration and count). (AWS Documentation)
AWS EventBridge DLQ guidance (DLQ behavior, limitations, permissions, message attributes). (AWS Documentation)

Efficiently Handling High Concurrency with AWS Lambda SnapStart: A Step-by-Step Guide

Renaldi — Mon, 18 Nov 2024 01:06:10 +0000

Introduction

Modern online services frequently face unexpected surges in user activity. It's essential that your system can process multiple simultaneous requests efficiently to keep users satisfied and engaged. To address performance challenges in serverless environments, AWS offers Lambda SnapStart. This enhancement reduces function initialization time, helping maintain responsiveness when demand increases. We'll explore a real-world example demonstrating when this capability becomes valuable, and provide detailed instructions for setting it up in your own environment.

Scenario Overview

Consider operating a web-based event admission system that sells access to live performances and gatherings. When highly anticipated shows become available for purchase, your platform experiences a sudden influx of concurrent visitors. To ensure smooth transaction processing during these peak periods, your system infrastructure must expand rapidly while maintaining quick response times for each customer interaction. By implementing Amazon's Lambda SnapStart functionality, you can minimize initialization delays in your cloud functions, enabling better performance during these intense usage periods.

What is AWS Lambda SnapStart?

AWS' Lambda SnapStart enhances function response times by performing pre-initialization and creating a cached memory state that can be reused for subsequent executions. This approach captures a ready-to-use version of your code, allowing new instances to launch more quickly. By eliminating the standard initialization delay typically experienced during first-time function calls, this capability particularly benefits applications that need to handle many simultaneous user requests.

Why Use Lambda SnapStart in This Scenario?

For an event ticketing service, speed is absolutely critical. When customers attempt to secure their spots, even slight delays can frustrate buyers and potentially cost you business. Implementing Amazon's SnapStart technology for serverless functions helps ensure rapid processing times, maintaining system responsiveness even during peak demand. This approach enables consistent, swift service delivery regardless of how many people are simultaneously trying to purchase tickets.

Step-by-Step Implementation Guide

Follow these steps to implement AWS Lambda with SnapStart for your ticketing platform.

Step 1: Create a New Lambda Function

On the AWS Lambda page, click on the "Create function" button.
Under "Create function", choose "Author from scratch".
Fill in the following details:
Function name: TicketingProcessor
Runtime: Select "Java 17"

Note: Lambda SnapStart currently supports Java runtimes. We'll use Java 17 for this example.

Under Permissions, expand the "Change default execution role" section.
Select "Create a new role with basic Lambda permissions".
Click on "Create function" at the bottom of the page.

Step 2: Write the Lambda Function Code

After the function is created, you'll be taken to the function's configuration page.
Scroll down to the "Code source" section.
Under "Code source", click on the file named LambdaFunction.java to open the code editor.
Replace the existing code with the following Java code:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import java.util.HashMap;
import java.util.Map;

public class TicketingProcessor implements RequestHandler<Map<String, String>, Map<String, String>> {

    // Simulate heavy initialization logic
    static {
        try {
            // Simulate time-consuming startup tasks
            Thread.sleep(5000); // 5-second delay to simulate cold start
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    @Override
    public Map<String, String> handleRequest(Map<String, String> event, Context context) {
        Map<String, String> response = new HashMap<>();
        response.put("message", "Ticket processed successfully!");
        return response;
    }
}

This code simulates a Lambda function with heavy initialization (the static block that sleeps for 5 seconds). SnapStart will help us bypass this delay in subsequent invocations.

Click on "Deploy" at the top right corner to save and deploy the code.

Step 3: Configure SnapStart for the Lambda Function

In the left-hand menu, under "Versioning", click on "Versions".
Click on "Publish new version" at the top right.
In the "Publish new version" dialog, for Version description, enter Initial version with SnapStart.
Under "SnapStart", select "Enable SnapStart".
Click on "Publish".

Note: If you don't see the SnapStart option, ensure that you are using a supported runtime (Java 11 or Java 17). Enabling SnapStart during the publishing of a new version tells AWS to take a snapshot after initialization, which will be used for faster startups.

Step 4: Test the Lambda Function

Navigate back to your function by clicking on "Code" in the left-hand menu.
Click on "Test" at the top right corner.
In the "Configure test event" dialog:
Select "Create new test event".
Event template: Choose "Hello World".
Event name: Enter TestEvent.
Leave the default JSON as is:

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

Click on "Create". Click on "Test" again to invoke the function. Check the "Execution result" section below. You should see a response
similar to:

{
  "message": "Ticket processed successfully!"
}

Note the "Duration" in the "Summary" section. It should show a reduced execution time due to SnapStart on subsequent invocations.

Step 5: Simulate High Concurrency
To test the function under high concurrency, we'll invoke it multiple times in quick succession.

Option 1: Use AWS Lambda Console's "Test" Feature Repeatedly
You can invoke the function multiple times manually to observe the performance improvement.
Option 2: Use AWS CLI to Invoke the Function Concurrently

Install AWS CLI: If you haven't installed AWS CLI, follow the installation guide here.
Configure AWS CLI: Run aws configure in your terminal and enter your AWS credentials.
In the AWS Lambda console, on your function's page, note the "ARN" at the top. It looks like arn:aws:lambda:region:account-id:function:TicketingProcessor.
Create a Script to Invoke the Function Concurrently. Create a file named invoke_lambda.sh with the following content:

#!/bin/bash

FUNCTION_NAME="TicketingProcessor"
REGION="your-region" # e.g., us-east-1

for i in {1..100}
do
  aws lambda invoke --function-name $FUNCTION_NAME --region $REGION --payload '{}' response_$i.json &
done

wait

Replace your-region with your AWS region, such as us-west-2.

Step 6: Provide the Relevant Permissions and Test

Make the Script Executable by running chmod +x invoke_lambda.sh in your terminal.
Run the Script by executing ./invoke_lambda.sh to invoke the Lambda function 100 times concurrently.
Check the Results.
The responses will be saved in files named response_1.json, response_2.json, ..., response_100.json.
You can also check the "Monitoring" tab in the AWS Lambda console to see the invocation metrics.

Step 7: Review Performance Metrics

In the AWS Lambda console, navigate to your function's page.
Click on the "Monitoring" tab.
Observe the metrics:
Invocations: Number of times your function was invoked.
Duration: Time taken for each invocation.
Concurrency: Number of concurrent executions.
Errors: Any errors that occurred during execution.
You should notice that the "Duration" metric shows reduced cold start times due to SnapStart, especially after the initial invocation.

Final Notes:

Ensure that your AWS Identity and Access Management (IAM) role has the necessary permissions to execute Lambda functions and access AWS services.
Be aware that invoking Lambda functions may incur costs. Refer to the AWS Lambda Pricing page for more details.

Conclusion
These implementation steps have shown you how to leverage Amazon's SnapStart capability to enhance your serverless application's responsiveness during peak loads. With this optimization in place, your event ticketing system can now better manage unexpected surges of visitor activity, maintaining quick response times and keeping your customers satisfied throughout their purchasing journey.

Additional Resources

Implementation of Melodistiq: Generating Lyrics and Melodies with AI

Renaldi — Thu, 25 Apr 2024 01:00:41 +0000

Introduction

Hello, fellow cloud enthusiasts and builders!

In today's blog post, we will dive into an intriguing project that combines artificial intelligence with music creation. This post will detail the implementation of a Python code designed to generate music lyrics and melodies using AI technologies. This follows the successful presentation of the "Unlocking Musical Creativity with AI: Generating Lyrics and Melodies" talk.

The Scenario

The goal of this project is to automate the creation of music lyrics and melodies. This can be particularly useful for musicians seeking inspiration or developers exploring the intersection of AI and creative arts. The process involves generating lyrics based on existing song data, creating a melody that fits the generated lyrics, and presenting the final composition in both audio and written formats.

Libraries and Tools Used

To accomplish this task, we utilize several Python libraries, each serving a distinct purpose:

Pandas (pandas): A powerful data manipulation library used here to handle and preprocess lyrical data.
Natural Language Toolkit (NLTK): This library is used for text processing to analyze, manipulate, and generate text data.
OpenAI's GPT-3.5: Leveraged to enhance the quality of the generated lyrics.
Mingus: An advanced music theory and notation package used to handle music data and generate melodies.
FastText: A library developed by Facebook for efficient learning of word representations and sentence classification.
FPDF: A library to generate PDF files, useful for presenting the lyrics in a readable format.

Let's walk through implementing our code now.

Implementation

Setting Up and Downloading Models
First, we load necessary models and corpora:

import fasttext
import fasttext.util
import nltk
nltk.download('cmudict')
nltk.download('punkt')
from nltk.corpus import cmudict
d = cmudict.dict()

fasttext.util.download_model('en', if_exists='ignore')
ft = fasttext.load_model('cc.en.300.bin')

Here, fastText and NLTK's CMU Pronouncing Dictionary are initialized. fastText is used later to find words similar to those not found in the CMU dictionary.

Data Collection
Data is loaded and preprocessed from a CSV file containing song lyrics:

import pandas as pd
df = pd.read_csv('EdSheeran.csv')
df = df.dropna()
lyrics = df['Lyric'].str.replace('\n', ' ').str.replace('\r', ' ').tolist()

This section reads a CSV file, cleans the data by removing missing values, and formats the lyrics into a list.

Data Preprocessing
The lyrics are tokenized and cleaned:

import re
from nltk.tokenize import word_tokenize
words = [word_tokenize(re.sub(r'\W+', ' ', lyric).lower()) for lyric in lyrics]
words = [word for sublist in words for word in sublist]

Here, special characters are removed, and the text is converted to lowercase to standardize the data.

N-gram Model and Lyrics Generation
An N-gram model is used to generate new lyrics:

N-gram Model and Lyrics Generation

An N-gram model is used to generate new lyrics:

from nltk.util import ngrams
from nltk.probability import FreqDist
import random

n_values = [2, 5, 7]
generated_lyrics = ""

for n in n_values:
    ngrams_list = list(ngrams(words, n, pad_left=True, pad_right=True))
    freq_dist = FreqDist(ngrams_list)

    def generate_lyrics(starting_ngram, freq_dist, num_words):
        generated_words = list(starting_ngram)
        for _ in range(num_words):
            next_word_candidates = [ngram[-1] for ngram in freq_dist.keys() if ngram[:n-1] == tuple(generated_words[-(n-1):])]
            if next_word_candidates:
                next_word = random.choice(next_word_candidates)
                generated_words.append(next_word)
            else:
                break
        return ' '.join(generated_words).replace(' ,', ',').replace(' .', '.').replace(' ;', ';')

    starting_ngram = random.choice(list(freq_dist.keys()))
    generated_lyrics += generate_lyrics(starting_ngram, freq_dist, 200)

This segment builds N-grams from the cleaned words and creates a frequency distribution to model the probabilities of word sequences. New lyrics are generated based on these probabilities.

Enhancing Lyrics with GPT-3.5
We then look to enhance our lyrics with the help of GPT-3.5.

import openai
import os
openai.api_key = os.getenv('openai_api_key')
conversations = {}
session_id=0;
conversations[session_id] = []

conversations[session_id].append({"role": "system", "content": "You are a helpful assistant who will transform the lyrics below into a song."})
conversations[session_id].append({"role": "user", "content": generated_lyrics})

response = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=conversations[session_id],
  max_tokens=200
)

gpt_lyrics = response.choices[0]["message"]["content"].strip()

Here, the OpenAI GPT-3.5 API is used to refine and enhance the generated lyrics, adding a layer of complexity and polish that might be lacking from the simple N-gram model. The environment variable openai_api_key is used to authenticate the API request.

Generating and Exporting the Melody
The code then generates a melody using the Mingus library based on the stress patterns of the lyrics:

from mingus.containers import Note, Bar, Track
from mingus.midi import midi_file_out
import mingus.extra.lilypond as lilypond

def generate_melody(lyrics):
    # Load the CMU Pronouncing Dictionary
    stress_pattern = []

    tokens = nltk.word_tokenize(lyrics)

    # Fix contractions
    fixed_tokens = [contractions.fix(token) for token in tokens]

    # Function to get the stress pattern of a word
    def get_stress(word):
        word = word.lower()
        phones = pronouncing.phones_for_word(word)
        if phones:
            stress_pattern = [int(s) for s in pronouncing.stresses(phones[0])]
            return [stress_pattern]
        else:
            # handle contractions
            if "'" in word:
                parts = word.split("'")
                stress_pattern = []
                for part in parts:
                    stress_pattern += get_stress(part)
                return stress_pattern
            # handle hyphenated words
            elif '-' in word:
                parts = word.split('-')
                stress_pattern = []
                for part in parts:
                    stress_pattern += get_stress(part)
                return stress_pattern
            else:
                print(f'Word not found in dictionary: {word}')
                # Find a similar word in the dictionary and use its stress pattern
                similar_word = find_similar_word(word)
                if similar_word:
                    return get_stress(similar_word)
                else:
                    # Use default pattern if no similar word is found
                    return [[0, 1, 2]]

    # Get the stress pattern of the lyrics
    for word in fixed_tokens:
        # remove punctuation
        word = re.sub(r'[^\w\s]', '', word)
        stress_pattern += get_stress(word)

    # Flatten the stress_pattern list
    stress_pattern = [item for sublist in stress_pattern for item in sublist]

    print(lyrics)
    print(tokens)
    print(["Here are the stress patterns:"] + stress_pattern)

    # Generate a melody based on the stress pattern
    track = Track()
    b = Bar()
    b.set_meter((4, 4))
    beats_in_current_bar = 0
    for stress in stress_pattern:
        if stress == 0:
            note = Note('C', 4)
        elif stress == 1:
            note = Note('E', 4)
        elif stress == 2:
            note = Note('G', 4)
        b + note
        beats_in_current_bar += 1
        if beats_in_current_bar == 4:
            track.add_bar(b)
            b = Bar()
            b.set_meter((4, 4))
            beats_in_current_bar = 0
    track.add_bar(b)


    return track

Here, we define a function generate_melody that takes a string of lyrics as input and generates a melody based on the phonetic stress pattern of the words. The function uses the nltk library to tokenize the lyrics, handles contractions, and determines the stress pattern of each word using the CMU Pronouncing Dictionary. We account for special cases like contractions, hyphenated words, and words not found in the dictionary, attempting to find similar words or applying a default stress pattern when necessary.

After extracting and flattening the stress pattern of the entire lyrics, the function uses this pattern to create a melody where different stress levels are mapped to specific musical notes (C, E, G) in a 4/4 time signature, adding these notes to a musical track using the mingus library. Each stress level in the pattern corresponds to a different note, and the function organizes these notes into bars, with each bar containing up to four beats. The track, comprising a series of bars filled with notes based on the lyrical stress pattern, is returned at the end of the function. This allows for the conversion of lyrical content into a basic musical representation, integrating elements of natural language processing and music composition.

We can then look into creating the MIDI file and sheet music:

formatted_lyrics = format_lyrics(gpt_lyrics)
formatted_lyrics_with_newlines = add_newlines(formatted_lyrics)
print("Here are the lyrics:" + formatted_lyrics_with_newlines)

melody = generate_melody(formatted_lyrics_with_newlines)

lilypond_string = lilypond.from_Track(melody)
with open('melody.ly', 'w') as f:
    f.write(lilypond_string)

print(melody)

As can be seen, we output the lyrics and MIDI.

Outputting Lyrics as PDF
Finally, we generate a PDF document containing the formatted lyrics, useful for singers or for archival purposes.

from fpdf import FPDF

pdf = FPDF()
pdf.add_page()
pdf.set_font('Arial', 'B', 16)
title = formatted_lyrics_with_newlines.split('\n')[0]
pdf.cell(0, 10, title, 0, 1, 'C')
pdf.set_font('Arial', '', 12)
pdf.multi_cell(0, 10, formatted_lyrics_with_newlines)
pdf.output('lyrics.pdf')

Now, we have our fully functioning Python code for making music! Now, we can decide where to deploy this. Leveraging the capability of AWS, I am going to deploy it on an EC2.

Setting Up the EC2 Instance

Go to the EC2 dashboard in AWS Management Console.
Click on "Launch Instance".
Choose an Amazon Machine Image (AMI), such as Amazon Linux 2 AMI or Ubuntu Server.
Select an instance type (e.g., t2.micro for testing purposes).
Configure instance details as required.
Add storage if the default isn’t enough.
Configure Security Group to allow SSH (port 22) and any other necessary ports (e.g., port 80 for web server).
Review and launch the instance by selecting or creating a new key pair.

Now that we have the EC2 instance ready, we can access it.

Accessing the EC2 Instance
Connect to your instance using SSH. For Windows, you can use PuTTY or any SSH client:
ssh -i /path/to/your-key.pem ec2-user@your-instance-ip

Environment Setup
Install Python and other necessary tools.

sudo yum update -y
sudo yum install python3 git -y

Now move the code over to EC2 with SCP or SFTP and install the relevant Python libraries. You can also clone the repo provided at the end of this post.
pip3 install pandas nltk openai pronouncing re fpdf fasttext

You also will need to install lilypond for usage with Yum.
sudo yum install lilypond -y

Set environment variables (for example, for the OpenAI API key):
export openai_api_key='your-api-key'

With the environment now set up, we can run and automate the script.

Running and Automating the Script
Run the script with the command below.
python3 main.py

To ensure the script runs automatically at system boot or keeps running after disconnections, we will use nohup to run the script in the background.
nohup python3 your_script.py &

Set up a systemd service if you want it to start at boot and restart on failures.

We can then monitor the script execution through checking the logs.
tail -f nohup.out

Optionally, configure CloudWatch to monitor the EC2 performance or to set up more advanced logging and alerting.

And with that, we've set up our deployed solution! It takes a bit of learning to wrap your head around the NLP and music theory bits, but it really is quite a straightforward approach to getting started on making music with AI.

Conclusion
This project showcases the power of AI in creative processes like songwriting. By integrating various technologies and libraries, we've created a system that not only generates lyrics but also composes a melody, presenting it in both audio and visual forms. This illustrates the potential for AI to assist in artistic expressions, providing tools that can inspire and enhance the creative capabilities of its users. Whether you're a developer, a musician, or an enthusiast in the realms of AI or music, this project offers fascinating insights and possibilities.

The repo to the project can be accessed at: https://github.com/renaldig/melodistiq-music-generator