AWS Lambda for ETL: Event-Driven Data Pipelines

aws lambda etl is the pattern that turns your data pipeline from a clock-driven batch job that wakes up every hour into an event-driven system that reacts the instant a file lands, a message arrives, or a record is streamed — and it is the single serverless building block that most cleanly maps "something changed upstream" to "run this transform now." A raw CSV dropped into a landing bucket, a JSON payload pushed onto a queue, a clickstream record flowing through a shard: each of these is an event, and each event can invoke a small stateless function that reads the payload, transforms it, and writes it downstream to a warehouse, a curated bucket, or another queue — without a single server you have to patch, scale, or keep warm. The whole appeal is that you stop paying for idle compute and start paying per invocation, while the platform handles scaling from zero to thousands of parallel executions on its own.

But the appeal hides a set of hard edges that decide whether the design survives contact with production. Every invocation is stateless and time-boxed to fifteen minutes; delivery is at-least-once, so the same event can arrive twice; a sudden burst can spin up so many concurrent executions that it stampedes a fragile downstream database; the first invocation after a quiet period pays a cold start penalty; and a poison-pill record can wedge an entire stream partition until you route it to a dead-letter queue. This guide is the senior-data-engineering walkthrough of the trade-offs that actually come up in design reviews and interviews — when serverless functions fit an ETL job and when the fifteen-minute wall sends you to a different tool, how the event source mapping polls SQS and Kinesis while S3 triggers push, how concurrency, memory and cold starts interact, how retries, DLQ, and idempotency keep an at-least-once world correct, and how fan-out and a Step Functions handoff let you scale past the limits. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse the trigger and streaming muscle on the event-processing practice library →, and sharpen the transform axis with the data-processing practice library →.

---

On this page

• When Lambda fits ETL (and when it doesn't)
• Event sources & triggers — S3 / SQS / Kinesis
• Concurrency, memory & cold starts
• Error handling — DLQ, retries, idempotency
• Patterns & limits — fan-out, Step Functions handoff
• Cheat sheet — AWS Lambda ETL recipes
• Frequently asked questions
• Practice on PipeCode

---

1. When Lambda fits ETL (and when it doesn't)

Event-driven, stateless, fifteen-minute compute — the fit is glue and transform, not heavy batch

The one-sentence invariant: AWS Lambda is event-driven, stateless compute that runs for at most fifteen minutes per invocation and scales horizontally by running one sandbox per concurrent event — which makes it the right engine for lightweight, per-event ETL (transform one file, enrich one message, fan work out) and the wrong engine for anything that needs long-running state, a large in-memory shuffle, or a single job that must chew through terabytes end to end. The fit question is not "can Lambda do ETL" — it obviously can — but "does this job's duration, per-event data volume, and concurrency shape stay inside Lambda's envelope, and does the cost curve beat Glue or EMR at your event rate." Get that judgment right and you get a pipeline that scales from zero to thousands of parallel transforms with no cluster to babysit; get it wrong and you either hit the fifteen-minute wall mid-file or pay more than a provisioned cluster would have cost.

The axes that matter.

• Job duration. A single Lambda invocation has a hard fifteen-minute ceiling. If one unit of work (one file, one batch, one shard read) cannot reliably finish in well under fifteen minutes with headroom, Lambda is the wrong home for that unit — you either shrink the unit (smaller files, smaller batches) or hand the job to Step Functions / Glue.
• Data volume per event. Lambda gives you up to 10 GB of memory and up to 10 GB of ephemeral /tmp. A per-event transform that streams a modest file is a great fit; a job that must load a 50 GB file fully into memory is not. The trick is to process per object or per record batch, never "the whole dataset."
• Concurrency shape. Lambda scales by concurrency — one sandbox per in-flight event. Spiky, bursty, event-triggered workloads are the sweet spot. A steady, always-on, high-throughput stream can still work but starts to look like a cluster you should have provisioned.
• Statelessness. Every invocation starts fresh. There is no durable local state between invocations you can rely on (the sandbox may be reused, but you must not depend on it). Aggregations that need cross-event state must externalise it to DynamoDB, Redis, or a warehouse.
• Cost curve. Lambda bills per millisecond × memory. At low-to-medium event rates it is dramatically cheaper than a warm cluster because you pay nothing when idle. Past a crossover rate, an always-busy Lambda fleet costs more than a right-sized Glue/EMR job. Every senior design names the crossover.

The 2026 reality — Lambda for the edges, big engines for the middle.

• Lambda wins the glue. The canonical fit is the trigger-and-transform step: an object lands in S3, Lambda validates/converts/enriches it and writes a curated copy; a message hits SQS, Lambda upserts it into a warehouse; a Kinesis record streams in, Lambda routes it. These are short, per-event, embarrassingly parallel jobs.
• Lambda wins fan-out and orchestration glue. Splitting a manifest into thousands of parallel tasks, invoking one worker per partition, reacting to a Step Functions state — Lambda is the worker and the glue between managed services.
• Glue / EMR / Spark win the heavy batch. A single job that joins two billion-row tables, does a wide shuffle, or must run for an hour belongs on a distributed engine. Forcing it into Lambda means artificial chunking, cross-invocation state, and a fifteen-minute Tetris game you will lose.
• Step Functions wins the long / multi-step workflow. Anything that needs to run longer than fifteen minutes, retry per step with backoff, branch on results, or coordinate dozens of Lambdas is a Step Functions state machine that uses Lambda for the short steps.

What interviewers listen for.

• Do you name the fifteen-minute limit unprompted when asked "would you use Lambda here?" — required answer.
• Do you say "delivery is at-least-once, so the handler must be idempotent" before being asked about duplicates? — senior signal.
• Do you frame the fit as duration × per-event volume × concurrency × cost, not "serverless is always cheaper"? — senior signal.
• Do you push back on "just Lambda the whole 40 GB nightly join" with "that's a Glue/EMR job; Lambda does the trigger and the fan-out"? — senior signal.
• Do you describe Lambda as "one stateless sandbox per event" rather than "a server that's always on"? — required answer.

Worked example — the fit / no-fit decision table

Detailed explanation. The single most useful artifact for a serverless-ETL design discussion is a memorised fit table: for a candidate job, score it on duration, per-event volume, concurrency, and state, then read off "Lambda", "Lambda + Step Functions", or "Glue/EMR." Walk through building the table for four real jobs that a data platform team faces in a given quarter.

• Job A. Convert each uploaded CSV (~50 MB) to Parquet as it lands in S3. Short, per-object, parallel, stateless.
• Job B. Upsert order events from an SQS queue into Redshift, ~500 messages/sec. Short per-batch, needs idempotency.
• Job C. Nightly join of a 2 B-row orders table with a 400 M-row customers table for a full re-aggregation. Long, huge shuffle, stateful.
• Job D. Reprocess 90 days of raw logs (18 TB, 120 000 objects) into a curated table. Massive fan-out, each object small.

Question. Score each job on the four axes and assign the right execution engine.

Input.

Job	Duration/unit	Volume/event	Concurrency	State
A — CSV→Parquet	~20 s/file	50 MB	bursty, high	none
B — SQS→Redshift	~2 s/batch	~2 KB×10	steady, medium	idempotency key
C — 2B×400M join	~40 min	terabytes	one big job	full shuffle
D — 90-day backfill	~15 s/object	small each	120k objects	none

Code.

# fit_engine.py — score an ETL job and pick the engine
from dataclasses import dataclass

@dataclass
class Job:
    max_unit_seconds: int      # longest a single unit of work takes
    peak_gb_in_memory: float   # data a single unit must hold at once
    units: int                 # how many units (drives fan-out)
    needs_cross_event_state: bool

def pick_engine(j: Job) -> str:
    # Hard wall: a unit that cannot finish under Lambda's ceiling is out
    if j.max_unit_seconds > 14 * 60:          # keep a 1-min safety margin
        return "Step Functions / Glue (exceeds 15-min wall)"
    if j.peak_gb_in_memory > 10:              # Lambda memory ceiling
        return "Glue / EMR (per-unit memory too large)"
    if j.needs_cross_event_state:
        return "Lambda + external state (DynamoDB) or a stream engine"
    if j.units > 10_000:
        return "Lambda fan-out via Step Functions Distributed Map"
    return "Lambda (event trigger + transform)"

jobs = {
    "A": Job(max_unit_seconds=20,   peak_gb_in_memory=0.5, units=1,       needs_cross_event_state=False),
    "B": Job(max_unit_seconds=2,    peak_gb_in_memory=0.1, units=1,       needs_cross_event_state=True),
    "C": Job(max_unit_seconds=2400, peak_gb_in_memory=40,  units=1,       needs_cross_event_state=True),
    "D": Job(max_unit_seconds=15,   peak_gb_in_memory=0.2, units=120_000, needs_cross_event_state=False),
}
for name, j in jobs.items():
    print(name, "->", pick_engine(j))

Step-by-step explanation.

1. The function checks the hard walls first — duration and memory. These are non-negotiable platform limits; no amount of cleverness moves them, so they short-circuit the decision. A unit over ~14 minutes or over 10 GB is disqualified from Lambda immediately.
2. Job A passes every check: 20 s/file, 0.5 GB, no state, one triggering event per file. It is the textbook Lambda fit — an S3 trigger runs the transform per object.
3. Job B needs cross-event state only in the weak sense of deduplication. That is not a shuffle; it is an idempotency key lookup, which Lambda handles by writing to a DynamoDB dedupe table. So B stays on Lambda with external state, not a stream engine.
4. Job C fails on both walls: 40 minutes and 40 GB in memory. This is a distributed shuffle — a Glue or EMR/Spark job. Trying to Lambda it means chunking the join by key range across thousands of invocations with cross-invocation state, which is a reimplementation of Spark you should not write.
5. Job D is the interesting one: each unit is tiny (15 s, 0.2 GB) but there are 120 000 of them. That is not a Lambda-vs-cluster question; it is a fan-out question. Step Functions Distributed Map iterates the object manifest and invokes one Lambda per object with bounded concurrency — the last section covers exactly this.

Output.

Job	Engine chosen	Why
A — CSV→Parquet	Lambda (S3 trigger + transform)	short, per-object, stateless
B — SQS→Redshift	Lambda + DynamoDB idempotency	short batch; dedupe externalised
C — 2B×400M join	Glue / EMR	40 min + 40 GB shuffle
D — 90-day backfill	Step Functions Distributed Map + Lambda	120k tiny units; fan-out

Rule of thumb. Never pick "serverless" because it is trendy. Score the job on (duration × per-unit memory × unit count × cross-event state). The two hard walls — fifteen minutes and 10 GB — disqualify a job from Lambda outright; everything else is a fan-out or external-state design, not a reason to reach for a cluster.

Worked example — the cost crossover versus a warm cluster

Detailed explanation. The "serverless is cheaper" claim is true only below a crossover event rate. Lambda bills per invocation and per GB-second of duration; a Glue/EMR cluster bills per hour it is up regardless of load. Below the crossover, Lambda's pay-nothing-when-idle wins by a mile; above it, the always-busy Lambda fleet loses to a right-sized cluster. Walk through the crossover for a transform job.

• Lambda price. Roughly $0.0000166667 per GB-second plus $0.20 per million requests (illustrative on-demand pricing; regions vary).
• Job. Each invocation uses 1 GB memory for 2 seconds → 2 GB-seconds → ~$0.0000333 compute + request cost per event.
• Cluster alternative. A small always-on job that costs ~$1.50/hour whether it processes 1 event or 1 million.

Question. At what steady event rate does the always-on cluster become cheaper than the Lambda fleet?

Input.

Parameter	Value
Lambda memory	1 GB
Lambda duration/event	2 s
Lambda GB-s price	$0.0000166667
Lambda request price	$0.20 / 1M
Cluster cost	$1.50 / hour

Code.

# crossover.py — where does an always-on cluster beat the Lambda fleet?
GB_S_PRICE   = 0.0000166667
REQ_PRICE    = 0.20 / 1_000_000
MEM_GB       = 1.0
DUR_S        = 2.0
CLUSTER_HR   = 1.50

cost_per_event = MEM_GB * DUR_S * GB_S_PRICE + REQ_PRICE
# events per hour where lambda_cost == cluster_cost
crossover_eph = CLUSTER_HR / cost_per_event
print(f"cost/event      = ${cost_per_event:.8f}")
print(f"crossover       = {crossover_eph:,.0f} events/hour")
print(f"                = {crossover_eph/3600:,.1f} events/second")

Step-by-step explanation.

1. Each event's Lambda cost is memory_gb × duration_s × price_per_gb_s plus the flat per-request charge: 1 × 2 × 0.0000166667 + 0.0000002 ≈ $0.00003353 per event.
2. The cluster is a fixed $1.50/hour regardless of throughput. The crossover is where hourly Lambda spend equals hourly cluster spend: cluster_hr / cost_per_event events per hour.
3. 1.50 / 0.00003353 ≈ 44,700 events/hour, or about 12.4 events/second, sustained, before the always-on cluster wins on pure compute price.
4. But the crossover is not the whole story: below the crossover, Lambda also removes cluster idle time, patching, and scaling ops — real money that does not appear in the compute line. So the practical crossover sits higher than the raw compute crossover.
5. Above the crossover — a steady firehose that keeps hundreds of Lambdas busy 24/7 — the fleet is both pricier and harder to reason about than a provisioned engine. That steady-high-throughput shape is the classic "you outgrew Lambda" signal.

Output.

Metric	Value
Cost per event	~$0.00003353
Crossover rate	~44,700 events/hour
Crossover rate	~12.4 events/second
Below crossover	Lambda cheaper (and no idle)
Far above crossover	Cluster cheaper; consider Glue/EMR

Rule of thumb. Compute the crossover before you commit. Bursty and low-to-medium steady rates favour Lambda decisively (you pay nothing at 3 AM); a sustained firehose well above the crossover favours a provisioned engine. Always add the operational savings of "no cluster to run" on Lambda's side of the ledger.

Worked example — the "10 GB file" streaming trap

Detailed explanation. A frequent failure is loading a whole large object into memory inside a Lambda handler — body.read() on a multi-gigabyte S3 object. It works in dev on a 20 MB sample and blows the memory ceiling (or the time budget) in production on a 4 GB file. The fix is to stream and transform line-by-line so memory stays flat regardless of object size, and to split genuinely huge objects upstream. Walk through the trap and the streaming fix.

• The trap. obj["Body"].read() pulls the entire object into RAM; a 4 GB file needs >4 GB memory and risks the time wall on parse.
• The fix. Iterate the streaming body in chunks/lines, transform each, and write out incrementally — constant memory.
• The guard. If a single object can exceed what one invocation can stream in ~14 minutes, split it upstream (or fan out by byte-range).

Question. Rewrite a load-everything handler into a streaming transform that holds constant memory for any object size within the time budget.

Input.

Component	Before	After
Read strategy	Body.read() (whole file)	iterate Body line-by-line
Peak memory	O(file size)	O(one line + buffer)
Max safe object	~memory ceiling	~what streams in 14 min
Failure mode	OOM / timeout	none (constant memory)

Code.

# streaming_transform.py — constant-memory S3 CSV -> JSONL transform
import boto3, io, json, csv

s3 = boto3.client("s3")

def handler(event, context):
    rec    = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key    = rec["object"]["key"]

    src = s3.get_object(Bucket=bucket, Key=key)["Body"]        # streaming body
    reader = csv.DictReader(io.TextIOWrapper(src, encoding="utf-8"))

    out_key = key.rsplit(".", 1)[0] + ".jsonl"
    buf = io.BytesIO()
    n = 0
    for row in reader:                                          # one row at a time
        row["ingested_at"] = context.aws_request_id            # cheap enrichment
        buf.write((json.dumps(row) + "\n").encode("utf-8"))
        n += 1
        if buf.tell() > 8 * 1024 * 1024:                       # flush every 8 MB
            _flush(bucket, out_key, buf, part=n)
            buf = io.BytesIO()
    if buf.tell():
        _flush(bucket, out_key, buf, part=n)
    return {"rows": n, "out": out_key}

def _flush(bucket, key, buf, part):
    buf.seek(0)
    s3.upload_fileobj(buf, bucket, f"{key}.part-{part}")       # incremental write

Step-by-step explanation.

1. get_object(...)["Body"] returns a streaming body, not bytes. Wrapping it in TextIOWrapper and handing it to csv.DictReader means rows are pulled lazily — memory stays at one row plus the output buffer, no matter how big the object is.
2. The transform enriches each row cheaply (here, stamping the request id) and writes to an in-memory buffer that is flushed to S3 every 8 MB. Flushing incrementally keeps peak memory bounded and avoids holding the whole output in RAM.
3. Because nothing scales with file size, a 50 MB file and a 4 GB file use the same memory. The only remaining limit is time: whether the object can be fully streamed and written inside the fifteen-minute wall.
4. If a single object is so large it cannot stream in ~14 minutes, no in-handler trick saves you — you split the object upstream (smaller exports) or fan out by S3 byte-range so each invocation handles a slice. That is a design change, not a code change.
5. The anti-pattern to unlearn is Body.read() (or pandas.read_csv(whole_object)) inside a Lambda for unbounded input. It is the number-one cause of "works in dev, OOMs in prod" serverless-ETL incidents.

Output.

Object size	Body.read() handler	streaming handler
20 MB	fine	fine
500 MB	needs ≥512 MB mem	flat ~128 MB mem
4 GB	OOM / timeout	flat mem; time-bound only
20 GB	impossible	split upstream / byte-range fan-out

Rule of thumb. In a Lambda ETL handler, never materialise an unbounded input in memory. Stream the input, transform per record, and flush the output incrementally so memory is O(1) in object size. When even streaming can't finish in the time budget, that is your signal to split upstream or fan out — not to raise the memory dial to 10 GB and hope.

Data engineering interview question on Lambda ETL fit

A senior interviewer might ask: "A team wants to move their hourly Airflow batch — which reads new files from S3, converts them to Parquet, and loads a warehouse — onto AWS Lambda so they 'stop paying for idle EC2.' Some of the incoming files are 30 MB, but a few are 8 GB. Walk me through how you'd decide what belongs on Lambda, what doesn't, and how you'd keep the design correct under at-least-once delivery."

Solution Using a per-object fit split with a size-based routing function

# router.py — S3 event -> route each object to the right engine by size
import boto3
s3  = boto3.client("s3")
sfn = boto3.client("stepfunctions")

SMALL_LIMIT_BYTES = 1_500_000_000    # ~1.5 GB: safely streamable in one Lambda
BIG_STATE_MACHINE = "arn:aws:states:...:stateMachine:big-file-etl"

def handler(event, context):
    routed = []
    for record in event["Records"]:                     # S3 can batch notifications
        b = record["s3"]["bucket"]["name"]
        k = record["s3"]["object"]["key"]
        size = record["s3"]["object"]["size"]

        if size <= SMALL_LIMIT_BYTES:
            transform_in_lambda(b, k)                    # short, streamable
            routed.append((k, "lambda"))
        else:
            sfn.start_execution(                         # hand big file to Step Functions
                stateMachineArn=BIG_STATE_MACHINE,
                name=f"etl-{k.replace('/', '_')}-{record['s3']['object']['eTag']}",
                input='{"bucket": "%s", "key": "%s"}' % (b, k),
            )
            routed.append((k, "stepfunctions"))
    return {"routed": routed}

def transform_in_lambda(bucket, key):
    # constant-memory streaming transform (see streaming_transform.py)
    ...

Step-by-step trace.

Step	Input (S3 event)	Action
1	orders/2026-09-05/a.csv size 30 MB	≤ 1.5 GB → transform in Lambda
2	orders/2026-09-05/b.csv size 45 MB	≤ 1.5 GB → transform in Lambda
3	orders/2026-09-05/huge.csv size 8 GB	> 1.5 GB → start Step Functions execution
4	duplicate delivery of a.csv	eTag-named execution / idempotent write dedupes
5	batch of 3 records in one event	loop routes each independently

Walking the trace: the router is itself a tiny Lambda triggered by S3. For each object in the (possibly batched) notification, it reads only the metadata — crucially the size field S3 includes in the event — and routes. Small objects are transformed inline by a streaming handler; the 8 GB object is handed to a Step Functions state machine that can run a chunked or Glue-backed job well past fifteen minutes. Because S3 delivery is at-least-once, a duplicate notification for a.csv must not double-load: the execution name is derived from the object eTag, so a repeat start_execution collides on the name and is rejected, and the Lambda transform path writes idempotently by object key + eTag.

Output:

Object	Size	Engine	Duplicate-safe by
a.csv	30 MB	Lambda stream	key + eTag write
b.csv	45 MB	Lambda stream	key + eTag write
huge.csv	8 GB	Step Functions	eTag execution name
a.csv (retry)	30 MB	Lambda stream	idempotent overwrite

Why this works — concept by concept:

• Per-object routing — the fit decision is made per event, not per pipeline. Small objects ride the cheap Lambda path; oversized objects are handed off. One S3 trigger serves both, so the team keeps the "no idle EC2" win without pretending an 8 GB file fits in fifteen minutes.
• Metadata-only decision — the router reads object.size from the event and never downloads the object to decide. The routing Lambda is milliseconds long and megabytes-free, so it scales trivially with the notification rate.
• Fifteen-minute wall respected — anything that might exceed the wall is escalated to Step Functions before a single byte is read, rather than discovered mid-transform when the invocation is killed at minute fifteen.
• At-least-once safety — the eTag-derived execution name and the idempotent write make duplicate S3 notifications harmless. This is the non-negotiable half of any event-driven ETL design; a router that double-loads on retry is a correctness bug, not a performance one.
• Cost — the router is O(1) per event (metadata only), the streaming transform is O(rows) with O(1) memory, and the Step Functions path is used only for the rare oversized object. Compared with a warm EC2 batch, idle cost drops to zero; compared with "Lambda everything," you never hit the wall. Net O(1) idle spend, O(rows) transform.

ETL
Topic — etl
ETL problems on serverless ingestion pipelines

Practice →

Data Topic — data-processing Data-processing problems on streaming transforms

Practice →

---

2. Event sources & triggers — S3 / SQS / Kinesis

S3 pushes, SQS and Kinesis are polled — the event source mapping is the knob you tune

The mental model in one line: an event-driven Lambda is only as good as its trigger, and triggers split into two families — S3 (and SNS, EventBridge) push an event straight at your function one-at-a-time-ish, while SQS and Kinesis (and DynamoDB Streams) are polled by an AWS-managed component called the event source mapping that batches records and invokes your function with a batch — so the batch size, batch window, concurrency, and failure behaviour you configure on that mapping decide throughput, latency, ordering, and how a single bad record affects the rest. Every senior serverless-ETL design lives or dies on getting the event source mapping knobs right; "it just triggers" is the answer that fails the interview.

Push versus poll — the two trigger families.

• Push sources (S3, SNS, EventBridge). The service invokes Lambda asynchronously: it hands the event to Lambda's internal queue and returns. Lambda retries async failures twice and can route them to a DLQ / destination. You do not configure batch size for S3; each notification carries one (occasionally a few) records.
• Poll sources (SQS, Kinesis, DynamoDB Streams). The event source mapping — an AWS-managed poller you attach to the function — reads records, forms a batch, and invokes Lambda synchronously with that batch. Throughput, latency, and failure semantics are all governed by the mapping's config, not by the function.
• Why it matters. Push sources give you low config and per-event fan-in; poll sources give you batching (fewer invocations, more efficiency) but require you to handle a batch correctly, including partial failure.

S3 event notifications — the object-created trigger.

• What fires. s3:ObjectCreated:* (Put, Post, CompleteMultipartUpload, Copy) and delete/restore events. You wire the bucket notification to invoke the Lambda.
• Filters. Prefix and suffix filters scope the trigger — e.g. prefix raw/ and suffix .csv so only raw CSVs invoke the function. Filtering at the source is cheaper than filtering in code.
• Delivery. At-least-once and not strictly ordered. A single upload occasionally yields more than one notification, and notifications can arrive out of order. Design the handler to be idempotent and order-independent.
• The multipart gotcha. A large multipart upload fires CompleteMultipartUpload, not Put; scope your event types so you don't miss big files.

SQS as an event source — batching and partial failure.

• Batch size & window. The mapping reads up to batchSize messages (max 10 000 for standard queues via the mapping, commonly 10) or waits up to maximumBatchingWindowInSeconds to fill a batch. Bigger batches = fewer invocations, higher per-invocation work.
• Visibility timeout. While a batch is in flight, its messages are invisible. The queue visibility timeout must be ≥ the function timeout (a common rule: 6× the function timeout) or messages reappear and get double-processed.
• Partial batch response. By default, if the handler throws, the whole batch returns to the queue and is retried — reprocessing the messages that already succeeded. Enabling ReportBatchItemFailures lets the handler return only the failed message IDs, so successes are not redelivered.
• FIFO queues. Preserve order within a message group and give exactly-once processing only if you also dedupe; standard queues are at-least-once and unordered.

Kinesis as an event source — shards and ordering.

• Shards = parallelism. A stream has N shards; the mapping runs (by default) one concurrent invocation per shard, preserving order within a shard. The partition key decides which shard a record lands on — same key, same shard, ordered.
• Parallelization factor. You can raise concurrency to up to 10 invocations per shard while still preserving per-partition-key order; this lifts throughput on hot shards.
• Batch size & iterator. The mapping reads up to batchSize records per shard per invocation, starting at TRIM_HORIZON (oldest) or LATEST. A batch spans one shard.
• Poison-pill risk. A record the handler always fails on will block its shard forever unless you cap retries (maximumRetryAttempts), cap record age (maximumRecordAgeInSeconds), split the batch on error (bisectBatchOnFunctionError), and route failures to an on-failure destination.

Common interview probes on triggers.

• "How does an S3 trigger deliver — once or at-least-once?" — at-least-once, unordered; handler must be idempotent.
• "Why set visibility timeout ≥ function timeout on SQS?" — else in-flight messages reappear and double-process.
• "What is partial batch response and why enable it?" — return only failed IDs so successes aren't retried.
• "What guarantees ordering in Kinesis?" — per-shard order via the partition key; one invocation per shard by default.

Worked example — S3 object-created trigger to a curated Parquet copy

Detailed explanation. The canonical push trigger: a CSV lands under raw/, Lambda converts it to Parquet under curated/, idempotently. The bucket notification is scoped with a prefix/suffix filter so only the right objects fire. Build the trigger config and the handler.

• Filter. Prefix raw/, suffix .csv — nothing else invokes the function.
• Idempotency. Output key is derived from input key; a re-delivered notification overwrites the same Parquet object (harmless).
• Events. Include Put and CompleteMultipartUpload so large uploads are not missed.

Question. Configure the S3 notification and write the idempotent transform handler.

Input.

Setting	Value
Bucket	data-lake-prod
Trigger events	ObjectCreated:Put, ObjectCreated:CompleteMultipartUpload
Filter	prefix raw/, suffix .csv
Output	curated/{same-stem}.parquet

Code.

// S3 bucket notification configuration (LambdaFunctionConfigurations)
{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:...:function:csv-to-parquet",
      "Events": ["s3:ObjectCreated:Put", "s3:ObjectCreated:CompleteMultipartUpload"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {"Name": "prefix", "Value": "raw/"},
            {"Name": "suffix", "Value": ".csv"}
          ]
        }
      }
    }
  ]
}

# handler.py — idempotent CSV -> Parquet on S3 object-created
import boto3, io
import pyarrow as pa
import pyarrow.csv as pacsv
import pyarrow.parquet as pq

s3 = boto3.client("s3")

def handler(event, context):
    results = []
    for rec in event["Records"]:                       # S3 may batch a few records
        bucket = rec["s3"]["bucket"]["name"]
        key    = rec["s3"]["object"]["key"]
        if not key.startswith("raw/") or not key.endswith(".csv"):
            continue                                    # defense in depth vs filter

        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
        table = pacsv.read_csv(io.BytesIO(body))        # small files only (see §1 trap)

        out_key = "curated/" + key[len("raw/"):-len(".csv")] + ".parquet"
        buf = io.BytesIO()
        pq.write_table(table, buf)
        buf.seek(0)
        # Deterministic key => re-delivery overwrites the same object (idempotent)
        s3.put_object(Bucket=bucket, Key=out_key, Body=buf.getvalue())
        results.append(out_key)
    return {"written": results}

Step-by-step explanation.

1. The notification config scopes the trigger at the source: only objects under raw/ ending in .csv invoke Lambda. This is cheaper and cleaner than invoking on every object and filtering in code — though the handler still double-checks (defense in depth).
2. Including CompleteMultipartUpload in the event list is what makes large uploads reliable — the AWS SDK uploads big files as multipart, which fires that event, not Put. Miss it and your biggest files silently never trigger.
3. The handler loops over event["Records"] because a single notification can carry more than one record. Assuming exactly one record is a latent bug that surfaces under load.
4. The output key is derived deterministically from the input key. Because S3 delivery is at-least-once, the same object may fire twice; writing to the same deterministic key means the second write simply overwrites identical bytes — idempotent by construction, no dedupe table needed.
5. This handler uses read_csv on the whole body, which is only safe because the filter/design guarantees small raw/ files. For unbounded input you'd switch to the streaming pattern from section 1 — the trigger config is identical.

Output.

Event	Input key	Output key
Put	raw/2026/09/a.csv	curated/2026/09/a.parquet
CompleteMultipartUpload	raw/2026/09/big.csv	curated/2026/09/big.parquet
Duplicate Put	raw/2026/09/a.csv	curated/2026/09/a.parquet (overwrite)
Non-matching	logs/x.json	(never fires; filtered)

Rule of thumb. Scope S3 triggers with prefix/suffix filters, always include CompleteMultipartUpload for large files, loop over all records in the event, and derive the output key deterministically so at-least-once delivery is harmless. The trigger config is where correctness starts, before any transform code runs.

Worked example — SQS batch consumer with partial batch response

Detailed explanation. An SQS-triggered Lambda receives a batch of up to 10 messages. If one message's transform fails and the handler throws, the entire batch is retried by default — re-processing the 9 that succeeded. Enabling ReportBatchItemFailures and returning the failed IDs fixes this. Build the consumer.

• Mapping config. batchSize: 10, functionResponseTypes: ["ReportBatchItemFailures"].
• Visibility timeout. Set on the queue to ≥ 6× the function timeout so in-flight batches don't reappear.
• Contract. The handler returns {"batchItemFailures": [{"itemIdentifier": messageId}, ...]} for only the messages that failed.

Question. Write an SQS consumer that processes each message independently and reports only the failures.

Input.

Setting	Value
Queue	orders-ingest
Batch size	10
Function timeout	30 s
Visibility timeout	180 s (6×)
Response type	ReportBatchItemFailures

Code.

# sqs_consumer.py — process each message; report only failures
import json

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        try:
            body = json.loads(msg["body"])
            process_order(body)                      # your transform / upsert
        except Exception as e:
            # Do NOT re-raise: that would fail the WHOLE batch
            print(f"failed messageId={msg['messageId']}: {e}")
            failures.append({"itemIdentifier": msg["messageId"]})
    # Only these IDs return to the queue for retry; the rest are deleted
    return {"batchItemFailures": failures}

def process_order(order):
    if order.get("total_cents") is None:             # a poison-pill example
        raise ValueError("missing total_cents")
    # ... idempotent upsert into the warehouse ...

Step-by-step explanation.

1. The mapping is configured with functionResponseTypes = ["ReportBatchItemFailures"]. This changes the contract: instead of "throw to fail the batch," the handler returns a list of the message IDs that failed.
2. The loop processes each message inside its own try/except. Critically, the handler never re-raises — re-raising would signal batch-level failure and force SQS to redeliver all 10 messages, double-processing the successful ones.
3. Successful messages are implicitly acknowledged (SQS deletes them) because they are not in the returned batchItemFailures. Failed messages are returned by ID and become visible again after the visibility timeout for another attempt.
4. The queue's visibility timeout (180 s) is set to well above the function timeout (30 s). If it were lower, a slow batch's messages would reappear while the batch was still processing — the classic double-processing bug. The 6× rule bakes in headroom for retries within Lambda.
5. A message that fails every time (a poison pill — here, missing total_cents) will retry until the queue's maxReceiveCount redrive policy ships it to a DLQ. Partial batch response ensures that poison pill never drags its 9 healthy batch-mates down with it.

Output.

Message	Result	Returned in batchItemFailures?
m1 (valid)	processed, deleted	no
m2 (missing total)	raised → recorded	yes
m3..m10 (valid)	processed, deleted	no
batch outcome	9 done, 1 retried	only m2 redelivered

Rule of thumb. For any SQS-triggered Lambda, enable ReportBatchItemFailures, catch per-message inside the loop, never re-raise, and return only the failed IDs. Pair it with a queue visibility timeout ≥ 6× the function timeout and a redrive policy to a DLQ. This is the difference between "one bad message poisons the batch" and "one bad message is quarantined."

Worked example — Kinesis shard consumer with ordering and parallelization

Detailed explanation. A Kinesis-triggered Lambda reads a batch of records from a single shard, in order. Records with the same partition key always land on the same shard, so per-key ordering holds. The parallelizationFactor lifts throughput on hot shards without breaking per-key order. Build the consumer.

• Ordering. Guaranteed within a shard; the partition key routes records to shards.
• Parallelization factor. Up to 10 concurrent batches per shard, still ordered per partition key.
• Checkpointing. The mapping advances the shard iterator only after a successful invocation (or per partial-failure config).

Question. Write a Kinesis consumer that transforms records preserving per-key order and explain how parallelization factor keeps order.

Input.

Setting	Value
Stream	clickstream
Shards	8
Batch size	500
Starting position	TRIM_HORIZON
Parallelization factor	4

Code.

# kinesis_consumer.py — ordered per-key transform of a shard batch
import base64, json

def handler(event, context):
    failures = []
    for rec in event["Records"]:                          # all from ONE shard, in order
        try:
            payload = base64.b64decode(rec["kinesis"]["data"])
            click = json.loads(payload)
            # partition key == user_id => all of a user's events are ordered here
            transform_click(click, seq=rec["kinesis"]["sequenceNumber"])
        except Exception as e:
            print(f"failed seq={rec['kinesis']['sequenceNumber']}: {e}")
            failures.append({"itemIdentifier": rec["kinesis"]["sequenceNumber"]})
    return {"batchItemFailures": failures}                 # ReportBatchItemFailures on the ESM

def transform_click(click, seq):
    # idempotent write keyed by (user_id, event_ts, seq)
    ...

Step-by-step explanation.

1. Every record in event["Records"] comes from one shard and is delivered in sequence-number order. The handler can therefore rely on order within the batch — e.g. to fold a user's clicks in the order they happened.
2. The Kinesis partition key (here user_id) is hashed to a shard. All of one user's events hit the same shard, so per-user ordering is preserved end to end even across many shards and many concurrent invocations.
3. parallelizationFactor: 4 lets up to 4 batches from the same shard run concurrently — but Kinesis still guarantees that records sharing a partition key are processed in order, because it groups by key across those concurrent batches. You get more throughput on a hot shard without sacrificing per-key ordering.
4. The consumer uses partial batch response too (returning failed sequenceNumbers). Combined with bisectBatchOnFunctionError on the mapping, a single poison record is isolated rather than blocking the shard — covered in section 4.
5. Checkpointing is automatic: the mapping advances the shard iterator past the batch only when the invocation succeeds (or past the successful prefix under partial-failure config). A thrown, unhandled error would re-read the same batch — the reason idempotency is mandatory.

Output.

Aspect	Behaviour
Order within shard	preserved (sequence order)
Order across shards	none (independent)
Same partition key	always same shard, ordered
parallelizationFactor 4	4× throughput, key order kept
Failed record	returned by sequenceNumber

Rule of thumb. Treat a Kinesis batch as an ordered, single-shard slice. Route related records with a stable partition key so per-key order holds, lift throughput with parallelizationFactor (not more shards, when the shard isn't the bottleneck), and always process idempotently because a failure re-reads the batch. Ordering is a partition-key property, not a wish.

Data engineering interview question on event source mappings

A senior interviewer might ask: "You have an SQS queue feeding a Lambda that upserts events into Redshift. Under a traffic spike, you see the same event applied twice and occasional 'message became visible again mid-processing' warnings, and one malformed message keeps failing the whole batch. Design the event source mapping and handler so successes aren't retried, in-flight messages don't reappear, and the poison message is quarantined."

Solution Using ReportBatchItemFailures with a tuned visibility timeout and DLQ redrive

# robust_sqs_consumer.py
import json

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        mid = msg["messageId"]
        try:
            body = json.loads(msg["body"])
            upsert_idempotent(body, dedupe_key=body["event_id"])   # safe under retry
        except Exception as e:
            print(f"quarantine-candidate {mid}: {e}")
            failures.append({"itemIdentifier": mid})
    return {"batchItemFailures": failures}

def upsert_idempotent(body, dedupe_key):
    # INSERT ... ON CONFLICT (event_id) DO UPDATE — Redshift/Postgres upsert
    ...

# Event source mapping + queue configuration (conceptual)
SQS queue: orders-ingest
  VisibilityTimeout          = 180        # >= 6 x function timeout (30s)
  RedrivePolicy:
    maxReceiveCount          = 5          # after 5 tries -> DLQ
    deadLetterTargetArn      = orders-ingest-dlq

Event source mapping (queue -> lambda):
  BatchSize                  = 10
  MaximumBatchingWindowInSec = 5
  FunctionResponseTypes      = [ReportBatchItemFailures]
  ScalingConfig.MaximumConcurrency = 20   # cap concurrency to protect Redshift

Step-by-step trace.

Step	Condition	Behaviour
1	batch of 10 arrives	handler processes each in its own try/except
2	m4 malformed, throws	recorded in batchItemFailures; not re-raised
3	m1–m3, m5–m10 succeed	deleted from queue (not in failures list)
4	m4 returned by ID	reappears after 180 s visibility timeout
5	m4 fails 5 times	redrive policy ships it to orders-ingest-dlq
6	duplicate of m1 delivered	event_id conflict → upsert is a no-op

Walking the trace: the visibility timeout of 180 s (6× the 30 s function timeout) means a batch stays invisible for the whole time it could plausibly be processing and retrying, so the "became visible mid-processing" double-apply disappears. Partial batch response deletes the 9 healthy messages and only returns the malformed one, so successes are never reprocessed. The malformed message retries up to maxReceiveCount = 5 and then lands in the DLQ, quarantined for a human — the batch is never blocked. And because the upsert dedupes on event_id, even a genuine at-least-once duplicate is a harmless no-op.

Output:

Symptom before	Fix	Result
Event applied twice	idempotent upsert on event_id	duplicate is a no-op
Message reappears mid-processing	visibility timeout 180 s (6×)	no premature redelivery
One message fails whole batch	ReportBatchItemFailures	only that ID retried
Poison message loops forever	redrive maxReceiveCount 5 → DLQ	quarantined after 5 tries
Spike stampedes Redshift	ScalingConfig max concurrency 20	bounded parallel upserts

Why this works — concept by concept:

• Partial batch response — returning batchItemFailures instead of throwing tells SQS to delete the successes and redeliver only the failed IDs. It converts "one bad message poisons ten" into "one bad message retries alone."
• Visibility timeout ≥ 6× function timeout — a batch stays invisible for the entire window it could be processing plus retries, which removes the race where a message reappears and a second invocation double-applies it.
• Idempotent upsert on a dedupe key — because delivery is at-least-once, correctness cannot depend on "exactly once." The ON CONFLICT upsert makes any duplicate a no-op, so retries and redeliveries are safe.
• Redrive policy to a DLQ — a poison message is bounded to maxReceiveCount attempts and then parked in a dead-letter queue for inspection, so it never blocks the pipeline indefinitely.
• Cost — O(batch) per invocation with O(1) dedupe lookups; capped concurrency bounds downstream load. Compared with the naive "throw on any error" consumer, this reprocesses only failures (not whole batches), so wasted re-work drops from O(batch × retries) to O(failures × retries).

Event
Topic — event-processing
Event-processing problems on queue and stream consumers

Practice →

ETL Topic — etl ETL problems on batch trigger design

Practice →

---

3. Concurrency, memory & cold starts

One sandbox per event — concurrency, memory, and cold starts are the three dials that decide throughput, latency, and blast radius

The mental model in one line: Lambda scales by running one isolated execution environment per concurrent event, so your throughput is concurrency = arrival_rate × average_duration, your per-invocation speed is set by the memory dial (more memory buys proportionally more CPU), and your tail latency is set by cold starts (the one-time init cost paid whenever a new sandbox is created) — and the three dials interact, because raising concurrency can stampede a downstream, capping it can throttle the source, and a big deployment package or a VPC attachment lengthens every cold start. Senior serverless-ETL tuning is choosing reserved concurrency to bound blast radius, provisioned concurrency to kill cold starts where latency matters, and the memory setting that minimises cost-per-invocation.

The concurrency model — three kinds of concurrency.

• Unreserved (on-demand) concurrency. By default all functions in an account share a regional pool (commonly 1 000, raisable). A burst spins up sandboxes until the pool or the burst rate limit is hit, then throttles with 429 TooManyRequestsException.
• Reserved concurrency. A cap you set per function. It does two things at once: it guarantees that function at least that many slots, and it limits it to at most that many — protecting downstreams and protecting other functions from being starved by this one.
• Provisioned concurrency. Pre-initialised sandboxes kept warm so there is no cold start for the first N concurrent requests. You pay for them whether used or not; you use it where p99 latency matters (synchronous APIs, latency-sensitive steps), rarely for pure batch ETL.
• The formula. Steady-state concurrency ≈ requests_per_second × average_duration_seconds. 100 rps × 0.2 s = 20 concurrent sandboxes. This is the number you reserve around.

Cold start anatomy — what the first invocation pays.

• Init phase. Creating a new sandbox downloads your code/layers, starts the runtime, and runs your module-level (init) code before the handler. That one-time cost is the cold start; subsequent invocations on the same warm sandbox skip it.
• What lengthens it. A large deployment package or many layers; heavy import-time work (loading a big model, opening connections at import); a VPC attachment (historically ENI setup, now much faster but still non-zero); and runtime choice (interpreted vs compiled startup).
• What shortens it. Smaller packages, lazy imports, moving one-time setup into init and reusing it across invocations, and provisioned concurrency for the paths that can't tolerate the penalty.
• Warm reuse. A sandbox is reused for subsequent events; connections, clients, and cached data created at init are reused — which is exactly why you create the DB client once at module scope, not per invocation.

Memory equals CPU — the power dial.

• The coupling. Memory is the only performance dial; CPU (and network) scale proportionally with it. At ~1 769 MB you get roughly one full vCPU; more memory gives more vCPUs.
• The counter-intuitive part. Raising memory can lower cost: if doubling memory more than halves duration (CPU-bound work), the GB-second product drops. Power-tuning finds the sweet spot.
• The method. Sweep memory settings, measure duration and cost per setting, pick the minimum-cost (or minimum-latency) point. AWS Lambda Power Tuning automates this sweep.

Throttling and back-pressure.

• Throttling. When concurrency hits the cap, sync invocations get 429; async invocations are retried internally; poll-based mappings slow their polling (natural back-pressure onto SQS/Kinesis).
• Back-pressure as a feature. Reserving low concurrency on a Lambda that writes to a small RDS instance means SQS simply holds messages when the cap is hit, smoothing the load instead of overwhelming the database.
• Account blast radius. Without per-function reserved concurrency, one runaway function can consume the whole account pool and throttle everything else. Reserve critical functions.

Common interview probes on concurrency.

• "How does Lambda scale?" — one concurrent sandbox per in-flight event; concurrency = rate × duration.
• "Reserved vs provisioned concurrency?" — reserved caps/guarantees slot count; provisioned pre-warms sandboxes to remove cold starts.
• "How do you make Lambda go faster?" — raise the memory dial (more CPU); it can even lower cost.
• "How do you stop Lambda from overwhelming a database?" — reserved concurrency cap → SQS back-pressure.

Worked example — reserved concurrency to protect a downstream database

Detailed explanation. A Lambda upserts into a small RDS Postgres that tolerates ~50 concurrent connections. Under a spike, unbounded Lambda concurrency opens hundreds of connections and topples the database. Reserved concurrency caps the function so it can never exceed the DB's connection budget; the SQS source absorbs the overflow. Build it.

• DB budget. ~50 safe concurrent connections.
• Cap. Reserve 40 concurrency on the function (headroom below 50).
• Overflow. SQS holds messages; the mapping polls only as fast as slots free up.

Question. Choose a reserved concurrency value and explain the back-pressure behaviour under a 10× spike.

Input.

Parameter	Value
Safe DB connections	50
Reserved concurrency	40
Connections per invocation	1 (pooled/reused)
Source	SQS (poll)
Spike	10× normal arrival

Code.

# db_writer.py — one pooled connection per warm sandbox
import os, psycopg2

# Created ONCE at init and reused across invocations on this warm sandbox.
_conn = psycopg2.connect(os.environ["DB_DSN"])

def handler(event, context):
    with _conn.cursor() as cur:
        for msg in event["Records"]:
            cur.execute(
                "INSERT INTO orders(id, total_cents) VALUES (%s, %s) "
                "ON CONFLICT (id) DO UPDATE SET total_cents = EXCLUDED.total_cents",
                (msg_id(msg), total(msg)),
            )
    _conn.commit()
    return {"ok": True}

# Reserved concurrency (set on the function)
aws lambda put-function-concurrency \
  --function-name db-writer \
  --reserved-concurrent-executions 40

Step-by-step explanation.

1. The database connection is created at module scope (init), so each warm sandbox holds exactly one reused connection. With reserved concurrency 40, at most 40 sandboxes exist, so at most 40 DB connections — comfortably under the 50 budget.
2. put-function-concurrency 40 both guarantees and caps: the function always has up to 40 slots and can never exceed them. This is the hard ceiling that protects the database.
3. Under a 10× spike, arrivals exceed 40 concurrent capacity. Because the source is SQS (poll-based), the event source mapping simply stops pulling faster than slots free up — messages stay in the queue, visible-timeout-protected, and drain as invocations complete. This is back-pressure, not failure.
4. Had the cap been absent, Lambda would have scaled to hundreds of sandboxes, opened hundreds of connections, and exhausted the DB's connection limit — a cascading outage. The cap trades a little latency (queue depth grows briefly) for stability.
5. The one subtlety: connection reuse depends on the sandbox staying warm. On a cold start a fresh connection is opened, which is why 40 (not 50) leaves headroom for the brief overlap while old sandboxes drain and new ones warm.

Output.

Load	Concurrency	DB connections	Queue depth
Normal	~8	~8	~0
10× spike	capped at 40	≤ 40	grows, then drains
No cap (hypothetical)	400+	400+ → DB down	0 (but outage)
After spike	falls to ~8	~8	back to ~0

Rule of thumb. When a Lambda writes to a capacity-limited downstream, set reserved concurrency below the downstream's safe limit and let a poll-based source (SQS/Kinesis) absorb overflow as back-pressure. Create the connection once at init and reuse it. Concurrency is your rate-limiter; the queue is your shock absorber.

Worked example — provisioned concurrency for a latency-sensitive step

Detailed explanation. A Lambda sits on a synchronous path where p99 latency matters (an enrichment call in a request flow). Cold starts add hundreds of milliseconds to the first request after idle. Provisioned concurrency keeps N sandboxes pre-initialised so those requests never pay init cost. Build it and reason about the trade-off.

• Problem. p99 spikes to ~800 ms on cold starts vs ~40 ms warm.
• Fix. Provision 10 warm sandboxes for the expected concurrent load.
• Trade-off. You pay for provisioned concurrency whether used or not; size it to steady-state, not peak.

Question. Configure provisioned concurrency and explain when it is (and isn't) worth it.

Input.

Parameter	Value
Warm duration	~40 ms
Cold start penalty	~760 ms
Steady concurrency	~10
Provisioned concurrency	10
Path	synchronous enrichment

Code.

# Provisioned concurrency on a published version/alias
aws lambda put-provisioned-concurrency-config \
  --function-name enrich \
  --qualifier prod \
  --provisioned-concurrent-executions 10

# Optional: autoscale provisioned concurrency on a schedule / utilization
aws application-autoscaling register-scalable-target \
  --service-namespace lambda \
  --resource-id function:enrich:prod \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --min-capacity 5 --max-capacity 50

# enrich.py — heavy init done ONCE, reused by every warm/provisioned sandbox
import json
MODEL = load_lookup_table()          # expensive; runs during init, not per request

def handler(event, context):
    key = event["key"]
    return {"enriched": MODEL.get(key, "unknown")}   # ~40 ms, no init cost

Step-by-step explanation.

1. Provisioned concurrency is attached to a published version/alias (prod), not $LATEST. AWS keeps 10 sandboxes fully initialised — init code already run — so the first 10 concurrent requests hit warm environments and skip the ~760 ms penalty.
2. The expensive load_lookup_table() runs during init, once per sandbox. With provisioned concurrency, that init happens ahead of traffic, so no request ever waits for it. This is the whole point: move the cost off the request path.
3. Sizing is to steady-state concurrency (~10), not peak. Beyond the provisioned count, extra load spills to on-demand sandboxes that do pay cold start — acceptable for the rare overflow, and you can autoscale provisioned concurrency to track it.
4. The trade-off is money: provisioned concurrency bills for the warm sandboxes continuously. It is worth it on latency-sensitive synchronous paths; it is usually not worth it for pure asynchronous batch ETL, where a few hundred ms of cold start on a multi-second job is irrelevant.
5. This is why most ETL Lambdas skip provisioned concurrency entirely: batch jobs care about throughput and cost, not tail latency. Reserve provisioned concurrency for the request-flow steps where p99 is a user-facing SLO.

Output.

Metric	On-demand only	Provisioned (10)
p50 latency	~40 ms	~40 ms
p99 latency (idle→spike)	~800 ms	~40 ms
Cost when idle	~$0	pay for 10 warm
Best for	batch ETL	latency-sensitive sync

Rule of thumb. Use provisioned concurrency only where cold-start tail latency is a user-facing SLO, size it to steady-state concurrency, and keep expensive setup in init so the pre-warmed sandboxes actually absorb it. For batch ETL, a cold start is a rounding error — don't pay to remove it.

Worked example — memory power-tuning to minimise cost

Detailed explanation. Because CPU scales with memory, a CPU-bound transform can get cheaper at higher memory if the speedup outpaces the price increase. Sweep memory settings, measure duration and GB-second cost, and pick the minimum-cost point. Walk through a power-tuning sweep.

• Workload. A CPU-bound Parquet compression step.
• Sweep. 512 MB → 1024 → 1769 → 3008 MB.
• Pick. The memory with the lowest duration × memory (cost), or lowest duration if latency-bound.

Question. Given a sweep, choose the cost-optimal memory setting.

Input.

Memory (MB)	Duration (ms)	GB-seconds
512	4200	2.10
1024	2100	2.10
1769	1150	1.98
3008	1000	2.94

Code.

# power_tune.py — pick the cheapest memory from a sweep
GB_S_PRICE = 0.0000166667
sweep = [
    (512,  4200),
    (1024, 2100),
    (1769, 1150),
    (3008, 1000),
]
def gb_seconds(mem_mb, dur_ms):
    return (mem_mb / 1024) * (dur_ms / 1000)

for mem, dur in sweep:
    gbs  = gb_seconds(mem, dur)
    cost = gbs * GB_S_PRICE
    print(f"{mem:>4} MB  {dur:>5} ms  {gbs:.2f} GB-s  ${cost:.8f}")

best = min(sweep, key=lambda md: gb_seconds(*md))
print("cheapest:", best[0], "MB")

Step-by-step explanation.

1. Cost is memory_gb × duration_s × price. At 512 MB the job is slow (4200 ms) and costs 2.10 GB-s; doubling to 1024 MB roughly halves duration, so cost is identical — proof the work is CPU-bound and scaling linearly with CPU.
2. At 1769 MB (≈ one full vCPU), duration drops to 1150 ms and GB-seconds fall to 1.98 — the minimum. Here the speedup slightly outpaces the price increase, so this is the cost-optimal point.
3. At 3008 MB the job is only marginally faster (1000 ms) but you pay for ~1.7× the memory, so GB-seconds jump to 2.94 — more expensive. Past the point where extra CPU stops helping, higher memory only raises cost.
4. The sweet spot is workload-specific: I/O-bound jobs barely speed up with memory (so the smallest memory that fits is cheapest), while CPU-bound jobs often minimise around one vCPU. You must measure, not guess.
5. If latency (not cost) is the objective, pick 3008 MB for its 1000 ms; if cost, pick 1769 MB. Power-tuning makes the trade explicit instead of leaving memory at a default 128 MB where CPU-bound jobs crawl.

Output.

Memory	Duration	GB-s	Verdict
512 MB	4200 ms	2.10	slow, same cost
1024 MB	2100 ms	2.10	faster, same cost
1769 MB	1150 ms	1.98	cost-optimal
3008 MB	1000 ms	2.94	fastest, pricier

Rule of thumb. Never leave a CPU-bound Lambda at 128 MB. Sweep memory, plot duration and GB-seconds, and pick the minimum-cost point (often near one vCPU) or the minimum-latency point if that's the objective. Memory is a performance dial, not just a capacity dial.

Data engineering interview question on concurrency and cold starts

A senior interviewer might ask: "Your S3-triggered transform Lambda writes to a Redshift cluster and a small RDS metadata DB. During a batch of 5 000 files landing at once, Redshift is fine but the RDS DB hits its connection limit and the whole pipeline stalls, and you also see elevated p99 from cold starts. Design the concurrency and connection strategy so the RDS DB is protected, throughput stays high, and cold starts stop hurting."

Solution Using reserved concurrency, connection reuse, and SQS back-pressure

# transform.py — S3 events buffered through SQS, capped concurrency, reused conn
import os, psycopg2, boto3

_rds = psycopg2.connect(os.environ["RDS_DSN"])   # once per warm sandbox
_redshift_data = boto3.client("redshift-data")

def handler(event, context):
    for msg in event["Records"]:                 # SQS batch (S3 -> SQS -> Lambda)
        obj = parse(msg["body"])
        rows = transform_object(obj)             # streaming, constant memory
        load_to_redshift(_redshift_data, rows)   # scales fine
        with _rds.cursor() as cur:               # metadata write: the bottleneck
            cur.execute(
                "INSERT INTO file_audit(key, rows) VALUES (%s, %s) "
                "ON CONFLICT (key) DO UPDATE SET rows = EXCLUDED.rows",
                (obj["key"], len(rows)),
            )
    _rds.commit()
    return {"ok": True}

# Wiring + limits
S3 (ObjectCreated) ---> SQS (buffer) ---> Lambda transform
  Reserved concurrency (transform)      = 40    # < RDS safe conn limit (50)
  SQS VisibilityTimeout                 = 180   # >= 6 x function timeout
  SQS RedrivePolicy.maxReceiveCount     = 5 -> DLQ
  RDS connection                        = created at init, reused
  Redshift writes                       = redshift-data API (no held conn)

Step-by-step trace.

Step	Event	Behaviour
1	5 000 files land	S3 fans notifications into an SQS buffer
2	mapping polls SQS	invokes transform up to reserved cap 40
3	40 sandboxes warm	≤ 40 RDS connections (reused per sandbox)
4	overflow	remaining messages wait in SQS (back-pressure)
5	cold starts	init opens RDS conn once; reused for the batch
6	queue drains	concurrency falls, connections released

Walking the trace: inserting an SQS buffer between S3 and the transform turns a 5 000-file thundering herd into a smoothly polled queue. Reserved concurrency 40 caps sandboxes below the RDS 50-connection limit, and because the connection is created at init and reused, 40 sandboxes mean at most 40 connections — the DB is safe. Redshift, which scales fine, is written via the connectionless redshift-data API so it never competes for the RDS budget. Cold starts still happen, but each sandbox pays init once and then processes many messages warm, so the amortised p99 impact across a 5 000-file batch is negligible — no provisioned concurrency needed for a batch job.

Output:

Symptom before	Fix	Result
RDS connection exhaustion	reserved concurrency 40 + reuse	≤ 40 connections
Thundering herd of 5 000	S3 → SQS buffer	smooth polling
Pipeline stalls	back-pressure in queue	drains, no failure
Cold-start p99	init once, reuse warm	amortised away
Redshift contention	redshift-data API	no held connections

Why this works — concept by concept:

• Reserved concurrency as a rate limiter — capping the function below the RDS connection budget makes it impossible to exhaust the database, converting a spike into bounded parallelism.
• Connection reuse at init — creating the connection at module scope means one connection per sandbox, reused across the batch, so concurrency (not invocation count) bounds connections.
• SQS buffer for back-pressure — placing a queue between the bursty S3 source and the capped consumer absorbs the herd; the mapping polls only as fast as slots free, so overflow waits instead of failing.
• Connectionless downstream for the scalable store — routing Redshift writes through the redshift-data API keeps the scalable warehouse off the scarce-connection path entirely.
• Cold-start amortisation — for batch ETL, each warm sandbox handles many messages, so the one-time init cost is spread thin; provisioned concurrency would be wasted spend here.
• Cost — O(files) work at O(cap) parallelism with O(cap) reused connections; the SQS buffer adds a cheap per-message poll. Compared with unbounded concurrency, this trades a little queue latency for a database that never falls over, at no extra compute cost.

Data
Topic — data-processing
Data-processing problems on parallel throughput tuning

Practice →

ETL Topic — etl ETL problems on concurrency and back-pressure

Practice →

---

4. Error handling — DLQ, retries, idempotency

At-least-once means retries are guaranteed — so handlers must be idempotent and poison pills must have somewhere to go

The mental model in one line: every Lambda event source delivers at-least-once and retries on failure, but how it retries depends on the invocation type — synchronous callers retry themselves, asynchronous sources (S3, SNS, EventBridge) retry twice then send to a dead-letter queue or on-failure destination, and poll sources (SQS, Kinesis) retry the batch until success, DLQ redrive, or an age/attempt cap — which means correct error handling is three disciplines working together: make the handler idempotent so retries are safe, cap and route failures so a poison pill lands in a DLQ instead of blocking forever, and bisect batches so one bad record doesn't sink its neighbours. Skip any one and an at-least-once world turns a transient blip into duplicated rows or a wedged partition.

Retry behaviour by invocation type.

• Synchronous (API Gateway, direct invoke). No automatic Lambda retry — the caller decides whether to retry. Errors surface to the caller immediately.
• Asynchronous (S3, SNS, EventBridge). Lambda's internal queue retries a failed invocation up to 2 more times (configurable) with delay, then routes the event to an on-failure destination (SQS/SNS/EventBridge/Lambda) or a legacy DLQ. The event is not lost — it is parked.
• Poll-based (SQS, Kinesis, DynamoDB Streams). The event source mapping retries the batch. For SQS, the queue's redrive policy sends a message to a DLQ after maxReceiveCount. For streams, maximumRetryAttempts and maximumRecordAgeInSeconds cap retries and an on-failure destination captures the failed batch metadata.
• The consequence. Because retries are guaranteed, the same event will be processed more than once at some point. Idempotency is not optional.

Dead-letter queues and destinations.

• DLQ (legacy). An SQS queue or SNS topic attached to the function for async invocations; failed events land there after retries. Still supported but destinations are richer.
• Lambda destinations. For async invocations, configure on-success and on-failure destinations that receive the full invocation record (request + response/error) — better than a bare DLQ because you get context.
• SQS redrive. For SQS sources, the DLQ is a property of the queue (RedrivePolicy → deadLetterTargetArn), not the function. After maxReceiveCount failed receives, the message moves to the DLQ.
• The DLQ is not a graveyard. It needs an alarm (messages > 0) and a redrive/replay path once the bug is fixed. An unwatched DLQ silently loses data to expiry.

Idempotency — the correctness backbone.

• Why. At-least-once + retries = duplicates. A non-idempotent write (blind INSERT, += amount) double-applies on retry; an idempotent write is a no-op the second time.
• Techniques. Deterministic keys + upsert (ON CONFLICT DO NOTHING/UPDATE); a dedupe/idempotency table with a conditional write (PutItem with attribute_not_exists); content-addressed output keys (write to s3://.../{hash}); dedupe on a natural business key (event_id).
• The idempotency table. A DynamoDB table keyed by idempotency_key with a TTL; the handler conditionally records "processed this key," and on a duplicate the conditional write fails, so the handler skips the side effect.
• Scope carefully. Idempotency must cover the side effect, not just the record. "Recorded processed" and "did the work" should commit together (or be ordered so a crash between them is recoverable).

Poison-pill handling on streams.

• Bisect on error. bisectBatchOnFunctionError: true splits a failing batch in half and retries each half, narrowing down to the single bad record instead of failing the whole batch repeatedly.
• Caps. maximumRetryAttempts and maximumRecordAgeInSeconds bound how long a bad record blocks the shard.
• On-failure destination. After caps, the failed batch's metadata (not the payload) goes to an SQS/SNS destination so you can investigate; the shard advances.
• Why it matters. Without these, one un-processable record blocks its Kinesis shard forever, halting all downstream processing for that partition.

Common interview probes on error handling.

• "What happens when an S3-triggered Lambda fails?" — async retry ×2, then on-failure destination / DLQ.
• "How do you make an ETL handler idempotent?" — deterministic key + upsert, or a conditional-write dedupe table.
• "How do you stop one bad Kinesis record blocking a shard?" — bisect on error + retry/age caps + on-failure destination.
• "Where does an SQS DLQ get configured?" — on the queue's redrive policy, after maxReceiveCount.

Worked example — async S3 invocation with retries and an on-failure destination

Detailed explanation. An S3-triggered (async) Lambda occasionally fails on a transient downstream blip. Lambda retries twice automatically; if it still fails, the event should land in an on-failure destination for investigation rather than vanish. Configure retries and the destination.

• Retries. maximumRetryAttempts: 2 (the async default) with maximumEventAgeInSeconds cap.
• Destination. On-failure → an SQS DLQ that carries the full invocation record.
• Alarm. CloudWatch alarm on DLQ depth > 0.

Question. Configure the async retry policy and on-failure destination for an S3-triggered transform.

Input.

Setting	Value
Invocation type	asynchronous (S3)
Max retry attempts	2
Max event age	3600 s
On-failure destination	transform-dlq (SQS)

Code.

# Async invocation config + destination
aws lambda put-function-event-invoke-config \
  --function-name csv-to-parquet \
  --maximum-retry-attempts 2 \
  --maximum-event-age-in-seconds 3600 \
  --destination-config '{
    "OnFailure": {"Destination": "arn:aws:sqs:...:transform-dlq"}
  }'

# handler.py — raise on real failure so retries + destination engage
def handler(event, context):
    for rec in event["Records"]:
        try:
            transform(rec)
        except TransientError:
            raise                     # let Lambda retry (×2), then -> on-failure dest
        except PermanentError as e:
            # unrecoverable: record context and DO NOT retry uselessly
            park_to_quarantine(rec, reason=str(e))
    return {"ok": True}

Step-by-step explanation.

1. For async sources like S3, Lambda manages retries internally. maximum-retry-attempts 2 means an initial attempt plus two retries (three total) before the event is considered failed and routed to the on-failure destination.
2. The handler raises on a transient error so Lambda's retry machinery engages — swallowing the exception would mark the invocation successful and skip retries entirely. Raising is the signal that says "please try again."
3. After retries are exhausted (or maximum-event-age passes), the full invocation record — event plus error — lands in transform-dlq. Unlike a bare DLQ, a destination carries the response/error context, so you know why it failed.
4. For a permanent error (malformed record that will never succeed), the handler parks it to a quarantine store immediately rather than burning three attempts on a hopeless case. Distinguishing transient from permanent avoids wasted retries.
5. A CloudWatch alarm on transform-dlq depth > 0 turns the destination into an actionable signal. Without the alarm, failures accumulate silently and expire — the DLQ becomes a data-loss trap, not a safety net.

Output.

Attempt	Outcome	Next
1	TransientError	retry
2	TransientError	retry
3	TransientError	→ on-failure destination
any	PermanentError	quarantined immediately
success	done	event acknowledged

Rule of thumb. For async (S3/SNS) Lambdas, set explicit retry attempts and an on-failure destination (richer than a bare DLQ), raise on transient errors so retries engage, quarantine permanent errors without wasteful retries, and alarm on DLQ depth. A failure must always have somewhere to go and someone to notice.

Worked example — idempotent S3-to-warehouse upsert with a dedupe table

Detailed explanation. An S3-triggered Lambda loads a file's rows into a warehouse. Because delivery is at-least-once, the same file can be processed twice, double-loading rows. A DynamoDB idempotency table keyed by the object's eTag records "this exact object version was loaded," and a conditional write makes the second attempt a no-op. Build it.

• Key. idempotency_key = bucket/key@eTag — unique per object version.
• Guard. PutItem with attribute_not_exists(pk); success = first time, ConditionalCheckFailed = duplicate.
• TTL. Expire keys after, say, 30 days.

Question. Make the S3-to-warehouse load idempotent so duplicate deliveries don't double-load.

Input.

Component	Value
Idempotency store	DynamoDB etl_idempotency
Key	bucket/key@eTag
Guard	conditional put (attribute_not_exists)
TTL	30 days

Code.

# idempotent_load.py
import boto3, time
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
TABLE = "etl_idempotency"

def already_done(key: str) -> bool:
    try:
        ddb.put_item(
            TableName=TABLE,
            Item={"pk": {"S": key}, "ttl": {"N": str(int(time.time()) + 30*86400)}},
            ConditionExpression="attribute_not_exists(pk)",   # only if new
        )
        return False                # we just claimed it -> first time
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return True             # someone already claimed it -> duplicate
        raise

def handler(event, context):
    for rec in event["Records"]:
        b   = rec["s3"]["bucket"]["name"]
        k   = rec["s3"]["object"]["key"]
        tag = rec["s3"]["object"]["eTag"]
        key = f"{b}/{k}@{tag}"
        if already_done(key):
            print(f"skip duplicate {key}")
            continue
        rows = read_and_transform(b, k)
        load_to_warehouse(rows)     # the side effect, guarded by the claim above
    return {"ok": True}

Step-by-step explanation.

1. The idempotency key includes the eTag, so it identifies a specific version of the object. Re-uploading changed content produces a new eTag (correctly reprocessed); a duplicate notification for the same bytes produces the same key (correctly skipped).
2. put_item with ConditionExpression="attribute_not_exists(pk)" atomically claims the key. On the first delivery the put succeeds and returns False (not done). On a duplicate the condition fails with ConditionalCheckFailedException, returning True (already done) — a single atomic operation, no read-then-write race.
3. Only after successfully claiming the key does the handler perform the side effect (load to warehouse). The claim gates the work, so two concurrent duplicate deliveries can't both load — exactly one wins the conditional put.
4. The ttl attribute lets DynamoDB expire old keys automatically, bounding the table's size. Thirty days comfortably exceeds any redelivery window while keeping storage flat.
5. One subtlety: if the claim succeeds but the load then crashes, the key is marked done while the work isn't. For strict correctness, either make the load itself idempotent (upsert) as a second layer, or record the claim after the load with an in-progress marker — belt and braces for critical pipelines.

Output.

Delivery	Key	Conditional put	Action
1st of file A	.../a@etag1	succeeds	load rows
duplicate of A	.../a@etag1	fails	skip
A re-uploaded	.../a@etag2	succeeds	load new version
concurrent dup	.../a@etag1	one wins	one loads, one skips

Rule of thumb. Guard every non-idempotent side effect with an atomic conditional write on a deterministic idempotency key (object version, event_id, or content hash), TTL the dedupe table, and layer an upsert underneath for the crash-after-claim edge. In an at-least-once world, the dedupe key is as important as the transform itself.

Worked example — Kinesis bisect-on-error to isolate a poison record

Detailed explanation. A single malformed record in a Kinesis batch makes the handler throw; the mapping retries the whole batch, fails again, and blocks the shard. bisectBatchOnFunctionError splits the batch to isolate the bad record; retry/age caps and an on-failure destination bound the damage. Configure it.

• Bisect. On error, split the batch and retry halves — narrows to the single bad record.
• Caps. maximumRetryAttempts and maximumRecordAgeInSeconds stop infinite blocking.
• Destination. On-failure → SQS with the failed batch metadata; shard advances.

Question. Configure the Kinesis event source mapping to isolate poison records and keep the shard moving.

Input.

Setting	Value
bisectBatchOnFunctionError	true
maximumRetryAttempts	5
maximumRecordAgeInSeconds	3600
on-failure destination	kinesis-failures (SQS)
functionResponseTypes	ReportBatchItemFailures

Code.

# Kinesis event source mapping (poison-pill safe)
aws lambda create-event-source-mapping \
  --function-name clickstream-consumer \
  --event-source-arn arn:aws:kinesis:...:stream/clickstream \
  --batch-size 500 \
  --maximum-batching-window-in-seconds 5 \
  --parallelization-factor 4 \
  --bisect-batch-on-function-error \
  --maximum-retry-attempts 5 \
  --maximum-record-age-in-seconds 3600 \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:...:kinesis-failures"}}'

# consumer.py — report the exact failing record; let bisect narrow it
import base64, json

def handler(event, context):
    failures = []
    for rec in event["Records"]:
        seq = rec["kinesis"]["sequenceNumber"]
        try:
            data = json.loads(base64.b64decode(rec["kinesis"]["data"]))
            transform(data)
        except Exception as e:
            print(f"bad record seq={seq}: {e}")
            failures.append({"itemIdentifier": seq})
    return {"batchItemFailures": failures}

Step-by-step explanation.

1. With ReportBatchItemFailures, the handler returns the exact failing sequenceNumbers, so the mapping retries only from the first failure — successes before it are checkpointed and never reprocessed.
2. If the whole invocation throws instead (or the reported failure persists), bisectBatchOnFunctionError splits the 500-record batch into two 250s and retries each. Repeated bisection homes in on the single poison record in ~log₂(500) ≈ 9 splits, instead of failing all 500 forever.
3. maximumRetryAttempts 5 and maximumRecordAgeInSeconds 3600 bound the blocking: after 5 attempts or once the record is an hour old, the mapping gives up on it rather than blocking the shard indefinitely.
4. When a record is finally abandoned, its batch metadata (shard, sequence range) is sent to the kinesis-failures SQS destination — not the payload, which you re-read from the stream if needed — and the shard iterator advances past it. Downstream processing resumes.
5. The combination is what keeps a stream healthy: partial failure reporting minimises reprocessing, bisection isolates the culprit, caps bound the blast radius in time, and the destination preserves a breadcrumb for investigation. Miss any one and a single bad record halts a partition.

Output.

Mechanism	Without it	With it
Report failures	whole batch reread	only tail reread
Bisect on error	500 fail forever	narrows to 1 record
Retry/age caps	shard blocked forever	bounded (5 tries / 1 h)
On-failure destination	no breadcrumb	metadata captured
Net	shard wedged	shard keeps moving

Rule of thumb. For Kinesis/DynamoDB-Streams ETL, always set bisectBatchOnFunctionError, retry and record-age caps, ReportBatchItemFailures, and an on-failure destination. A stream without poison-pill handling is one malformed record away from a silently stalled partition.

Data engineering interview question on idempotency and DLQs

A senior interviewer might ask: "Your SQS-triggered Lambda credits customer wallets from a payments event stream. Because delivery is at-least-once, you occasionally double-credit a wallet, and one malformed event retries endlessly. Design the handler and infrastructure so each event is applied exactly once in effect, and the poison event is quarantined without blocking the queue."

Solution Using an idempotency table with a conditional write plus a DLQ redrive

# wallet_credit.py — effect-once wallet credit under at-least-once delivery
import boto3, json, time
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
IDEMP = "wallet_idempotency"

def claim(event_id: str) -> bool:
    """Return True if we are the first to process this event_id."""
    try:
        ddb.put_item(
            TableName=IDEMP,
            Item={"pk": {"S": event_id}, "ttl": {"N": str(int(time.time()) + 30*86400)}},
            ConditionExpression="attribute_not_exists(pk)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        mid = msg["messageId"]
        try:
            body = json.loads(msg["body"])
            event_id = body["event_id"]
            if not claim(event_id):
                continue                          # duplicate -> effect-once skip
            credit_wallet(body["wallet_id"], body["amount_cents"])   # side effect
        except KeyError as e:
            failures.append({"itemIdentifier": mid})   # malformed -> retry -> DLQ
        except Exception:
            failures.append({"itemIdentifier": mid})   # transient -> retry
    return {"batchItemFailures": failures}

# Infra
SQS wallet-events:
  VisibilityTimeout   = 180
  RedrivePolicy       = { maxReceiveCount: 5, deadLetterTargetArn: wallet-events-dlq }
Event source mapping: FunctionResponseTypes = [ReportBatchItemFailures], BatchSize = 10
DynamoDB wallet_idempotency: pk = event_id, TTL enabled
CloudWatch alarm: wallet-events-dlq ApproximateNumberOfMessagesVisible > 0

Step-by-step trace.

Step	Input	Behaviour
1	event_id=E1, credit $5	claim(E1) succeeds → credit applied
2	duplicate event_id=E1	claim(E1) fails → skip (no double-credit)
3	event_id=E2, malformed (no wallet_id)	KeyError → reported failure
4	E2 retried 5×	still malformed → redrive to DLQ
5	transient DB blip on E3	reported failure → retried, then succeeds
6	DLQ depth > 0	CloudWatch alarm pages on-call

Walking the trace: the claim conditional put is the exactly-once effect gate — the first delivery of E1 claims the key and credits the wallet; every subsequent delivery of E1 fails the conditional write and skips the credit, so the wallet is credited once no matter how many times SQS delivers the message. The malformed E2 is reported (not re-raised), so only it is retried; after maxReceiveCount = 5 it moves to the DLQ, quarantined, while healthy messages flow. A transient blip on E3 is retried and eventually succeeds because the claim + credit are safe to repeat until they stick. The DLQ alarm makes the parked poison event visible to a human.

Output:

Event	Deliveries	Credits applied	Final location
E1 ($5)	3 (at-least-once)	1	done
E2 (malformed)	5	0	DLQ
E3 (transient)	2	1	done
wallet balance	—	+$10 (E1+E3)	correct

Why this works — concept by concept:

• Conditional-write claim — the atomic PutItem with attribute_not_exists is the exactly-once effect primitive: only the first delivery claims the event_id, so the credit runs once even though delivery is at-least-once.
• Effect-once, not delivery-once — you cannot make SQS deliver exactly once, so you make the side effect idempotent. The claim gates the credit; duplicates become no-ops.
• Partial batch response — reporting only failed IDs means a malformed event retries alone and never forces healthy credits to reprocess.
• DLQ redrive + alarm — bounding the malformed event to five attempts parks it in a DLQ instead of blocking the queue, and the alarm turns "parked" into "noticed."
• Cost — one O(1) conditional write per event plus the credit; duplicates short-circuit before the side effect. Compared with a non-idempotent handler, financial correctness is guaranteed at the cost of a single small DynamoDB write per unique event.

Event
Topic — event-processing
Event-processing problems on exactly-once effects

Practice →

Data Topic — data-processing Data-processing problems on dedupe and retries

Practice →

---

5. Patterns & limits — fan-out, Step Functions handoff

Fan-out to scale horizontally, hand off at the fifteen-minute wall — the two moves that take Lambda past its limits

The mental model in one line: Lambda's two structural limits — a single invocation is time-boxed to fifteen minutes and processes one event's worth of work — are overcome by two complementary patterns: fan-out, where one event is sprayed to many parallel Lambdas (via SNS, or a Map over a manifest) so a big workload becomes thousands of small independent ones, and handoff, where a job too long or too multi-step for one invocation is escalated to Step Functions (or its Distributed Map) which orchestrates many Lambda steps with per-step retries, branching, and no overall time cap — and knowing which to reach for, plus the hard limits (payload size, /tmp, deployment package) that bound each, is the senior signal. Fan-out scales the width; Step Functions scales the length and coordination.

Fan-out patterns — one event, many workers.

• SNS fan-out. Publish one event to an SNS topic; many Lambdas (and SQS queues) subscribe and each gets a copy. Decouples producer from N independent consumers — the classic pub/sub fan-out.
• SQS fan-out with a splitter. A Lambda reads a manifest and enqueues one SQS message per work item; a fleet of consumers drains the queue with bounded concurrency. Good for back-pressure-controlled fan-out.
• Map-style parallel fan-out. Step Functions Map (inline) or Distributed Map iterates a collection (e.g. every object under an S3 prefix, every line of a manifest) and runs one Lambda per item, with configurable max concurrency.
• When to fan out. When the workload is a large set of independent small units (thousands of files, partitions, or shards) — fan-out turns O(N) serial work into O(N / concurrency) wall-clock.

The fifteen-minute wall and the Step Functions handoff.

• The wall. No single invocation runs past fifteen minutes. A backfill, a multi-hour export, or a chained multi-step job cannot live in one Lambda.
• The handoff. Escalate to a Step Functions state machine that calls Lambda for each short step and orchestrates the long-running whole — retries per state, Wait/Choice/Parallel states, and a total runtime up to a year (standard workflows).
• Chunking under orchestration. A long job becomes many short Lambda steps (process chunk, checkpoint, loop) coordinated by Step Functions, each step comfortably under fifteen minutes.
• When to hand off. Long duration, ordered multi-step pipelines, human-approval waits, or anything needing per-step retry/branch logic that would be ugly to hand-roll inside one Lambda.

Distributed Map — large-scale S3 fan-out.

• What it is. A Step Functions state that reads a large dataset (millions of S3 objects or CSV/JSON lines) and runs a child workflow / Lambda per item at very high concurrency (up to ~10 000), with batching and tolerated-failure thresholds.
• Why not plain Map. Inline Map is limited (bounded array, lower concurrency, in-state payload). Distributed Map is built for millions of items and offloads iteration to the service.
• ETL use. The go-to for "reprocess every object in this prefix" backfills — it enumerates the objects, fans out, bounds concurrency to protect downstreams, and aggregates results.

Hard limits to remember.

• Payload. Sync invocation request/response ≤ 6 MB; async event ≤ 256 KB. Big payloads go by reference (an S3 key), never inline.
• /tmp ephemeral storage. 512 MB by default, configurable up to 10 GB. Scratch space for a single invocation; not shared or durable.
• Deployment package. 50 MB zipped / 250 MB unzipped for zip; up to 10 GB for container images. Big dependencies push you to container images or layers.
• Timeout & memory. 15 min max timeout; 128 MB–10 GB memory. Environment variables ≤ 4 KB total.

Common interview probes on patterns and limits.

• "How do you process 100 000 S3 objects with Lambda?" — fan-out via Step Functions Distributed Map with bounded concurrency.
• "A job takes 40 minutes — what do you do?" — hand off to Step Functions; chunk into sub-15-minute Lambda steps.
• "How do you pass a large payload between steps?" — by reference (S3 key), not inline; mind the 6 MB / 256 KB limits.
• "SNS vs SQS for fan-out?" — SNS for pub/sub to N independent consumers; SQS for buffered, back-pressured fan-out to a worker pool.

Worked example — SNS fan-out to independent consumers

Detailed explanation. An order event must trigger three independent things: update the warehouse, refresh a search index, and notify a fraud service. Coupling them in one Lambda makes one slow consumer block the others. SNS fan-out publishes once and lets three Lambdas each react independently. Build it.

• Topic. order-events SNS topic.
• Subscribers. Three Lambdas (warehouse, search, fraud), each with its own retry/DLQ.
• Isolation. A failure in one consumer doesn't affect the others.

Question. Wire an SNS fan-out so one order event drives three independent consumers.

Input.

Component	Value
Topic	order-events (SNS)
Consumer 1	warehouse-loader Lambda
Consumer 2	search-indexer Lambda
Consumer 3	fraud-scorer Lambda
Delivery	each gets its own copy

Code.

# producer.py — publish ONCE; SNS fans out to all subscribers
import boto3, json
sns = boto3.client("sns")
TOPIC = "arn:aws:sns:...:order-events"

def on_order(order):
    sns.publish(
        TopicArn=TOPIC,
        Message=json.dumps(order),
        MessageAttributes={
            "event_type": {"DataType": "String", "StringValue": "OrderPlaced"}
        },
    )

# warehouse_loader.py — one of three independent subscribers
import json
def handler(event, context):
    for record in event["Records"]:                 # SNS -> Lambda event shape
        order = json.loads(record["Sns"]["Message"])
        upsert_order(order)                          # idempotent
    return {"ok": True}
# search_indexer.py and fraud_scorer.py subscribe to the same topic independently.

Step-by-step explanation.

1. The producer publishes the order once to the SNS topic. SNS delivers an independent copy to each subscribed Lambda, so the producer neither knows nor cares how many consumers exist — new consumers subscribe without touching the producer.
2. Each subscriber Lambda has its own retry policy and on-failure destination/DLQ. If search-indexer is failing, warehouse-loader and fraud-scorer proceed unaffected — the failure is isolated to one branch.
3. SNS→Lambda is asynchronous, so each consumer gets async retry semantics (retry then DLQ). Each consumer must still be idempotent because SNS delivery is at-least-once.
4. MessageAttributes allow subscription filter policies — a consumer can subscribe only to event_type = OrderPlaced and ignore other event types on the same topic, filtering at SNS instead of in code.
5. For consumers that need buffering/back-pressure rather than direct invocation, subscribe an SQS queue to the topic and let a worker pool drain it — SNS fan-out and SQS buffering compose. This is the standard "fan-out then back-pressure" shape.

Output.

Consumer	Gets event?	Failure isolated?	Retry/DLQ
warehouse-loader	yes	yes	own
search-indexer	yes	yes	own
fraud-scorer	yes	yes	own
producer	publishes once	N/A	N/A

Rule of thumb. For one-event-to-many-independent-reactions, publish once to SNS and let each consumer subscribe with its own retry/DLQ and filter policy. Add an SQS queue between SNS and a consumer when that consumer needs buffering. Never couple independent consumers into one Lambda — one slow branch shouldn't block the rest.

Worked example — Step Functions handoff for a multi-hour backfill

Detailed explanation. A backfill must reprocess a month of data — far more than fifteen minutes of work. A single Lambda can't do it, so a Step Functions state machine drives a loop: each iteration invokes a Lambda that processes one day (well under fifteen minutes), checkpoints progress, and the state machine loops until done. Build the workflow.

• Structure. A Choice/Map loop over days; each day is one Lambda step.
• Checkpointing. Each step records the last completed day so a failure resumes, not restarts.
• No time cap. The state machine can run for hours/days (standard workflow) while each step stays short.

Question. Design a Step Functions workflow that backfills a month of data via short Lambda steps.

Input.

Component	Value
Workflow type	Standard (long-running)
Step	process-day Lambda (≤ 15 min)
Iteration	one per day, 30 days
Checkpoint	last completed day in DynamoDB

Code.

// backfill.asl.json — Step Functions state machine (simplified)
{
  "StartAt": "NextDay",
  "States": {
    "NextDay": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:function:process-day",
      "Retry": [
        {"ErrorEquals": ["TransientError"], "MaxAttempts": 4, "BackoffRate": 2.0, "IntervalSeconds": 10}
      ],
      "Catch": [
        {"ErrorEquals": ["States.ALL"], "Next": "ParkFailure"}
      ],
      "Next": "MoreDays?"
    },
    "MoreDays?": {
      "Type": "Choice",
      "Choices": [
        {"Variable": "$.done", "BooleanEquals": false, "Next": "NextDay"}
      ],
      "Default": "Succeed"
    },
    "ParkFailure": {"Type": "Task", "Resource": "arn:aws:lambda:...:function:park-failure", "Next": "Fail"},
    "Fail":    {"Type": "Fail"},
    "Succeed": {"Type": "Succeed"}
  }
}

# process_day.py — one short step; returns whether more days remain
import boto3
ddb = boto3.client("dynamodb")

def handler(event, context):
    day = next_unprocessed_day(event)      # read checkpoint
    process_one_day(day)                   # << 15 minutes of work
    checkpoint(day)                        # durable progress
    remaining = days_left(day)
    return {"last_day": day, "done": remaining == 0}

Step-by-step explanation.

1. Step Functions owns the loop and the clock. Each NextDay task invokes process-day for a single day's data — a unit deliberately sized to finish well under fifteen minutes — and returns done: false until the last day.
2. The MoreDays? Choice state loops back to NextDay while work remains. The overall backfill can run for hours across 30 iterations, but no individual Lambda ever approaches the time wall.
3. Per-step Retry with exponential backoff handles transient errors at the orchestration layer, so the Lambda stays simple. A permanent failure is caught and routed to ParkFailure, preserving context.
4. Each step checkpoints the last completed day durably. If the workflow fails and is restarted, next_unprocessed_day resumes from the checkpoint rather than reprocessing from the start — the backfill is restartable.
5. This is the canonical "job too long for Lambda" answer: don't fight the fifteen-minute wall, decompose the job into short idempotent steps and let Step Functions orchestrate duration, retries, and branching.

Output.

Iteration	Day processed	done	Next state
1	2026-08-01	false	NextDay
2	2026-08-02	false	NextDay
…	…	false	NextDay
30	2026-08-30	true	Succeed
on error	(parked)	—	ParkFailure → Fail

Rule of thumb. When a job exceeds fifteen minutes, hand it to Step Functions and decompose it into short, idempotent, checkpointed Lambda steps. Put retries and branching in the state machine, keep each step small, and make progress durable so a restart resumes. Orchestration beats fighting the time wall.

Worked example — Distributed Map over an S3 manifest

Detailed explanation. A backfill must reprocess 120 000 S3 objects. A serial loop would take hours and a single Lambda can't hold the list. Step Functions Distributed Map enumerates the objects and runs one Lambda per object at high, bounded concurrency, tolerating a small failure rate. Build it.

• Source. An S3 prefix (or a manifest file) of 120 000 objects.
• Concurrency. Max 500 concurrent child executions (bounded to protect downstreams).
• Tolerated failure. Continue if < 1% of items fail; collect failures.

Question. Configure a Distributed Map to reprocess 120 000 objects with bounded concurrency and a failure threshold.

Input.

Setting	Value
Item source	S3 prefix raw/2026/
Max concurrency	500
Tolerated failure %	1
Per-item worker	reprocess-object Lambda

Code.

// distributed_map.asl.json — Distributed Map over an S3 prefix
{
  "StartAt": "ReprocessAll",
  "States": {
    "ReprocessAll": {
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {"Bucket": "data-lake-prod", "Prefix": "raw/2026/"}
      },
      "ItemProcessor": {
        "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
        "StartAt": "Reprocess",
        "States": {
          "Reprocess": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:...:function:reprocess-object",
            "End": true
          }
        }
      },
      "MaxConcurrency": 500,
      "ToleratedFailurePercentage": 1,
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {"Bucket": "data-lake-prod", "Prefix": "backfill-results/"}
      },
      "End": true
    }
  }
}

# reprocess_object.py — one object per invocation, idempotent
def handler(event, context):
    bucket = event["Bucket"]
    key    = event["Key"]                 # provided per item by the Map
    reprocess(bucket, key)                # streaming, constant memory, idempotent
    return {"key": key, "status": "ok"}

Step-by-step explanation.

1. The ItemReader with s3:listObjectsV2 makes Step Functions enumerate the 120 000 objects itself — the list never lives inside a Lambda, sidestepping payload and memory limits.
2. Mode: DISTRIBUTED runs each item as an independent child execution, so the Map scales to very high item counts (millions) that inline Map cannot handle.
3. MaxConcurrency: 500 bounds how many reprocess-object Lambdas run at once — protecting downstreams (warehouse, RDS) exactly like reserved concurrency did in section 3, but at the orchestration layer.
4. ToleratedFailurePercentage: 1 lets the backfill continue even if a handful of objects fail, rather than aborting the whole 120 000-object job on the first bad file. Failures are recorded; you fix and rerun just those.
5. The ResultWriter aggregates per-item results to S3, giving a durable manifest of successes/failures for audit and targeted retry. Each worker processes exactly one object idempotently, so a retried item is safe. This is the production answer to "reprocess everything in this prefix."

Output.

Aspect	Value
Objects enumerated	120,000 (by the service)
Peak concurrency	500 workers
Wall-clock	~ (120k × 15 s) / 500 ≈ 1 hour
Tolerated failures	up to 1,200 (1%) before abort
Results	written to backfill-results/

Rule of thumb. For massive S3 fan-out, use Step Functions Distributed Map: let the service enumerate items, bound MaxConcurrency to protect downstreams, set a ToleratedFailurePercentage so a few bad objects don't abort the run, and write results for audit. It turns a many-hour serial backfill into a bounded-concurrency parallel one — the modern replacement for hand-rolled fan-out.

Data engineering interview question on scaling past Lambda limits

A senior interviewer might ask: "You need to reprocess 2 million small JSON files (each ~1 MB) from an S3 prefix into a partitioned Parquet table, protecting a downstream warehouse that tolerates ~300 concurrent writers, and the whole job must be restartable and take hours if needed. A single Lambda obviously can't do it. Design the solution and justify each choice against Lambda's limits."

Solution Using a Step Functions Distributed Map handoff with bounded concurrency

// reprocess_2m.asl.json — Distributed Map, bounded, restartable
{
  "StartAt": "Backfill",
  "States": {
    "Backfill": {
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {"Bucket": "lake", "Prefix": "raw/json/"}
      },
      "ItemBatcher": {"MaxItemsPerBatch": 20},
      "ItemProcessor": {
        "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
        "StartAt": "ToParquet",
        "States": {
          "ToParquet": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:...:function:json-to-parquet",
            "Retry": [{"ErrorEquals": ["TransientError"], "MaxAttempts": 4, "BackoffRate": 2.0}],
            "End": true
          }
        }
      },
      "MaxConcurrency": 300,
      "ToleratedFailurePercentage": 1,
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {"Bucket": "lake", "Prefix": "backfill-manifests/"}
      },
      "End": true
    }
  }
}

# json_to_parquet.py — process a small BATCH of objects per invocation
import boto3, io, json
import pyarrow as pa, pyarrow.parquet as pq
s3 = boto3.client("s3")

def handler(event, context):
    # ItemBatcher hands us up to 20 items -> fewer invocations, still short
    for item in event["Items"]:
        b, k = item["Bucket"], item["Key"]
        rows = [json.loads(l) for l in s3.get_object(Bucket=b, Key=k)["Body"].iter_lines()]
        part = "curated/" + partition_path(rows) + "/" + k.split("/")[-1] + ".parquet"
        buf = io.BytesIO(); pq.write_table(pa.Table.from_pylist(rows), buf); buf.seek(0)
        s3.put_object(Bucket=b, Key=part, Body=buf.getvalue())   # deterministic key = idempotent
    return {"ok": True}

Step-by-step trace.

Step	Choice	Justification vs Lambda limit
1	Distributed Map enumerates 2M keys	list never enters a Lambda (payload/memory limit)
2	ItemBatcher 20/invocation	20 × ~1 MB × short = well under 15 min & 10 GB
3	MaxConcurrency 300	≤ warehouse's 300-writer budget (back-pressure)
4	deterministic Parquet key	at-least-once safe (idempotent overwrite)
5	ToleratedFailurePercentage 1	a few bad files don't abort 2M-file run
6	ResultWriter manifest	restartable: rerun only failed items

Walking the trace: the 2-million-object list is enumerated by Step Functions, never materialised in a Lambda, so the payload and memory limits are irrelevant. ItemBatcher groups 20 small objects per invocation, keeping each invocation short (seconds) and small (tens of MB) — comfortably inside the fifteen-minute and 10 GB walls. MaxConcurrency 300 caps parallel writers at the warehouse's tolerance, applying back-pressure exactly where section 3 applied reserved concurrency, but at the orchestration layer. Deterministic output keys make each object's write idempotent, so retries and at-least-once execution never duplicate rows. The tolerated-failure threshold keeps a handful of malformed files from aborting the whole run, and the ResultWriter manifest records every outcome so a rerun targets only the failures — the restartable requirement satisfied.

Output:

Requirement	Mechanism	Met?
2M files, single Lambda can't	Distributed Map fan-out	yes
≤ 300 concurrent writers	MaxConcurrency 300	yes
Restartable / hours OK	Standard workflow + manifest	yes
At-least-once safe	deterministic keys	yes
A few bad files tolerated	ToleratedFailurePercentage 1	yes

Why this works — concept by concept:

• Distributed Map fan-out — the service enumerates and iterates millions of items, so no Lambda ever holds the list; the payload/memory limits that would sink a hand-rolled splitter simply don't apply.
• Item batching for right-sized units — grouping 20 small objects per invocation keeps each unit short and small, respecting the fifteen-minute and 10 GB walls while cutting invocation count 20×.
• Bounded MaxConcurrency as back-pressure — capping concurrent workers at the downstream's tolerance protects the warehouse, the orchestration-layer analogue of reserved concurrency.
• Deterministic keys for idempotency — content-addressed output keys make every write a safe overwrite, so the guaranteed retries of an at-least-once system never duplicate data.
• Cost — O(N / concurrency) wall-clock at O(concurrency) parallelism, with a result manifest enabling O(failures) targeted reruns instead of reprocessing all N. Compared with a single hand-rolled fan-out Lambda, it removes the payload, memory, and time walls entirely and makes the backfill restartable by construction.

Event
Topic — event-processing
Event-processing problems on fan-out and orchestration

Practice →

ETL
Topic — etl
ETL problems on large-scale backfills

Practice →

---

Cheat sheet — AWS Lambda ETL recipes

• Fit / no-fit rule. Lambda fits per-event ETL when a unit of work finishes in well under 15 minutes and holds under 10 GB memory, is stateless (or externalises state to DynamoDB), and the event rate sits below the cost crossover with Glue/EMR. Score every job on duration × per-unit volume × unit count × cross-event state; the two hard walls disqualify a unit outright, and huge unit counts mean fan-out, not a cluster.
• Push vs poll triggers. S3/SNS/EventBridge push async (retry ×2 → DLQ/destination, one event each, no batch knob). SQS/Kinesis/DynamoDB-Streams are polled by the event source mapping which batches and invokes sync — tune batchSize, maximumBatchingWindowInSeconds, and failure handling on the mapping.
• S3 trigger config. Scope with prefix + suffix filters; include both ObjectCreated:Put and ObjectCreated:CompleteMultipartUpload (large uploads fire the latter); loop over all event["Records"]; derive output keys deterministically so at-least-once re-delivery overwrites idempotently.
• SQS ESM knobs. batchSize (≤10 typical), maximumBatchingWindowInSeconds, visibility timeout ≥ 6× function timeout, FunctionResponseTypes=[ReportBatchItemFailures] (return only failed IDs — never re-raise), and a redrive policy (maxReceiveCount → DLQ). Cap load with ScalingConfig.MaximumConcurrency.
• Kinesis ESM knobs. batchSize per shard, parallelizationFactor up to 10 (more throughput, per-partition-key order preserved), TRIM_HORIZON vs LATEST, plus poison-pill guards: bisectBatchOnFunctionError, maximumRetryAttempts, maximumRecordAgeInSeconds, on-failure destination. Order is a partition-key property, per shard.
• Concurrency types. Unreserved = shared regional pool (throttles with 429). Reserved = per-function cap that both guarantees and limits slots (protects downstreams). Provisioned = pre-warmed sandboxes that remove cold starts (pay always; use for latency-sensitive sync, rarely batch). Steady concurrency ≈ rps × avg_duration_s.
• Cold-start fixes. Shrink the deployment package; lazy-import heavy libs; do one-time setup (DB clients, model loads) at init and reuse across warm invocations; avoid needless VPC attachment; use provisioned concurrency only where p99 is a user-facing SLO. For batch ETL, cold starts amortise away — don't pay to remove them.
• Memory = CPU (power tuning). CPU/network scale with the memory dial (~1 vCPU at ~1769 MB). Sweep memory, measure duration and GB-seconds, pick the minimum-cost point (often ~1 vCPU for CPU-bound work) or minimum-latency point. Never leave a CPU-bound job at 128 MB.
• Error handling stack. At-least-once ⇒ retries guaranteed ⇒ handlers must be idempotent. Async: maximum-retry-attempts + on-failure destination (richer than DLQ). SQS: redrive → DLQ. Streams: bisect + caps + on-failure destination. Alarm on every DLQ depth > 0 and build a redrive/replay path.
• Idempotency template. Deterministic key (bucket/key@eTag, event_id, or content hash) + atomic conditional write (PutItem with attribute_not_exists) to a TTL'd DynamoDB dedupe table; run the side effect only after claiming the key; layer an ON CONFLICT upsert underneath for the crash-after-claim edge. Make the effect exactly-once, not the delivery.
• Fan-out patterns. SNS for pub/sub to N independent consumers (each with its own retry/DLQ and filter policy). SQS splitter for buffered, back-pressured fan-out to a worker pool. Step Functions Map / Distributed Map for iterating a manifest or S3 prefix at bounded concurrency.
• Step Functions handoff. When a job exceeds 15 minutes or needs multi-step retry/branch logic, hand off to a Standard state machine that calls Lambda for short, idempotent, checkpointed steps. Use Distributed Map for million-object S3 backfills: service-side enumeration, MaxConcurrency back-pressure, ToleratedFailurePercentage, and a ResultWriter manifest for restartable reruns.
• Hard limits. Sync request/response ≤ 6 MB, async event ≤ 256 KB (pass big payloads by S3 reference); /tmp 512 MB–10 GB; zip package 50 MB/250 MB (containers up to 10 GB); env vars ≤ 4 KB; timeout ≤ 15 min; memory 128 MB–10 GB.
• Cost crossover. Lambda wins bursty and low-to-medium steady rates (nothing paid at idle, no cluster ops); a sustained firehose well above the compute crossover (cluster_hourly / lambda_cost_per_event) favours a provisioned engine. Always add "no cluster to run" to Lambda's side of the ledger.

Frequently asked questions

Is AWS Lambda good for ETL?

Yes, for the right shape of ETL. aws lambda etl shines on event-driven, per-event work — transform a file the moment it lands, enrich a message off a queue, route a streamed record — because it scales from zero to thousands of parallel invocations with no cluster to run and you pay only per invocation. It is a poor fit for a single long-running job that needs a big in-memory shuffle or must chew through terabytes end to end: those hit the 15-minute wall or the 10 GB memory ceiling and belong on Glue, EMR, or a Step Functions-orchestrated set of short Lambda steps. The rule of thumb is: Lambda for the trigger, transform, and fan-out; a distributed engine for the heavy batch in the middle.

What is the 15-minute Lambda limit and how do I work around it?

Every Lambda invocation is hard-capped at fifteen minutes of wall-clock time; when it's reached, the invocation is killed regardless of progress. You do not work around it by trying to make one invocation faster past a point — you decompose the job. Split the work into units that each finish comfortably under fifteen minutes (smaller files, record batches, per-day chunks), and orchestrate the whole with Step Functions, which has no overall time cap and calls Lambda for each short, idempotent, checkpointed step. For massive parallel work like reprocessing millions of S3 objects, Step Functions Distributed Map fans the units out at bounded concurrency. If a single indivisible unit genuinely can't fit, that unit belongs on Glue/EMR, not Lambda.

How do S3 triggers work with Lambda?

You attach an S3 event notification to a bucket so that object events (s3:ObjectCreated:*, deletes, restores) invoke your Lambda asynchronously, scoped by prefix and suffix filters so only the objects you care about fire it. Delivery is at-least-once and not strictly ordered, so the same object can occasionally trigger the function more than once — your handler must be idempotent (derive output keys deterministically or dedupe on the object eTag). Two gotchas: include ObjectCreated:CompleteMultipartUpload (large uploads fire that, not Put), and loop over every record in event["Records"] because a notification can carry more than one. Because it's an async source, failures retry twice and then route to an on-failure destination or DLQ.

What is the difference between reserved and provisioned concurrency?

Reserved concurrency is a per-function cap on the number of concurrent executions: it both guarantees the function at least that many slots and limits it to at most that many, which is how you protect a fragile downstream (cap below its connection budget) and stop one function from starving the account pool. Provisioned concurrency is different — it keeps a set number of execution environments pre-initialised and warm so those invocations skip the cold start entirely; you pay for them whether used or not. Use reserved concurrency to shape blast radius and back-pressure (nearly always in ETL), and provisioned concurrency only where cold-start tail latency is a user-facing SLO (rarely for pure batch ETL, where a cold start is a rounding error).

How do I make a Lambda ETL job idempotent?

Because delivery is at-least-once and retries are guaranteed, the same event will be processed more than once, so you make the effect idempotent rather than hoping for exactly-once delivery. The core techniques: derive deterministic output keys (content-addressed S3 objects, or bucket/key@eTag) so a re-run overwrites identical bytes; use upserts (INSERT ... ON CONFLICT DO NOTHING/UPDATE) instead of blind inserts; and for side effects that can't be made naturally idempotent, gate them with an atomic conditional write to a DynamoDB dedupe table keyed by event_id (PutItem with attribute_not_exists) — the first delivery claims the key and does the work, every duplicate fails the condition and skips it. TTL the dedupe table and, for critical paths, layer an upsert under the claim to cover a crash between claiming and doing the work.

When should I use Step Functions instead of Lambda?

Reach for Step Functions whenever the work is longer than fifteen minutes, has multiple steps that each need their own retry/backoff, must branch on results, waits for something (a human approval, a poll), or fans out over a huge collection. Step Functions is the orchestrator; Lambda is the worker it calls for each short step, so you get per-step retries, Choice/Parallel/Wait states, durable checkpointing, and no overall time cap — without hand-rolling that logic inside one function. For large-scale S3 ETL, Distributed Map iterates millions of objects at bounded concurrency with a tolerated-failure threshold and a result manifest. If your job is a single short per-event transform, plain Lambda is simpler and cheaper; the moment you need duration, coordination, or branching, hand off.

Practice on PipeCode

• Drill the ETL practice library → for the serverless ingestion, S3-trigger transform, batch-consumer, and large-scale backfill problems senior interviewers love.
• Rehearse on the event-processing practice library → for queue/stream consumers, partial batch response, fan-out, and exactly-once-effect scenarios.
• Sharpen the transform axis with the data-processing practice library → for streaming transforms, dedupe, and concurrency-tuning patterns.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the fit/no-fit decision and the event-driven design trade-offs against real graded inputs.

Lock in event-driven ETL muscle memory

Docs explain the services. PipeCode drills explain the decision — when Lambda fits ETL and when the 15-minute wall sends you to Step Functions, when partial batch response saves a queue, when reserved concurrency protects a database, when an idempotency key turns at-least-once into effect-once. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice event-processing problems →