A lending platform runs its loan approval workflow as a Lambda durable function (Lambda's checkpoint-and-replay execution mode — on every resume, the handler re-runs from the top and skips completed durable operations using their stored results). The flow: validate the application (a context.step()), pull the credit report (a step), then a review loop — context.wait() for 6 hours, check whether a human reviewer has approved, repeat for up to 5 days. Function timeout: 90 seconds per invocation. Execution timeout: 7 days.
At the top of the handler — before any step — the code downloads a 40 MB compliance ruleset from S3 (object storage) and parses it. It takes about 50 seconds. The engineer put it there deliberately: "several steps need it, so load it once."
The workflow ships. Applications flow through reviews for two days. Then, gradually, applications stop progressing. By Thursday, 1,900 applications are frozen mid-review. The business logic has thrown zero errors. The durable execution history shows executions alive and well within their 7-day window — just never advancing past their latest wait. The only anomaly: invocation duration on the function is pinned at exactly 90 seconds.
What is happening?
A) The executions exceeded the durable execution timeout — but the execution timeout (7 days here, up to 1 year) is a separate setting from the Lambda function timeout, and the history shows executions only 2–3 days old and still active
B) context.wait() keeps the function running and billing during the 6-hour waits, and the accumulated wait time consumed the timeout — but a wait suspends the execution entirely; each resume is a brand-new invocation with a fresh 90-second function timeout
C) The 50-second ruleset load sits outside any durable operation — so it is not checkpointed, and it re-executes on every single resume; each resume pays 50 seconds of S3 load plus replay of a growing checkpoint log before reaching any new work, and after enough review cycles a resume can no longer finish inside 90 seconds — it is killed mid-replay, retried, and killed again
D) The checkpoint log hit its per-execution size limit and Lambda silently stopped scheduling replays — but exceeding durable execution limits surfaces as explicit errors in the execution history, not a silent stall, and the 90-second duration signature points at the invocation clock
Answer in the comments.
Top comments (5)
The answer is C.
Replay is the whole trick behind durable functions (checkpoint-and-replay — the handler re-runs from the top on every resume; completed steps and waits return their stored results instead of re-executing), and it has one iron rule: only durable operations are skipped.
context.step()results are checkpointed.context.wait()completions are checkpointed. Everything else in your handler is just code — and code runs again, from the top, on every resume.This workflow resumes a lot. Every 6-hour review cycle ends a wait, which triggers a fresh invocation. Each of those invocations starts at line 1: download 40 MB from S3 (object storage), parse for ~50 seconds, then replay the checkpoint log — validate (skipped), credit report (skipped), wait 1 (skipped), check 1 (skipped), wait 2 (skipped)... The replay itself is fast per checkpoint, but the log grows with every cycle. By review cycle 8 or 9, 50 seconds of un-checkpointed loading plus replay overhead plus the actual status check no longer fit inside the 90-second function timeout. The invocation is killed mid-flight.
Here is the vicious part: a timed-out resume doesn't fail the durable execution — the platform retries the resume. The retry starts at line 1, pays the same 50-second tax, and dies the same death. The execution is alive, unexpired, and permanently unable to make progress. No business exception is ever thrown, because the business logic never gets reached. The only witness is
Durationpinned at the timeout.Two fixes:
Fix 1 — Move the heavy work into the world replay can skip. Load the ruleset inside a
context.step()that returns only the small values the workflow actually needs downstream (thresholds, version id) — checkpointed once, skipped on every replay. If steps genuinely need the full parsed object, cache it at module scope keyed on the ruleset version: warm environments reuse it, and a cold resume pays the load once inside a step boundary, not on every replay of history.Fix 2 — Design for replay speed as history grows. Keep the code path outside steps trivial and deterministic (that is also the correctness rule — no clocks, randomness, or I/O outside durable operations, or replay diverges). Use one step per side effect rather than dozens of fine-grained checkpoints, and set the function timeout with headroom for the longest replay your workflow can accumulate — a durable function's resume cost is a function of its history, not just its next action.
The general pattern is worth naming: this is durable execution, the same model as Temporal or Azure Durable Functions, and it always carries this contract — replay must be fast and deterministic, and anything expensive or side-effectful belongs inside a checkpointed activity. The 15-minute wall came down; the replay discipline came with it.
B —
context.wait()is the opposite of running: the execution checkpoints, suspends, and stops billing entirely. When the wait elapses, Lambda starts a new invocation with a fresh function timeout. Waits cost nothing here — the resumes were the problem.A — The execution timeout and the function timeout are different clocks. The execution timeout (up to 1 year) bounds the entire durable execution's lifetime; the function timeout bounds each individual invocation. These executions were days from their execution deadline. They were dying 90 seconds at a time.
D — Durable executions do have per-execution limits on operations and checkpointed state, and hitting them matters at high step counts. But limit violations surface as errors in the execution history. A silent stall with duration pinned at the function timeout is the signature of an invocation clock, not a quota.
Also, it would mean a lot to me if you could support my content and stay in touch 🙏
YouTube: youtube.com/@system-design-lab
LinkedIn: linkedin.com/in/joud-awad/
Medium Blog: joudwawad.medium.com/
Substack: joudawad.substack.com/