Orchestrating Our Agents SDLC on Temporal: The Hard Parts Nobody Warns You About

We turned a manual, prompt-by-prompt AI coding workflow into a durable, self-driving software development lifecycle. Here is everything that broke on the way, and how we fixed it.

Lesson learned: an AI agents SDLC is not a script you write once and walk away from. It is a distributed system that runs continuously, and it demands the same operational seriousness as any other production service you own.

Where this started: doing it by hand

Before there was any orchestration, there was a human (me) driving an AI coding assistant one prompt at a time.

The loop was always the same. Feed the assistant a spec. Ask it to investigate the codebase. Ask it to plan. Ask it to write the code. Ask it to write tests. Ask it to review its own work with a fresh context. Ask it to fix what the review found. Translate the strings. Open the pull request. Every step fed the next one. Every step had a clear input, a clear output, and a clear pass/fail signal.

After doing this by hand enough times, the realization was uncomfortable and obvious at the same time:

This is not creative work. This is a deterministic pipeline with an AI in the middle of each box.

And deterministic, repeatable, multi-step pipelines are a solved problem. We do not run them by hand. We orchestrate them.

That is the whole origin of this project. We had a manual agent-driven SDLC that we ran with our hands, we noticed it was really a state machine, and we decided to make it durable and automated. The tool we reached for was Temporal, a durable execution engine that survives crashes, retries failed steps, and remembers exactly where every run is. We self-host it on EKS, right next to the worker fleet.

This article is about what that actually cost us in engineering pain, and how we dealt with each problem. Everything here is generalized so you can apply it to your own stack, whatever agent, model, or cloud you are on.

The pipeline and the agents

Our SDLC is a fixed sequence of specialized agents, each with one job and a fresh context. An intent analyst turns a raw request into a structured spec. An investigator maps the codebase and finds what already exists to reuse. An architect (our most expensive model, because planning has the highest leverage) writes a file-by-file plan, which a separate plan critic has to approve before a single line is written. An editor then does the actual coding, and the same editor is reused later to apply fixes. Tests fan out in parallel: a unit-test writer and an e2e-test writer run at once, gated by a coverage critic. A staff QA agent verifies the build against the spec and the live UI in a bounded loop, sending work back to the editor until it passes. Then three reviewers on different models review in parallel and a consolidator merges their findings, feeding a fix loop. A fan-out of translators, one per locale, localizes every new string, and a merge step stitches them back together. Finally a PR opener assembles the pull request, a PR critic checks it, and a human takes it from there. The cheaper the model can be for a job, the cheaper it is: planning and review run on a top-tier model, editing on a mid-tier one, and mechanical work like translation on the cheapest. Every tier is served through Amazon Bedrock, so switching a step to a different model is a configuration change, not a new integration.

Our SDLC workflow

A reality check before you copy this

Let me be blunt, because the marketing around “AI agents” makes this sound easier than it is.

Temporal-as-agent-orchestrator is not a low-code toy. It is not for non-technical people. If your mental model of “AI automation” is dragging boxes in a visual builder, this is the wrong article.

To do this you need to be comfortable with:

• Distributed systems thinking (workers, queues, retries, at-least-once execution).
• The workflow-versus-activity split, and Temporal’s determinism constraints.
• Object storage, secrets management, and container orchestration.
• The fact that your “automation” is now a fleet of long-lived worker processes that you deploy, monitor, upgrade, and get paged for.

If you are not ready to own infrastructure full time, you are not ready to run an agents SDLC on Temporal.

I will keep coming back to that theme, because it is the single most underestimated part of the whole thing. The code that defines the pipeline is maybe a third of the work. The rest is operating it.

With that out of the way, here are the real problems.

Screenshot from our a workflow run in our Temporal

Determinism: the constraint that shapes everything

Temporal splits your code into two worlds:

• Workflow code is the orchestration brain. It must be deterministic, because Temporal achieves durability by replaying the entire recorded event history from the beginning to rebuild state after a crash or a worker restart. Replaying the same history must produce the exact same sequence of decisions, or the engine raises a non-determinism error and refuses to proceed.
• Activity code is where side effects live. Network calls, disk writes, randomness, clocks, and yes, LLM calls, all belong here. An activity’s result is recorded once and replayed from history, never recomputed.

This sounds academic until an AI pipeline smashes into it. An LLM call is the least deterministic thing in your entire system. It does network I/O, it is nondeterministic by design, and it costs money each time. So the rule writes itself:

Every model call, every file write, every API request lives in an activity. The workflow only decides what runs next.

The traps are subtle and they bite late:

• Reaching for the wall clock inside workflow code. Use the engine’s deterministic time source, never the language’s native clock.
• Randomness inside the workflow. If you need a random choice, generate it in an activity and pass it back.
• Importing heavy side-effecting libraries at workflow module load. Your AWS SDK (boto3) or database driver doing I/O at import time can poison determinism. Isolate those imports so the workflow definition stays pure.

None of this is exotic once you internalize it. But it is a genuine mental shift, and it is the number one thing new team members get wrong. Onboarding an engineer into this model is a real, measurable cost, not a footnote.

Passing state between steps: the payload limit will find you

Here is the first wall almost everyone hits.

Temporal moves inputs and outputs of workflows and activities through its persistence layer. Those payloads are capped. A single payload is limited to about 2 MB, and the whole gRPC request that carries it has a hard ceiling around 4 MB. The limit is there for a good reason: every payload becomes part of the durable history that gets replayed, so a fat payload is a fat history forever. Blow past it and you get a blob-size error that, in practice, kills the run.

Now look at what an agents SDLC wants to pass around:

• Multi-thousand-line specs.
• Whole plans and multi-step reports.
• Generated source files.
• The raw request and response bodies of model calls, which can be enormous.

You will blow through the payload limit almost immediately. And even if a payload squeaks under the cap, stuffing large blobs into workflow history bloats replay and slows everything down.

The fix is the pattern every serious Temporal user converges on, sometimes called the claim-check pattern:

Do not pass the data. Pass a pointer to the data.

We offload every large artifact to object storage (S3 in our case) and thread only a small reference through the workflow: a URI, a content hash, and a filename. The shape looks like this:

# What the workflow passes between steps.
# NOT the 200KB plan, just the address of it.
@dataclass
class ArtifactRef:
    uri: str          # s3://bucket/<workflow_id>/<phase>/<file>
    sha256: str       # integrity check on download
    filename: str     # stable name to rematerialize under

Each step uploads its output to object storage and returns an ArtifactRef. The next step downloads the artifacts it needs into local scratch space before it invokes the model. The content hash guards against silent corruption or a truncated upload.

This works, but be honest about what you just signed up for:

• A new failure mode. Object storage can 404, throttle, or hand you a stale read. Your workflow now depends on it being healthy.
• Cleanup. Artifacts accumulate forever unless you add lifecycle rules or a scrubbing step. We ended up doing both.
• Access management. Your worker fleet needs credentials to read and write the bucket. On EKS that means IAM Roles for Service Accounts (IRSA) wired correctly, which is its own project.
• Key discipline. The object key has to be built from something globally unique and stable across retries. This detail caused us real pain, which is the next story.

The cross-pod /tmp trap: the bug that taught us the most

This is the one to internalize, because it is invisible until you scale past one worker.

When you run a single worker process, local disk feels shared. A step writes a file to local scratch space, a later step reads it back, everything works. You will build a mental model where /tmp is a shared filesystem.

It is not. The moment you run more than one worker replica (and you will, for throughput and availability), Temporal schedules steps across whichever worker is free. Step A lands on worker 1 and writes a file. Step B lands on worker 2 and looks for that file. It is not there. Worker 2’s local disk is a completely different disk.

We hit this in three separate places, and each one looked like a different bug at first:

1. A fan-out merge that read local scratch. We fan out one child step per unit of work (think one per language, one per file), then a merge step combines the results. The children wrote their outputs to local disk. The merge step read local disk. With two workers, the children spread across both pods, and the merge only ever saw the half that happened to land on its own pod. Half the work silently vanished, and the merge cheerfully reported success on the half it could see.
2. A shared input file that only one pod had. A pre-step computed a canonical input once and wrote it to local disk. The fan-out children were told to read that file. Children scheduled on the other pod found nothing, and the agent, being an agent, improvised a polite little narrative explaining that its input file did not exist. That narrative then got captured as if it were the step’s real output.
3. A latent parameter bug that only surfaced after we moved to object storage. When the local-disk version worked by filename alone, we were passing the wrong identifier as the storage key and nobody noticed, because that field was never used for a path. The day we switched the consumer to build an object-storage key from that field, every download 404'd at once.

The lesson is short and absolute:

In a multi-worker fleet, local disk is a lie. Anything that crosses a step boundary must go through shared storage or the workflow payload. Never through the local filesystem.

Every one of those bugs disappeared the moment the artifact, both inputs and outputs, went through object storage with the reference passed in the workflow. The fix is not clever. The discipline to apply it everywhere, before it bites, is the hard part.

Rerunning a single step without redoing the whole run

Here is a mismatch that trips up newcomers.

Temporal has a concept called replay. It sounds like exactly what you want when a step goes wrong. It is not. Replay means Temporal re-executes your workflow code against recorded history to rebuild in-memory state. It does not re-run your activities. It replays the decisions, not the side effects. That is a durability mechanism, not a “run this LLM call again” button.

What you actually want, constantly, during development and during incident recovery, is: “re-execute this one step against the current state, and leave the other twelve alone.”

The model that worked for us:

• A step filter on the workflow input. The trigger accepts an optional list of step names. When present, the workflow walks its state machine but only executes the named steps. This lets you rerun a single phase, or a contiguous slice, without paying for the rest.
• Idempotency by artifact check. Before a step runs its expensive model call, it checks whether its output artifact already exists for this run. If it does, it short-circuits. Reruns become cheap and safe, and a retried step does not double-charge you.
• Run-scoped keys. Every artifact key is namespaced by the unique run identifier. Rerunning a step overwrites its own slot and never collides with another run.

Design for “rerun one step” from day one. You will use it far more than you expect, both to debug and to recover cheaply.

Testing backdoors: how to validate the pipeline without going broke

A full production run of an agents SDLC is expensive. Ours uses a top-tier model for planning, a mid-tier model for editing, and a cheap model for mechanical work. A single end-to-end run lands in the tens of dollars. You cannot iterate on the orchestration itself at that price. You need cheaper paths in, and you need them built with guardrails so they never fire in production.

We built two.

Backdoor 1: run the entire pipeline on a cheap model

A single global flag forces every step onto the cheapest available model, regardless of what that step normally uses. The full pipeline runs end to end for the price of a coffee instead of a nice dinner. This is how you validate control flow, artifact hand-off, retries, and the S3 plumbing without burning real budget.

if settings.force_cheap_model_testing_mode:
    active_model = CHEAP_MODEL          # every phase, no exceptions
else:
    active_model = phase.preferred_model  # production assignment

Two hard-won caveats:

• A cheap model validates plumbing, not quality. When we ran the whole thing on the cheapest model, the mechanical steps passed, but the code-writing steps produced beautiful reports and made zero actual edits. The cheap model was strong enough to describe the work and too weak to do it. That is fine, as long as you know that the cheap path proves the pipeline moves, not that the output is good.
• The flag must default to off, loudly. A test backdoor that can accidentally run in production is a footgun. Default off, log conspicuously when on, and never let the production trigger set it.

Backdoor 2: run a single step in isolation

The same step filter that powers cheap reruns doubles as a test harness. Want to test just the review step? Trigger the workflow with a filter of exactly one step name. Want a near-free smoke test that proves a worker can pick up a task, call the model, and write an artifact? Keep a dedicated trivial step that is opt-in only and never part of a default run.

Bake in a cheap-model full run and a single-step run before you need them. The alternative is debugging a forty-dollar pipeline forty dollars at a time.

The design principle across both backdoors: test paths are injected through configuration (which model, which steps), never forked into a separate code path. A separate “test mode” codebase drifts from production and lies to you. One code path, parameterized, is the only version that stays honest.

Triggering a run from GitHub Actions

We did not want a bespoke UI just to start a run, and we did not want to expose the orchestrator to the public internet. GitHub Actions turned out to be the perfect front door. A manual workflow_dispatch job gives you a free, permissioned, audited form: a dropdown and a few text fields that anyone on the team can fill in without touching kubectl. The job itself does almost nothing. It takes the inputs, builds a small JSON body, and makes a single authenticated call to the orchestrator's API, which is what actually starts the durable run.

The one detail worth copying: run that job on a self-hosted runner that lives inside the same network as your workers - in our case, inside the EKS cluster - and talk to the orchestrator over internal DNS. That way the trigger endpoint never has to be publicly reachable, and you get GitHub’s identity and audit trail for free instead of building your own auth.

on:
  workflow_dispatch:
    inputs:
      target:      { type: string, required: true }   # what to build
      branch:      { type: string, required: false }  # source branch
      environment: { type: string, default: dev }     # which env to run against
      steps:       { type: string, required: false }  # optional single-step / subset filter
      cheap_mode:  { type: boolean, default: false }   # the cheap-model test path

jobs:
  trigger:
    runs-on: self-hosted            # in-cluster runner, reaches the API over internal DNS
    steps:
      - name: Start the run
        run: |
          curl -fsS -X POST "http://orchestrator.internal/runs" \
            -H 'content-type: application/json' \
            -d "$(jq -n \
                  --arg t  '${{ inputs.target }}' \
                  --arg b  '${{ inputs.branch }}' \
                  --arg e  '${{ inputs.environment }}' \
                  '{target:$t, branch:$b, environment:$e}')"

The same input surface is where your test backdoors live: the steps field is your single-step rerun path, and the cheap_mode toggle is your cheap-model full run. Both are just parameters on the same trigger, which keeps them honest, because there is exactly one way to start a run and the test paths are options on it, not a separate code path.

Do not build a launcher. You already have one. A workflow_dispatch form plus a self-hosted runner gives you a permissioned, audited trigger for free, and keeps the orchestrator off the public internet.

A couple of notes:

• I placed the test-backdoor callback deliberately so this section reinforces the “one parameterized code path, not a separate test mode” point from earlier in the article. If you drop this section in right after the Testing backdoors section, that callback lands naturally. It also works as its own standalone section near the end.
• Everything is generic: orchestrator.internal, generic field names, no product or company specifics. Adjust the endpoint path and field names to taste.

Cost: durable execution can quietly inflate the bill

Cost shows up in more places than the token meter.

• Model tokens. The obvious one. Per-step model selection is your biggest lever: do not send a planning-tier model to do mechanical work.
• A run that fails late is pure waste. We had a run escalate near the very end without producing a usable result. It had already spent most of a full run’s budget on the earlier steps. Money spent, nothing shipped. This is the expensive failure mode of a long pipeline, and it is an argument for gating and early exits, not just retries.
• Long-lived workers cost money even when idle. Durable execution keeps worker processes running and polling. That is compute you pay for around the clock, separate from any model spend.
• History and storage growth. Durable runs accumulate event history and artifacts. Left unmanaged, both grow without bound and both cost money.

There is a hard ceiling hiding here that specifically bites agents. A single workflow execution’s event history is capped (tens of thousands of events, tens of megabytes), with a warning threshold well before that. A tight agent loop that iterates many times, each iteration recording several events, will march toward that ceiling. The escape hatch is to periodically close the current run and start a fresh one with the same identity and a clean history, carrying forward only the state you need. If your agent has any kind of unbounded “keep going until done” loop, you must either bound the iterations or plan for this rollover from the start. We bound our review-and-fix loops to a fixed number of rounds, which sidesteps the problem and, not coincidentally, also caps the cost of a single run.

The durable-execution model trades money for reliability. That is usually a good trade. Just make sure you can see the bill, per run and per step, or it will surprise you.

Long-running steps, heartbeats, and slow model responses

Agent steps are slow. A single reasoning-heavy model call with tool use can run for many minutes. Temporal, sensibly, wants to know a step is still alive so it can reschedule genuinely dead ones.

Temporal gives you several timeouts to reason about: how long a single attempt may run, how long the total including retries may run, how long a task may sit in the queue before a worker picks it up, and the maximum gap between heartbeats. For agents, two of them matter most:

• Set your run-time timeout to reality. A single attempt has to accommodate the slowest legitimate model response plus tool calls, with margin. Too tight and Temporal kills healthy work.
• Heartbeat during the long gaps, or you will pay twice. We heartbeat on every message the model streams back, and we run a baseline heartbeat on a timer so that even a long silent stretch of tool execution keeps the lease alive. This is not just about avoiding confusion. If a step stops heartbeating, Temporal assumes the worker died and retries the step, which fires a second, identical, expensive model call while the first one is very likely still running. Missing heartbeats do not just look bad in a dashboard, they double your token bill on your slowest steps.

Producer step:
  start a background heartbeat every 30s
  for each streamed message from the model:
     record output, heartbeat with progress
  stop the background heartbeat

Streaming responses are a gift here: each chunk is a natural heartbeat opportunity. If your model client only returns at the end, you must run a timer-based heartbeat or you will fight false timeouts forever.

Retries, backoff, and the difference between “retry” and “escalate”

Model APIs are flaky. Rate limits, transient 5xx, the occasional refusal. Temporal’s retry policies handle the transient cases well: exponential backoff, capped attempts, per-activity tuning.

One gotcha up front: the default retry policy retries forever. On a rate-limited model API, that default turns a temporary 429 into a retry storm that hammers the provider and burns your quota. Cap the attempts, cap the maximum interval, and when the provider hands you a Retry-After header, honor it by setting the next retry delay explicitly instead of letting blind backoff guess.

The deeper nuance that matters for agents is what should not be retried at all.

Some failures are not transient. If a step produced output that fails a hard validation, retrying the identical step usually reproduces the identical failure while burning tokens each time. You want those to stop retrying and escalate, either to a different branch of the workflow or to a human.

We model this with a small set of non-retryable error types. When a step raises one of those, Temporal does not spin; the workflow catches it and decides what to do next.

Retry transient infrastructure failures. Escalate deterministic content failures. Retrying an agent that is confidently wrong just pays to be told the same wrong thing again.

Agents lie about success, so put mechanical gates between them

This one is philosophical and extremely practical.

Agents are optimistic narrators. Ask a coding agent whether it ran the type checker and it will often tell you, sincerely, that everything passed. Then you look, and the workspace has half a dozen compile errors from hallucinated APIs.

We learned the hard way that a prompt instruction like “you must run the build and confirm it is green before returning” is necessary but not sufficient. Agents skip mandated steps and report success anyway.

The fix is to stop trusting self-reports and add deterministic gates:

A prompt-level requirement is one refactor away from being ignored. A mechanical gate is enforcement.

Concretely: after a code-writing step returns, an activity independently runs the real type check and build against the resulting workspace. If it fails, the step fails, with the compiler output attached, and the workflow escalates instead of advancing. The agent’s opinion of its own work is an input, not the verdict.

This also fixed an observability lie we will get to below: once you have an external gate, “the agent said it passed” and “it actually passed” become two separate, comparable facts.

Machine-parseable contracts between agent and orchestrator

Related trap: your orchestrator has to make control-flow decisions based on what an agent produced. Pass or fail. Revise or approve. If you parse that decision out of free-form prose, you are one polite paragraph away from a stuck workflow.

We had a review step escalate the entire run for a dumb reason: the agent wrote a perfectly good summary of its fixes but never emitted the exact verdict token the orchestrator was grepping for. No token, no decision, escalate.

The boundary between an agent and your control flow must be a strict, machine-parseable contract. Require an exact verdict line, and validate it. If the agent does not produce it, that itself is a failure, not a maybe.

Prose is for humans. Control flow needs a token.

Observability: wiring LLM tracing into a durable engine

Temporal gives you excellent visibility into its own world: every workflow, every step, every retry, every input and output, all queryable. What it does not give you is the thing you most want to see in an agents SDLC: what the model actually did. Tokens, prompts, responses, tool calls, cost per step.

For that we plugged in a dedicated LLM tracing tool (Langfuse, self-hosted on the same EKS cluster, in our case). You end up running two observability planes: one for orchestration and durability, one for prompts, tokens, and cost. Getting them to line up was more friction than expected.

Screenshot from our Langfuse, showing a cost breakdown of a specific LLM call

Span propagation across process boundaries. Our model calls happen inside a subprocess spawned by the step. To make the model’s own spans nest under the right workflow-and-step span, we inject the trace context (the W3C trace parent and state) into the subprocess environment. Without that, you get two disconnected trees: Temporal’s view of the run, and a pile of orphan LLM spans with no idea which run or step they belong to. Correlating them by hand is misery. Propagate the trace context explicitly, or you do not really have distributed tracing, you have two separate logs.

The tracing backend is infrastructure you now own. This is the part nobody warns you about. Your LLM tracing tool has its own database, its own object storage, its own failure modes. Ours quietly stopped ingesting traces for a long time. The dashboard showed zero traces and everyone assumed the instrumentation was broken. The real cause: the object storage backing the tracing tool had run out of inodes (not out of space, out of inodes, because it wrote millions of tiny objects), and every write was failing silently behind a generic error. The fix was half cleanup and half migrating that storage to something with proper lifecycle management.

Your observability stack is production infrastructure. If you do not monitor the thing that monitors your agents, you will discover it is down only when you desperately need it.

Watch out for observability that lies. The best bug of the whole project: a per-step metric that counted how many tool calls the agent made. It always reported zero. For a while we believed the agents were doing nothing. They were doing plenty; the counter was checking the wrong attribute shape on the SDK’s response objects and silently matching nothing. A step that wrote dozens of files logged “0 tool calls.” An instrument that reports zero when the truth is fifty-five is worse than no instrument, because it actively misleads triage.

Verify your instrumentation against ground truth at least once. A metric you never validated is a rumor.

Versioning workflows when the agent logic changes weekly

Agent pipelines change constantly. You reorder steps, add a gate, change a prompt, split one step into two. Meanwhile, Temporal may have long-running executions in flight that started under the old code. Determinism means you cannot just change the workflow and redeploy; a replay of an old run against new code will diverge from its recorded history and fail.

Temporal has first-class tools for this: patch-style version gates you place in the workflow code, and worker-level versioning (build IDs pinned to executions) so old and new runs route to compatible workers. Recent guidance leans toward worker versioning as the default for teams that can run versioned deployments. The practical takeaways:

• If your runs are short-lived, the simplest safe move is to drain old runs before rolling out incompatible workflow changes.
• If your runs are long, you must use the versioning primitives deliberately. Every incompatible change to workflow (not activity) code needs a version gate.
• Changing activity code is far more forgiving than changing workflow code. Push behavior into activities partly for this reason: it is the layer you can evolve freely.

This is a recurring tax. Frequent agent-logic changes and long-lived durable executions are in natural tension, and you pay for it in versioning discipline.

Human-in-the-loop is a feature, not an afterthought

A fully autonomous agents SDLC that opens a pull request and merges it with nobody looking is a great way to ship confident nonsense to production.

Temporal’s signals, updates, and timers make human checkpoints clean to model. A workflow can pause and wait for an external approval, with a timeout that escalates if nobody responds. We keep humans at the high-leverage gates: the plan looks right before code gets written, and the final change gets human review before it lands.

Autonomy is a dial, not a switch. Put the human where a wrong decision is most expensive, and let the machine run everything cheap and reversible.

Local development and debugging nondeterministic agents

Testing a nondeterministic thing on a deterministic engine is inherently awkward, and no amount of tooling fully removes the awkwardness.

Temporal’s test framework is genuinely good: a time-skipping test server, and replay tests that catch determinism violations before they reach production. Use both. Replay tests in particular are your safety net against accidentally breaking workflow determinism during a refactor.

But the agent itself will never be reproducible. Replay tests catch code drift, not model drift. Two things made debugging bearable:

• Persist the raw model exchanges. We dump every request and response body to disk per run and per step. When a step misbehaves, the transcript is right there. This is how we proved the “0 tool calls” metric was lying and how we caught agents writing to the wrong output paths.
• Structured, queryable step records. Every step writes a record with its inputs, outputs, token counts, cost, and duration. When something goes wrong three steps back, you are reading records, not scrolling logs.

You cannot make the agent deterministic. You can make its behavior fully auditable after the fact. Do that.

The part nobody budgets for: this is a system you run forever

Here is the theme I promised to keep hammering.

The workflow code is not the product. The running system is the product, and it never stops needing you. A partial, honest list of what “operating an agents SDLC on Temporal” actually involves:

• A worker fleet to deploy, scale, and roll safely on every change.
• Version pinning between workflow code, worker builds, and in-flight executions.
• A persistence store for Temporal itself (Amazon RDS, in our case), with its own capacity and backup story.
• The object storage layer for artifacts, with lifecycle and cleanup.
• The LLM tracing stack, with its own database and storage to monitor.
• Cloud credentials and workload identity (IAM roles and IRSA) for every component that touches a model API or a bucket, which rotate and expire and break at the worst time.
• Monitoring and alerting on all of the above, plus somebody on call when a run wedges at 2am.

During this project we lost time to expiring AWS SSO sessions, VPN and DNS resolution flakiness getting to the cluster, identity federation for model-provider auth breaking after a platform change, and an Amazon RDS password rotation that desynced and crashlooped a service. None of that is agent work. All of it is the cost of running the system that runs the agents.

Budget for operations as a standing cost, not a one-time setup. If you cannot staff the “run it forever” part, do not start the “build it once” part.

A comparison, briefly

Temporal is not the only way to orchestrate agents, and it is heavier than the alternatives.

It helps to separate three different jobs people all call “orchestration”:

• Batch and data scheduling (the DAG schedulers): great at “run this graph of jobs on a schedule,” not built for durable per-step replay or human waits.
• Agent frameworks and graph libraries (the AI-native tools): purpose-built for the agent loop itself, with streaming, memory, tool routing, and LLM evals baked in. They let you wire up multi-step agent flows in a single process with almost no infrastructure.
• Durable execution engines (Temporal, and lighter cousins backed by a database you already run): built to survive a crash mid-run, resume exactly where you were, retry one step with backoff, wait days for a human signal, and keep a queryable record of thousands of concurrent runs.

The AI-native frameworks give you streaming, memory, and prompt management for free, and Temporal makes you build all of that yourself. What Temporal gives you in return is durability that the lightweight tools simply do not have. A very common and very sane pattern is to run both: a durable engine for the long-running orchestration, and an agent framework for the reasoning loop inside a step.

That durability is the entire reason to pay Temporal’s tax.

Choose a durable engine when a half-finished run is expensive and “just start over” is not acceptable. Choose something lighter when it is.

Before you commit: a checklist

• Are your steps genuinely a deterministic pipeline, or are you forcing structure onto something exploratory? Temporal rewards the former and fights the latter.
• Can you keep every side effect, especially model calls, inside activities without contorting your design?
• Have you designed artifact hand-off through object storage from the start, with references in the workflow and nothing large in the payload?
• Do you have a single-step rerun path and a cheap-model full run, with guardrails so test paths never hit production?
• Is every agent-to-orchestrator decision a strict, machine-parseable contract?
• Do you have mechanical gates that verify agent claims instead of trusting them?
• Is your LLM tracing stack monitored like the production infrastructure it is?
• Do you have the people and the on-call maturity to run this indefinitely?

If most of those are “yes,” Temporal will serve you extremely well. If most are “no,” you will spend more time fighting the engine than shipping features.

Conclusion: worth it, with eyes open

Turning a manual, prompt-by-prompt agent workflow into a durable orchestrated SDLC was one of the highest-leverage things we built. Runs survive crashes. Failed steps retry themselves. A run that dies at step nine resumes at step nine. We can rerun one phase for pennies, trace every model call to its cost, and gate the agents behind checks they cannot talk their way past.

But none of that came from the happy-path tutorial. It came from the payload limit, the cross-pod disk trap, the lying metric, the silent tracing outage, the confidently-wrong agents, and the steady operational drip of auth, DNS, and rollout problems.

Temporal makes an agents SDLC durable. It does not make it easy, and it does not make it self-maintaining. It is a system, and systems have owners.

If you have the engineering maturity and the appetite to own infrastructure, orchestrating your agents SDLC on a durable execution engine is absolutely worth it. If you were hoping for a set-and-forget button, this is not that, and anyone who tells you otherwise has not run one in production.

We did it by hand, we made it deterministic, and then we spent the real effort making the determinism survive contact with reality. That last part is the job.

Further reading and resources

Temporal core concepts

• Temporal documentation home
• Workflow definition and determinism constraints
• Temporal core concepts (workflows, activities, workers, task queues)
• Temporal courses (free, hands-on)

Determinism, replay, and versioning

• Workflow versioning with patching (TypeScript)
• Workflow versioning (Go)
• Worker versioning and build IDs
• Replay testing to avoid non-determinism (Bitovi)

Payload size and passing large state (the claim-check pattern)

• Troubleshooting the blob size limit error
• External storage / large payload codec
• Community thread: handling large workflow/activity payloads

Timeouts, heartbeats, and long-running work

• Activity timeouts explained
• Detecting activity failures and heartbeating
• Very long-running workflows and Continue-As-New
• Workflow execution limits (event history caps)

Retries, backoff, and flaky model APIs

• Retry policies
• Handling HTTP 429 / Retry-After in activities (Temporal AI cookbook)
• Rate-limiting downstream APIs from Temporal

Human-in-the-loop

• Signals, updates, queries, and timers

Testing and local development

• Testing suite (Python)
• Testing suite (Go)
• Practical guide to Temporal workflow testing

Observability: LLM-level tracing on top of Temporal

• Langfuse OpenTelemetry integration
• Langfuse SDK instrumentation and trace context
• OpenLLMetry / OpenLIT (open-source LLM OpenTelemetry)
• W3C Trace Context specification

Temporal for AI agents specifically

• Building dynamic AI agents with Temporal (official)
• Temporal AI cookbook
• Production pitfalls of Temporal for AI agent orchestration (xGrid)

Self-hosting and cost

• Self-hosted Temporal defaults and persistence stores
• Temporal Cloud pricing (Actions model)

Comparisons and alternatives

• LangGraph vs Temporal (LangChain, vendor perspective)
• Temporal alternatives overview (ZenML)
• A practical guide to Temporal: what it does and when to use it (HackerNoon)
• Restate (lightweight durable execution)
• DBOS (durable execution on Postgres)

The AI-native agent frameworks (for the reasoning loop inside a step)

• LangGraph
• Model Context Protocol (MCP)