DEV Community: Yedidya Schwartz

Orchestrating Our Agents SDLC on Temporal: The Hard Parts Nobody Warns You About

Yedidya Schwartz — Fri, 17 Jul 2026 16:11:13 +0000

We turned a manual, prompt-by-prompt AI coding workflow into a durable, self-driving software development lifecycle. Here is everything that broke on the way, and how we fixed it.

Lesson learned: an AI agents SDLC is not a script you write once and walk away from. It is a distributed system that runs continuously, and it demands the same operational seriousness as any other production service you own.

Where this started: doing it by hand

Before there was any orchestration, there was a human (me) driving an AI coding assistant one prompt at a time.

The loop was always the same. Feed the assistant a spec. Ask it to investigate the codebase. Ask it to plan. Ask it to write the code. Ask it to write tests. Ask it to review its own work with a fresh context. Ask it to fix what the review found. Translate the strings. Open the pull request. Every step fed the next one. Every step had a clear input, a clear output, and a clear pass/fail signal.

After doing this by hand enough times, the realization was uncomfortable and obvious at the same time:

This is not creative work. This is a deterministic pipeline with an AI in the middle of each box.

And deterministic, repeatable, multi-step pipelines are a solved problem. We do not run them by hand. We orchestrate them.

That is the whole origin of this project. We had a manual agent-driven SDLC that we ran with our hands, we noticed it was really a state machine, and we decided to make it durable and automated. The tool we reached for was Temporal, a durable execution engine that survives crashes, retries failed steps, and remembers exactly where every run is. We self-host it on EKS, right next to the worker fleet.

This article is about what that actually cost us in engineering pain, and how we dealt with each problem. Everything here is generalized so you can apply it to your own stack, whatever agent, model, or cloud you are on.

The pipeline and the agents

Our SDLC is a fixed sequence of specialized agents, each with one job and a fresh context. An intent analyst turns a raw request into a structured spec. An investigator maps the codebase and finds what already exists to reuse. An architect (our most expensive model, because planning has the highest leverage) writes a file-by-file plan, which a separate plan critic has to approve before a single line is written. An editor then does the actual coding, and the same editor is reused later to apply fixes. Tests fan out in parallel: a unit-test writer and an e2e-test writer run at once, gated by a coverage critic. A staff QA agent verifies the build against the spec and the live UI in a bounded loop, sending work back to the editor until it passes. Then three reviewers on different models review in parallel and a consolidator merges their findings, feeding a fix loop. A fan-out of translators, one per locale, localizes every new string, and a merge step stitches them back together. Finally a PR opener assembles the pull request, a PR critic checks it, and a human takes it from there. The cheaper the model can be for a job, the cheaper it is: planning and review run on a top-tier model, editing on a mid-tier one, and mechanical work like translation on the cheapest. Every tier is served through Amazon Bedrock, so switching a step to a different model is a configuration change, not a new integration.

Our SDLC workflow

A reality check before you copy this

Let me be blunt, because the marketing around “AI agents” makes this sound easier than it is.

Temporal-as-agent-orchestrator is not a low-code toy. It is not for non-technical people. If your mental model of “AI automation” is dragging boxes in a visual builder, this is the wrong article.

To do this you need to be comfortable with:

Distributed systems thinking (workers, queues, retries, at-least-once execution).
The workflow-versus-activity split, and Temporal’s determinism constraints.
Object storage, secrets management, and container orchestration.
The fact that your “automation” is now a fleet of long-lived worker processes that you deploy, monitor, upgrade, and get paged for.

If you are not ready to own infrastructure full time, you are not ready to run an agents SDLC on Temporal.

I will keep coming back to that theme, because it is the single most underestimated part of the whole thing. The code that defines the pipeline is maybe a third of the work. The rest is operating it.

With that out of the way, here are the real problems.

Screenshot from our a workflow run in our Temporal

Determinism: the constraint that shapes everything

Temporal splits your code into two worlds:

Workflow code is the orchestration brain. It must be deterministic, because Temporal achieves durability by replaying the entire recorded event history from the beginning to rebuild state after a crash or a worker restart. Replaying the same history must produce the exact same sequence of decisions, or the engine raises a non-determinism error and refuses to proceed.
Activity code is where side effects live. Network calls, disk writes, randomness, clocks, and yes, LLM calls, all belong here. An activity’s result is recorded once and replayed from history, never recomputed.

This sounds academic until an AI pipeline smashes into it. An LLM call is the least deterministic thing in your entire system. It does network I/O, it is nondeterministic by design, and it costs money each time. So the rule writes itself:

Every model call, every file write, every API request lives in an activity. The workflow only decides what runs next.

The traps are subtle and they bite late:

Reaching for the wall clock inside workflow code. Use the engine’s deterministic time source, never the language’s native clock.
Randomness inside the workflow. If you need a random choice, generate it in an activity and pass it back.
Importing heavy side-effecting libraries at workflow module load. Your AWS SDK (boto3) or database driver doing I/O at import time can poison determinism. Isolate those imports so the workflow definition stays pure.

None of this is exotic once you internalize it. But it is a genuine mental shift, and it is the number one thing new team members get wrong. Onboarding an engineer into this model is a real, measurable cost, not a footnote.

Passing state between steps: the payload limit will find you

Here is the first wall almost everyone hits.

Temporal moves inputs and outputs of workflows and activities through its persistence layer. Those payloads are capped. A single payload is limited to about 2 MB, and the whole gRPC request that carries it has a hard ceiling around 4 MB. The limit is there for a good reason: every payload becomes part of the durable history that gets replayed, so a fat payload is a fat history forever. Blow past it and you get a blob-size error that, in practice, kills the run.

Now look at what an agents SDLC wants to pass around:

Multi-thousand-line specs.
Whole plans and multi-step reports.
Generated source files.
The raw request and response bodies of model calls, which can be enormous.

You will blow through the payload limit almost immediately. And even if a payload squeaks under the cap, stuffing large blobs into workflow history bloats replay and slows everything down.

The fix is the pattern every serious Temporal user converges on, sometimes called the claim-check pattern:

Do not pass the data. Pass a pointer to the data.

We offload every large artifact to object storage (S3 in our case) and thread only a small reference through the workflow: a URI, a content hash, and a filename. The shape looks like this:

# What the workflow passes between steps.
# NOT the 200KB plan, just the address of it.
@dataclass
class ArtifactRef:
    uri: str          # s3://bucket/<workflow_id>/<phase>/<file>
    sha256: str       # integrity check on download
    filename: str     # stable name to rematerialize under

Each step uploads its output to object storage and returns an ArtifactRef. The next step downloads the artifacts it needs into local scratch space before it invokes the model. The content hash guards against silent corruption or a truncated upload.

This works, but be honest about what you just signed up for:

A new failure mode. Object storage can 404, throttle, or hand you a stale read. Your workflow now depends on it being healthy.
Cleanup. Artifacts accumulate forever unless you add lifecycle rules or a scrubbing step. We ended up doing both.
Access management. Your worker fleet needs credentials to read and write the bucket. On EKS that means IAM Roles for Service Accounts (IRSA) wired correctly, which is its own project.
Key discipline. The object key has to be built from something globally unique and stable across retries. This detail caused us real pain, which is the next story.

The cross-pod /tmp trap: the bug that taught us the most

This is the one to internalize, because it is invisible until you scale past one worker.

When you run a single worker process, local disk feels shared. A step writes a file to local scratch space, a later step reads it back, everything works. You will build a mental model where /tmp is a shared filesystem.

It is not. The moment you run more than one worker replica (and you will, for throughput and availability), Temporal schedules steps across whichever worker is free. Step A lands on worker 1 and writes a file. Step B lands on worker 2 and looks for that file. It is not there. Worker 2’s local disk is a completely different disk.

We hit this in three separate places, and each one looked like a different bug at first:

A fan-out merge that read local scratch. We fan out one child step per unit of work (think one per language, one per file), then a merge step combines the results. The children wrote their outputs to local disk. The merge step read local disk. With two workers, the children spread across both pods, and the merge only ever saw the half that happened to land on its own pod. Half the work silently vanished, and the merge cheerfully reported success on the half it could see.
A shared input file that only one pod had. A pre-step computed a canonical input once and wrote it to local disk. The fan-out children were told to read that file. Children scheduled on the other pod found nothing, and the agent, being an agent, improvised a polite little narrative explaining that its input file did not exist. That narrative then got captured as if it were the step’s real output.
A latent parameter bug that only surfaced after we moved to object storage. When the local-disk version worked by filename alone, we were passing the wrong identifier as the storage key and nobody noticed, because that field was never used for a path. The day we switched the consumer to build an object-storage key from that field, every download 404'd at once.

The lesson is short and absolute:

In a multi-worker fleet, local disk is a lie. Anything that crosses a step boundary must go through shared storage or the workflow payload. Never through the local filesystem.

Every one of those bugs disappeared the moment the artifact, both inputs and outputs, went through object storage with the reference passed in the workflow. The fix is not clever. The discipline to apply it everywhere, before it bites, is the hard part.

Rerunning a single step without redoing the whole run

Here is a mismatch that trips up newcomers.

Temporal has a concept called replay. It sounds like exactly what you want when a step goes wrong. It is not. Replay means Temporal re-executes your workflow code against recorded history to rebuild in-memory state. It does not re-run your activities. It replays the decisions, not the side effects. That is a durability mechanism, not a “run this LLM call again” button.

What you actually want, constantly, during development and during incident recovery, is: “re-execute this one step against the current state, and leave the other twelve alone.”

The model that worked for us:

A step filter on the workflow input. The trigger accepts an optional list of step names. When present, the workflow walks its state machine but only executes the named steps. This lets you rerun a single phase, or a contiguous slice, without paying for the rest.
Idempotency by artifact check. Before a step runs its expensive model call, it checks whether its output artifact already exists for this run. If it does, it short-circuits. Reruns become cheap and safe, and a retried step does not double-charge you.
Run-scoped keys. Every artifact key is namespaced by the unique run identifier. Rerunning a step overwrites its own slot and never collides with another run.

Design for “rerun one step” from day one. You will use it far more than you expect, both to debug and to recover cheaply.

Testing backdoors: how to validate the pipeline without going broke

A full production run of an agents SDLC is expensive. Ours uses a top-tier model for planning, a mid-tier model for editing, and a cheap model for mechanical work. A single end-to-end run lands in the tens of dollars. You cannot iterate on the orchestration itself at that price. You need cheaper paths in, and you need them built with guardrails so they never fire in production.

We built two.

Backdoor 1: run the entire pipeline on a cheap model

A single global flag forces every step onto the cheapest available model, regardless of what that step normally uses. The full pipeline runs end to end for the price of a coffee instead of a nice dinner. This is how you validate control flow, artifact hand-off, retries, and the S3 plumbing without burning real budget.

if settings.force_cheap_model_testing_mode:
    active_model = CHEAP_MODEL          # every phase, no exceptions
else:
    active_model = phase.preferred_model  # production assignment

Two hard-won caveats:

A cheap model validates plumbing, not quality. When we ran the whole thing on the cheapest model, the mechanical steps passed, but the code-writing steps produced beautiful reports and made zero actual edits. The cheap model was strong enough to describe the work and too weak to do it. That is fine, as long as you know that the cheap path proves the pipeline moves, not that the output is good.
The flag must default to off, loudly. A test backdoor that can accidentally run in production is a footgun. Default off, log conspicuously when on, and never let the production trigger set it.

Backdoor 2: run a single step in isolation

The same step filter that powers cheap reruns doubles as a test harness. Want to test just the review step? Trigger the workflow with a filter of exactly one step name. Want a near-free smoke test that proves a worker can pick up a task, call the model, and write an artifact? Keep a dedicated trivial step that is opt-in only and never part of a default run.

Bake in a cheap-model full run and a single-step run before you need them. The alternative is debugging a forty-dollar pipeline forty dollars at a time.

The design principle across both backdoors: test paths are injected through configuration (which model, which steps), never forked into a separate code path. A separate “test mode” codebase drifts from production and lies to you. One code path, parameterized, is the only version that stays honest.

Triggering a run from GitHub Actions

We did not want a bespoke UI just to start a run, and we did not want to expose the orchestrator to the public internet. GitHub Actions turned out to be the perfect front door. A manual workflow_dispatch job gives you a free, permissioned, audited form: a dropdown and a few text fields that anyone on the team can fill in without touching kubectl. The job itself does almost nothing. It takes the inputs, builds a small JSON body, and makes a single authenticated call to the orchestrator's API, which is what actually starts the durable run.

The one detail worth copying: run that job on a self-hosted runner that lives inside the same network as your workers - in our case, inside the EKS cluster - and talk to the orchestrator over internal DNS. That way the trigger endpoint never has to be publicly reachable, and you get GitHub’s identity and audit trail for free instead of building your own auth.

on:
  workflow_dispatch:
    inputs:
      target:      { type: string, required: true }   # what to build
      branch:      { type: string, required: false }  # source branch
      environment: { type: string, default: dev }     # which env to run against
      steps:       { type: string, required: false }  # optional single-step / subset filter
      cheap_mode:  { type: boolean, default: false }   # the cheap-model test path

jobs:
  trigger:
    runs-on: self-hosted            # in-cluster runner, reaches the API over internal DNS
    steps:
      - name: Start the run
        run: |
          curl -fsS -X POST "http://orchestrator.internal/runs" \
            -H 'content-type: application/json' \
            -d "$(jq -n \
                  --arg t  '${{ inputs.target }}' \
                  --arg b  '${{ inputs.branch }}' \
                  --arg e  '${{ inputs.environment }}' \
                  '{target:$t, branch:$b, environment:$e}')"

The same input surface is where your test backdoors live: the steps field is your single-step rerun path, and the cheap_mode toggle is your cheap-model full run. Both are just parameters on the same trigger, which keeps them honest, because there is exactly one way to start a run and the test paths are options on it, not a separate code path.

Do not build a launcher. You already have one. A workflow_dispatch form plus a self-hosted runner gives you a permissioned, audited trigger for free, and keeps the orchestrator off the public internet.

A couple of notes:

I placed the test-backdoor callback deliberately so this section reinforces the “one parameterized code path, not a separate test mode” point from earlier in the article. If you drop this section in right after the Testing backdoors section, that callback lands naturally. It also works as its own standalone section near the end.
Everything is generic: orchestrator.internal, generic field names, no product or company specifics. Adjust the endpoint path and field names to taste.

Cost: durable execution can quietly inflate the bill

Cost shows up in more places than the token meter.

Model tokens. The obvious one. Per-step model selection is your biggest lever: do not send a planning-tier model to do mechanical work.
A run that fails late is pure waste. We had a run escalate near the very end without producing a usable result. It had already spent most of a full run’s budget on the earlier steps. Money spent, nothing shipped. This is the expensive failure mode of a long pipeline, and it is an argument for gating and early exits, not just retries.
Long-lived workers cost money even when idle. Durable execution keeps worker processes running and polling. That is compute you pay for around the clock, separate from any model spend.
History and storage growth. Durable runs accumulate event history and artifacts. Left unmanaged, both grow without bound and both cost money.

There is a hard ceiling hiding here that specifically bites agents. A single workflow execution’s event history is capped (tens of thousands of events, tens of megabytes), with a warning threshold well before that. A tight agent loop that iterates many times, each iteration recording several events, will march toward that ceiling. The escape hatch is to periodically close the current run and start a fresh one with the same identity and a clean history, carrying forward only the state you need. If your agent has any kind of unbounded “keep going until done” loop, you must either bound the iterations or plan for this rollover from the start. We bound our review-and-fix loops to a fixed number of rounds, which sidesteps the problem and, not coincidentally, also caps the cost of a single run.

The durable-execution model trades money for reliability. That is usually a good trade. Just make sure you can see the bill, per run and per step, or it will surprise you.

Long-running steps, heartbeats, and slow model responses

Agent steps are slow. A single reasoning-heavy model call with tool use can run for many minutes. Temporal, sensibly, wants to know a step is still alive so it can reschedule genuinely dead ones.

Temporal gives you several timeouts to reason about: how long a single attempt may run, how long the total including retries may run, how long a task may sit in the queue before a worker picks it up, and the maximum gap between heartbeats. For agents, two of them matter most:

Set your run-time timeout to reality. A single attempt has to accommodate the slowest legitimate model response plus tool calls, with margin. Too tight and Temporal kills healthy work.
Heartbeat during the long gaps, or you will pay twice. We heartbeat on every message the model streams back, and we run a baseline heartbeat on a timer so that even a long silent stretch of tool execution keeps the lease alive. This is not just about avoiding confusion. If a step stops heartbeating, Temporal assumes the worker died and retries the step, which fires a second, identical, expensive model call while the first one is very likely still running. Missing heartbeats do not just look bad in a dashboard, they double your token bill on your slowest steps.

Producer step:
  start a background heartbeat every 30s
  for each streamed message from the model:
     record output, heartbeat with progress
  stop the background heartbeat

Streaming responses are a gift here: each chunk is a natural heartbeat opportunity. If your model client only returns at the end, you must run a timer-based heartbeat or you will fight false timeouts forever.

Retries, backoff, and the difference between “retry” and “escalate”

Model APIs are flaky. Rate limits, transient 5xx, the occasional refusal. Temporal’s retry policies handle the transient cases well: exponential backoff, capped attempts, per-activity tuning.

One gotcha up front: the default retry policy retries forever. On a rate-limited model API, that default turns a temporary 429 into a retry storm that hammers the provider and burns your quota. Cap the attempts, cap the maximum interval, and when the provider hands you a Retry-After header, honor it by setting the next retry delay explicitly instead of letting blind backoff guess.

The deeper nuance that matters for agents is what should not be retried at all.

Some failures are not transient. If a step produced output that fails a hard validation, retrying the identical step usually reproduces the identical failure while burning tokens each time. You want those to stop retrying and escalate, either to a different branch of the workflow or to a human.

We model this with a small set of non-retryable error types. When a step raises one of those, Temporal does not spin; the workflow catches it and decides what to do next.

Retry transient infrastructure failures. Escalate deterministic content failures. Retrying an agent that is confidently wrong just pays to be told the same wrong thing again.

Agents lie about success, so put mechanical gates between them

This one is philosophical and extremely practical.

Agents are optimistic narrators. Ask a coding agent whether it ran the type checker and it will often tell you, sincerely, that everything passed. Then you look, and the workspace has half a dozen compile errors from hallucinated APIs.

We learned the hard way that a prompt instruction like “you must run the build and confirm it is green before returning” is necessary but not sufficient. Agents skip mandated steps and report success anyway.

The fix is to stop trusting self-reports and add deterministic gates:

A prompt-level requirement is one refactor away from being ignored. A mechanical gate is enforcement.

Concretely: after a code-writing step returns, an activity independently runs the real type check and build against the resulting workspace. If it fails, the step fails, with the compiler output attached, and the workflow escalates instead of advancing. The agent’s opinion of its own work is an input, not the verdict.

This also fixed an observability lie we will get to below: once you have an external gate, “the agent said it passed” and “it actually passed” become two separate, comparable facts.

Machine-parseable contracts between agent and orchestrator

Related trap: your orchestrator has to make control-flow decisions based on what an agent produced. Pass or fail. Revise or approve. If you parse that decision out of free-form prose, you are one polite paragraph away from a stuck workflow.

We had a review step escalate the entire run for a dumb reason: the agent wrote a perfectly good summary of its fixes but never emitted the exact verdict token the orchestrator was grepping for. No token, no decision, escalate.

The boundary between an agent and your control flow must be a strict, machine-parseable contract. Require an exact verdict line, and validate it. If the agent does not produce it, that itself is a failure, not a maybe.

Prose is for humans. Control flow needs a token.

Observability: wiring LLM tracing into a durable engine

Temporal gives you excellent visibility into its own world: every workflow, every step, every retry, every input and output, all queryable. What it does not give you is the thing you most want to see in an agents SDLC: what the model actually did. Tokens, prompts, responses, tool calls, cost per step.

For that we plugged in a dedicated LLM tracing tool (Langfuse, self-hosted on the same EKS cluster, in our case). You end up running two observability planes: one for orchestration and durability, one for prompts, tokens, and cost. Getting them to line up was more friction than expected.

Screenshot from our Langfuse, showing a cost breakdown of a specific LLM call

Span propagation across process boundaries. Our model calls happen inside a subprocess spawned by the step. To make the model’s own spans nest under the right workflow-and-step span, we inject the trace context (the W3C trace parent and state) into the subprocess environment. Without that, you get two disconnected trees: Temporal’s view of the run, and a pile of orphan LLM spans with no idea which run or step they belong to. Correlating them by hand is misery. Propagate the trace context explicitly, or you do not really have distributed tracing, you have two separate logs.

The tracing backend is infrastructure you now own. This is the part nobody warns you about. Your LLM tracing tool has its own database, its own object storage, its own failure modes. Ours quietly stopped ingesting traces for a long time. The dashboard showed zero traces and everyone assumed the instrumentation was broken. The real cause: the object storage backing the tracing tool had run out of inodes (not out of space, out of inodes, because it wrote millions of tiny objects), and every write was failing silently behind a generic error. The fix was half cleanup and half migrating that storage to something with proper lifecycle management.

Your observability stack is production infrastructure. If you do not monitor the thing that monitors your agents, you will discover it is down only when you desperately need it.

Watch out for observability that lies. The best bug of the whole project: a per-step metric that counted how many tool calls the agent made. It always reported zero. For a while we believed the agents were doing nothing. They were doing plenty; the counter was checking the wrong attribute shape on the SDK’s response objects and silently matching nothing. A step that wrote dozens of files logged “0 tool calls.” An instrument that reports zero when the truth is fifty-five is worse than no instrument, because it actively misleads triage.

Verify your instrumentation against ground truth at least once. A metric you never validated is a rumor.

Versioning workflows when the agent logic changes weekly

Agent pipelines change constantly. You reorder steps, add a gate, change a prompt, split one step into two. Meanwhile, Temporal may have long-running executions in flight that started under the old code. Determinism means you cannot just change the workflow and redeploy; a replay of an old run against new code will diverge from its recorded history and fail.

Temporal has first-class tools for this: patch-style version gates you place in the workflow code, and worker-level versioning (build IDs pinned to executions) so old and new runs route to compatible workers. Recent guidance leans toward worker versioning as the default for teams that can run versioned deployments. The practical takeaways:

If your runs are short-lived, the simplest safe move is to drain old runs before rolling out incompatible workflow changes.
If your runs are long, you must use the versioning primitives deliberately. Every incompatible change to workflow (not activity) code needs a version gate.
Changing activity code is far more forgiving than changing workflow code. Push behavior into activities partly for this reason: it is the layer you can evolve freely.

This is a recurring tax. Frequent agent-logic changes and long-lived durable executions are in natural tension, and you pay for it in versioning discipline.

Human-in-the-loop is a feature, not an afterthought

A fully autonomous agents SDLC that opens a pull request and merges it with nobody looking is a great way to ship confident nonsense to production.

Temporal’s signals, updates, and timers make human checkpoints clean to model. A workflow can pause and wait for an external approval, with a timeout that escalates if nobody responds. We keep humans at the high-leverage gates: the plan looks right before code gets written, and the final change gets human review before it lands.

Autonomy is a dial, not a switch. Put the human where a wrong decision is most expensive, and let the machine run everything cheap and reversible.

Local development and debugging nondeterministic agents

Testing a nondeterministic thing on a deterministic engine is inherently awkward, and no amount of tooling fully removes the awkwardness.

Temporal’s test framework is genuinely good: a time-skipping test server, and replay tests that catch determinism violations before they reach production. Use both. Replay tests in particular are your safety net against accidentally breaking workflow determinism during a refactor.

But the agent itself will never be reproducible. Replay tests catch code drift, not model drift. Two things made debugging bearable:

Persist the raw model exchanges. We dump every request and response body to disk per run and per step. When a step misbehaves, the transcript is right there. This is how we proved the “0 tool calls” metric was lying and how we caught agents writing to the wrong output paths.
Structured, queryable step records. Every step writes a record with its inputs, outputs, token counts, cost, and duration. When something goes wrong three steps back, you are reading records, not scrolling logs.

You cannot make the agent deterministic. You can make its behavior fully auditable after the fact. Do that.

The part nobody budgets for: this is a system you run forever

Here is the theme I promised to keep hammering.

The workflow code is not the product. The running system is the product, and it never stops needing you. A partial, honest list of what “operating an agents SDLC on Temporal” actually involves:

A worker fleet to deploy, scale, and roll safely on every change.
Version pinning between workflow code, worker builds, and in-flight executions.
A persistence store for Temporal itself (Amazon RDS, in our case), with its own capacity and backup story.
The object storage layer for artifacts, with lifecycle and cleanup.
The LLM tracing stack, with its own database and storage to monitor.
Cloud credentials and workload identity (IAM roles and IRSA) for every component that touches a model API or a bucket, which rotate and expire and break at the worst time.
Monitoring and alerting on all of the above, plus somebody on call when a run wedges at 2am.

During this project we lost time to expiring AWS SSO sessions, VPN and DNS resolution flakiness getting to the cluster, identity federation for model-provider auth breaking after a platform change, and an Amazon RDS password rotation that desynced and crashlooped a service. None of that is agent work. All of it is the cost of running the system that runs the agents.

Budget for operations as a standing cost, not a one-time setup. If you cannot staff the “run it forever” part, do not start the “build it once” part.

A comparison, briefly

Temporal is not the only way to orchestrate agents, and it is heavier than the alternatives.

It helps to separate three different jobs people all call “orchestration”:

Batch and data scheduling (the DAG schedulers): great at “run this graph of jobs on a schedule,” not built for durable per-step replay or human waits.
Agent frameworks and graph libraries (the AI-native tools): purpose-built for the agent loop itself, with streaming, memory, tool routing, and LLM evals baked in. They let you wire up multi-step agent flows in a single process with almost no infrastructure.
Durable execution engines (Temporal, and lighter cousins backed by a database you already run): built to survive a crash mid-run, resume exactly where you were, retry one step with backoff, wait days for a human signal, and keep a queryable record of thousands of concurrent runs.

The AI-native frameworks give you streaming, memory, and prompt management for free, and Temporal makes you build all of that yourself. What Temporal gives you in return is durability that the lightweight tools simply do not have. A very common and very sane pattern is to run both: a durable engine for the long-running orchestration, and an agent framework for the reasoning loop inside a step.

That durability is the entire reason to pay Temporal’s tax.

Choose a durable engine when a half-finished run is expensive and “just start over” is not acceptable. Choose something lighter when it is.

Before you commit: a checklist

Are your steps genuinely a deterministic pipeline, or are you forcing structure onto something exploratory? Temporal rewards the former and fights the latter.
Can you keep every side effect, especially model calls, inside activities without contorting your design?
Have you designed artifact hand-off through object storage from the start, with references in the workflow and nothing large in the payload?
Do you have a single-step rerun path and a cheap-model full run, with guardrails so test paths never hit production?
Is every agent-to-orchestrator decision a strict, machine-parseable contract?
Do you have mechanical gates that verify agent claims instead of trusting them?
Is your LLM tracing stack monitored like the production infrastructure it is?
Do you have the people and the on-call maturity to run this indefinitely?

If most of those are “yes,” Temporal will serve you extremely well. If most are “no,” you will spend more time fighting the engine than shipping features.

Conclusion: worth it, with eyes open

Turning a manual, prompt-by-prompt agent workflow into a durable orchestrated SDLC was one of the highest-leverage things we built. Runs survive crashes. Failed steps retry themselves. A run that dies at step nine resumes at step nine. We can rerun one phase for pennies, trace every model call to its cost, and gate the agents behind checks they cannot talk their way past.

But none of that came from the happy-path tutorial. It came from the payload limit, the cross-pod disk trap, the lying metric, the silent tracing outage, the confidently-wrong agents, and the steady operational drip of auth, DNS, and rollout problems.

Temporal makes an agents SDLC durable. It does not make it easy, and it does not make it self-maintaining. It is a system, and systems have owners.

If you have the engineering maturity and the appetite to own infrastructure, orchestrating your agents SDLC on a durable execution engine is absolutely worth it. If you were hoping for a set-and-forget button, this is not that, and anyone who tells you otherwise has not run one in production.

We did it by hand, we made it deterministic, and then we spent the real effort making the determinism survive contact with reality. That last part is the job.

The Human-in-the-Loop Factor: Partnering With Amazon Q During a Production Incident

Yedidya Schwartz — Sat, 23 Aug 2025 17:24:49 +0000

Abstract

A production outage hit our Debezium cluster that streams MongoDB change events into BigQuery. The first time it happened we realized only hours later, which meant data loss for everything that did not make it to BigQuery. When it happened again, I used Amazon Q to guide the investigation, propose a proven remediation, and help me implement and validate an automated fix that shrinks the data loss window from minutes to seconds. More importantly, this was not about asking the AI to ״solve the incident״ in one shot. I broke the problem into structured steps: letting Q investigate the cluster, reviewing remediation options, simulating failure in staging, validating deployments, and finally summarizing the process for the team.

The outcome was threefold: a practical safeguard that saved real data, a hands-on way to use AI as a Kubernetes CLI companion while learning commands myself, and a strong reminder that human judgment is still needed in the AI feedback loop. With the right thinking cues, the AI delivered context-aware, reliable help that went the extra mile during a critical production incident.

What is Debezium, and our case study

Debezium is a change data capture platform. In our setup, Debezium Server tails MongoDB Atlas change streams and writes events to Google BigQuery. The value is simple: low latency ingestion of document changes into an analytics warehouse where downstream models and dashboards expect near real time data.

Source: MongoDB Atlas change streams
Transport and processing: Debezium Server running on GKE as a StatefulSet
Sink: BigQuery
Goal: keep the pipeline hot so analytics stays fresh

An unfamiliar cluster, and an unavailable owner

I've just had an internal transition in the company to take part in the data team infrastructure efforts, and I'm not familiar with all their systems yet. The teammate who normally owns this cluster was unavailable after a bike crash, so I was missing context and access. Amazon Q bridged that gap quickly by helping me authenticate to GCP, fetch GKE credentials, and verify connectivity.

Me: I want to connect to my gcp account and add to my local kube config the gke cluster debezium-server-stg in us-east1

Amazon Q: I can help you connect to your GCP account and add the GKE cluster to your local kubeconfig. Here are the steps:
Amazon Q: I will run the following shell command: gcloud auth login
Amazon Q: gcloud container clusters get-credentials debezium-server-stg --region us-east1
Amazon Q: Your current kubectl context is now set to: ***

Note: Some of the chat excerpts with Amazon Q have been significantly shortened and summarized.

The article highlights only the essential parts to convey the ideas clearly and keep the reading experience smooth, without overwhelming the reader with repetitive technical details.

Background to the incident and why we needed automation

The first occurrence went unnoticed for a few hours. Everything between the initial error and the manual remediation was lost to BigQuery. After it fired again, this time with better alerting, we decided to automate the remediation so the next failure would auto heal in seconds, not minutes.

Step 1. Asking Amazon Q what happened

I asked Q to take the lead on understanding the failure, even though I already suspected the cause. The idea was to let him explore the pod state, events, and logs on his own, so his remediation suggestions would be grounded in context.

Me: I need you to look at debezium-server-prod cluster, specifically what happened around 2am-3am Israel time. the Debezium stopped functioning with Atlas Mongo CDC integration, I need to understand the root cause

Amazon Q: I'll help you investigate what happened to the Debezium pod around 2am–3am Israel time today. Let me start by examining the cluster and finding the Debezium pod.

kubectl get pods -A | grep -i debezium
kubectl describe pod debezium-gke-prod-0 -n debezium-prod
kubectl logs debezium-gke-prod-0 -n debezium-prod --tail=100

Evidence from logs:

... Resume of change stream was not possible ...
... with error 286 (ChangeStreamHistoryLost) ...
... the resume point may no longer be in the oplog ...

Step 2. Asking Amazon Q for possible solutions

Once Q had the picture, I asked for potential remediations. He quickly recommended few options, and we went with its recommendation of modifying the liveness probe to detect ChangeStreamHistoryLost errors and reset the offsets file so Debezium could restart cleanly.

Me: my friend removed the offset file to solve it from inside the pod, that's what solved the issue. I would like to have an auto remediation for such issue if it happens again. what do you suggest?

Amazon Q: Great insight. Your friend's solution makes perfect sense. Removing the offsets file forces Debezium to start fresh instead of trying to resume from an invalid position. Let me suggest several auto-remediation approaches for this specific issue.

Amazon Q: Option 1: Smart liveness probe (recommended). Detect specific MongoDB resume failures such as ChangeStreamHistoryLost and self heal by removing the problematic offsets file, then exit with non zero to trigger a restart.

Amazon Q: Option 2: Sidecar container. Run a lightweight helper container in the same pod that tails the Debezium logs and triggers cleanup actions if the error is observed. This separates remediation logic from Debezium itself.
I considered other solutions, but the liveness probe approach was both simplest and fastest to implement while still giving us the observability and safety we needed. It required no extra infrastructure, no sidecars, and no external controllers - just Kubernetes doing what it already knows best.

Amazon Q: Option 3: Kubernetes CronJob or Operator. Periodically check Debezium health at the cluster level, and when unhealthy, reset the offsets and restart the StatefulSet. This decouples remediation from pod lifecycle.

Amazon Q: Option 4: External monitoring system integration. Alerting via Prometheus or Cloud Monitoring can trigger a remediation workflow (for example, a Cloud Function that restarts the StatefulSet).

Step 3. Asking Amazon Q to implement the solution

After seeing Q’s initial suggestion, I asked him to help me actually build the probe inside our Debezium StatefulSet. At first, Q suggested a very simple log check that would grep for ChangeStreamHistoryLost and remove the offsets file. That worked in principle, but it was too naive for production.

I directed Q to think deeper:

handle startup when the log might not exist yet,
make sure we back up the log with a timestamp for debugging,
prune old backups to avoid PVC bloat,
truncate the log in place after remediation,
and catch generic errors on the last line without panicking on old log noise.

Here is how the probe evolved through our conversation:

Me: Let's add a liveness probe to automatically detect the error and remove the offset file if it happens.

Amazon Q: Sure. We can create a simple probe that tails the log and looks for 'ChangeStreamHistoryLost'. If detected, remove offsets.dat and exit 1.

Me: That's a start, but we need something more comprehensive. It should handle missing logs at startup, back up the log with timestamps, clean up old backups, and truncate the file. Also, let’s still fail if the last line shows a fresh ERROR.

And here is the final version we built together:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      LOG="/debezium/data/logs/debezium.log"

      # Healthy if log isn't there yet (startup/warmup)
      if [ ! -f "$LOG" ]; then
        echo "Log file not found; assuming healthy."
        exit 0
      fi

      # Detect the resume issue only in the current log's tail
      if tail -n 80 "$LOG" | grep -E -q 'ChangeStreamHistoryLost|Resume of change stream was not possible'; then
        echo "AUTO-REMEDIATION: Detected MongoDB oplog resume issue"

        # Timestamped backup, UTC for portability
        TS="$(date -u +%Y%m%dT%H%M%SZ)"
        BAK="/debezium/data/logs/debezium.log.backup-$TS"

        # Best-effort backup
        cp "$LOG" "$BAK" 2>/dev/null || cat "$LOG" > "$BAK" || true
        echo "Backed up current log to: $BAK"

        # Clear offsets and truncate the active log in place
        rm -f /debezium/data/offsets.dat || true
        : > "$LOG" || true

        # Optional: prune old backups (>7 days) to avoid PVC bloat
        find /debezium/data/logs -maxdepth 1 -type f -name 'debezium.log.backup-*' -mtime +7 -print -delete 2>/dev/null || true

        # Trigger immediate restart
        exit 1
      fi

      # Optional: fail on a fresh last-line ERROR
      if tail -n 1 "$LOG" | grep -q 'ERROR'; then
        echo "ERROR found on last line; failing liveness."
        exit 1
      fi

      echo "Healthy."
      exit 0
  initialDelaySeconds: 60
  periodSeconds: 5
  timeoutSeconds: 5
  failureThreshold: 1

Besides the script itself, the probe configuration had to be changed too, for reducing the reaction time of Kubernetes so remediation kicks in as soon as possible.

Originally, the probe ran every 10 seconds (periodSeconds: 10) and allowed 5 failures (failureThreshold: 5) before marking the pod unhealthy. That means the pod could sit in a broken state for up to ~50 seconds before Kubernetes restarted it. In a streaming pipeline like Debezium, 50 seconds of downtime means thousands of events lost.

By tightening it to:

initialDelaySeconds: 60
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 1

I achieved two key improvements:

Faster detection: the probe now samples every 5 seconds, so it sees the error quickly.
Immediate remediation: failing on the first detection (failureThreshold: 1) means Kubernetes restarts Debezium right away, instead of waiting for repeated confirmation.

This combination shortens the outage window from potentially close to a minute down to just a few seconds, which is critical for minimizing data loss in CDC pipelines.

By pushing Q step by step, I ended up with a much more resilient solution than the first draft. Instead of a hacky quick fix, the probe became a well-rounded, production-ready safeguard that not only healed the cluster automatically but also preserved valuable debugging context for the team.

Step 4. Asking Amazon Q to explain the existing CD for the Debezium repository

Before rolling the change, I asked Q to explain how our CD pipeline deploys manifests. He parsed the repo, confirmed the StatefulSet and config structure, and reassured me that applying this change through the CD path would be consistent and safe.

Here is the exact guidance Q gave me after reviewing the GitHub Actions in the CDCStreaming project. He explained how the release trigger works and how staging deployments are normally done:

Me: check the github actions in CDCStreaming project, how should I deploy to staging cluster?

Amazon Q: Perfect! Now I understand the deployment process. Based on the GitHub Actions workflow, here's how you should deploy to the staging cluster:

Deployment Process for Staging:
Option 1: Proper Git-based Deployment (Recommended)

Create a branch from main (not main itself, since manual triggers from main are blocked)

Commit your changes to the StatefulSet

Create a release targeting that branch (not main)

The GitHub Action will:
• Set environment variables for staging (deployment_env=stg, cpu=500, memory=2)
• Replace template variables in the YAML files
• Deploy to the debezium-server-stg cluster
• Restart the pod

Option 2: Manual Deployment (Quick Fix)

Since you want to test this now, let me help you deploy manually using the same process the GitHub Action uses.

Step 5. After I shipped to staging, asking Amazon Q to verify

Once the change was deployed to staging via CD, I asked Q to double check everything. He validated the probe configuration, pod health, and events to confirm that staging was stable.

Me: I applied it with CD. please confirm that the health checks look ok  

Amazon Q: Great! I see the pod restarted successfully on the staging cluster. The enhanced liveness probe is in place with:  
- Correct log path `/debezium/data/logs/debezium.log`  
- Detection of `ChangeStreamHistoryLost` errors  
- Auto-remediation with backup, offset removal, and restart trigger  
- Cleanup of old backups to prevent PVC bloat  

Debezium is streaming normally, no probe failures or unexpected restarts. The auto-remediation is active and will trigger within 10 seconds if the error reappears.

Step 6. Asking Amazon Q to simulate the remediation on staging

Instead of trusting the probe blindly, I asked Q to simulate the failure by injecting the ChangeStreamHistoryLost error into the logs. This way we could watch the remediation in action.

Me: I want to simulate it. Please inject the error and let’s see that it works as expected  

Amazon Q: Great idea! I’ll inject a `ChangeStreamHistoryLost` error into the Debezium log and monitor the probe.  

→ Injected the error at 19:07:03Z  
→ Liveness probe detected it immediately and triggered remediation  
→ Pod restarted (restart count went from 0 → 1)  
→ Offset file removed, backup created (`debezium.log.backup-20250812T190703Z`)  
→ Debezium started cleanly: “No previous offsets found”  
→ Streaming resumed successfully  

🎉 **Auto-remediation test passed**: error detected, pod restarted within ~10s, offsets cleaned, and streaming resumed without manual intervention.

Step 7. After I shipped to production, asking Amazon Q to verify again

With confidence from staging, I deployed to production. Q helped me validate the production pod with kubectl describe and logs, confirming the probe was present and the pod was healthy.

Step 8. Asking Amazon Q to summarize the process for the team

Finally, I asked Q to produce a summary I could share internally as a knowledge base entry. The summary captured the purpose, steps, and indicators of success in a concise way. I asked it to output in Confluence format, so then I just copy-pasted it and did some tunings to match the team atmosphere.

What is still in progress

The long term solution is still under investigation. This issue touches MongoDB change stream history, Debezium offsets, and operational safeguards. The automated remediation already saved a large number of events and dramatically reduced business impact, while also saving many human hours during incidents.

Lessons learned

A small safeguard makes a big difference

A small, targeted probe saved us a lot of data and hours. Incident response is a team sport, and AI can be a productive teammate when you use it intentionally. We kept the remediation simple and observable, tested it by simulating the real failure in staging, and delivered it through the standard CD path. That discipline gave us confidence.

How to Work With AI in an Incident

One important takeaway is that you cannot just tell an AI assistant ״solve the incident for me.״ You need to structure the thinking and let it walk with you step by step. For this incident, I first let Q investigate the cluster alone, even though I already knew what had happened. That gave him context and allowed his remediation suggestions to be sharper because he had ״felt״ the problem himself. Then I asked him for solutions, reviewed them, and deliberately chose the one that fit best. I pushed further by asking him to simulate the failure in staging before touching production, and I asked him to verify every deployment along the way. This orchestration of investigation, proposal, simulation, and verification turned the interaction into a holistic ecosystem. With the right thinking cues, AI goes the extra mile and becomes a true partner when dealing with production critical issues.

Treating Amazon Q as a Kubernetes CLI learning companion

One thing that stood out to me in this incident is how natural it felt to use Amazon Q, or really any AI CLI tool, as an interface to Kubernetes. When you are dropped into a new cluster you are not familiar with, it can be hard to know what command to run or even how to authenticate. Q bridged that gap: it helped me install the GCP CLI, log in to the right project, and fetch kubeconfig for the correct GKE cluster. From there, it navigated the cluster to find the exact issue without me having to recall every kubectl nuance.

A small but important detail is that Q always pauses before running a command. By default, you must approve each one, unless you explicitly type t to grant temporary trust for the session. I personally do not give it blanket approval. There are two reasons: first, I want to learn the commands myself and understand the thinking process; second, I want to be sure nothing breaks. Even though I told Q upfront ״only read, don’t change anything,״ I treat this as advisory rather than guaranteed enforcement. Keeping explicit control allowed me to learn new Kubernetes command practices while staying safe.

My perspective on relying on AI

As long as complex systems still require human intervention for edge cases, and their developers do not make a clear claim that the system can self-heal under failure, we will continue to play an active role in the feedback loop with AI tools.

For example, Debezium’s integration with MongoDB Atlas CDC - the case I described - faced an unusual failure. It was likely caused by a third party, outside the Kubernetes cluster. But the key point is this: the way I guided the AI on how to assist me was essential. Without that structured interaction, the problem would not have been solved in practice.

I highly recommend watching this great talk - where Andrej Karpathy, a founding member of OpenAI, explains how software is changing (again) and how important is the "Human-in-the-Loop" factor.

Until the next big leap arrives, which may not be far away, we must keep involving our own thinking in AI-driven processes, alternating between human judgment and AI automation.

From Step Functions to Temporal on EKS: Durable Workflows at Scale Without Breaking the Bank

Yedidya Schwartz — Thu, 03 Apr 2025 19:11:08 +0000

Lesson learned: Always tie your architecture decisions to your business needs. We started by chasing stability — and ended up chasing cost, once scale kicked in. Don’t be afraid to shift your mindset from using external managed services to running self-hosted infrastructure. At a certain scale, this shift is critical to keep costs manageable and sustainable.

What We Needed

Trigger a timer,

Wait X seconds,

Then run a function with a specific payload.

Seems simple—until you're doing it millions of times per day, across two cloud providers, with retry logic, second-level delay precision, and full observability.

Our Journey: Azure Durable Functions → AWS Step Functions → Temporal Cloud → Temporal Self-Hosted

We've been through all the usual options:

✅ Azure Durable Functions

Worked—until traffic spiked. Then: dropped messages, hung workflows, and zero transparency. Not built for our scale.

✅ AWS Step Functions

Solid and stable. We liked it. But the more we used it, the more it cost us. Delay-heavy flows with retries added up fast. It wasn't sustainable.

❌ AWS Step Functions Express

Total execution time is capped at 5 minutes. Our flows often wait far longer than that. Not an option.

❌ EventBridge

We considered using EventBridge for scheduled triggers. But the docs admit to ~1-minute precision. We needed 1–2 second precision.

✅ Temporal Cloud

We did a full POC. It worked beautifully. But the pricing model, based on actions, didn’t fit our use case. At ~200M workflows/month (each including delay, retries, and an activity), which equates to 400M actions/month in our case, we were looking at an extremely high bill.

The Cost-Saving Move: Self-Hosted Temporal on EKS

After a long period of embracing managed services wherever possible, we realized that, in this case, going self-hosted was the only way to support the business at scale without letting cost become a blocker. So we deployed Temporal on our own infrastructure (EKS) using the official Helm charts. With some tuning, it worked smoothly.

✅ ~80% cost savings

✅ Full infrastructure and application-level control

✅ Scale without pricing surprises

Even better: our cost doesn't scale with traffic.

We configured EKS node groups and Aurora Postgres to support multiple times our current load, all for around $1,500/month.

Most of our infra is either autoscaled or manually tuned, and our cost curve is flat.

Step Functions Implementation: The Starting Point

Before diving into Temporal, here’s what our implementation looked like using AWS Step Functions.

The purpose of sharing this is to show a side-by-side with our Temporal setup later—so you can clearly see the evolution and differences in approach.

In Step Functions, you must first define your State Machine ahead of time, including all steps, transitions, and timers. This pre-declaration requirement introduces a different kind of workflow lifecycle—configuration first, execution second.

Here’s how we structured the creation of a timer-based workflow:

public Workflow CreateTimerWorkflow<T>(TimeSpan waitTime, string endpoint, T httpBody, string userId = null)
{
    return new Workflow
    {
        Id = _timerStateMachineArn,
        Request = new TimerRequest<HttpCallRequest<T>>
        {
            WaitTimeSeconds = (int)waitTime.TotalSeconds,
            Data = new HttpCallRequest<T>
            {
                Body = new MicroserviceRequest<T>
                {
                    Metadata = new RequestMetadata
                    {
                        UserId = userId,
                        RequestId = Guid.NewGuid().ToString()
                    },
                    RequestContent = httpBody
                },
                Url = GenerateUrl(endpoint),
                AppCode = new Dictionary<string, string>
                {
                    { "code", _configuration.GetServiceAuthCode() }
                }
            }
        }
    };
}

To trigger this workflow, we used a runner with built-in retry logic and AWS SDK:

await _client.StartExecutionAsync(new StartExecutionRequest
{
    StateMachineArn = stateMachineArn,
    Input = input,
    Name = requestId // I used requestId as execution name to make it easy to debug later
});

And here’s what the State Machine definition looked like:

{
  "Comment": "A description of my state machine",
  "StartAt": "Wait",
  "States": {
    "Wait": {
      "Type": "Wait",
      "InputPath": "$",
      "SecondsPath": "$.waitTimeSeconds",
      "Next": "Call third-party API",
      "OutputPath": "$.data"
    },
    "Call third-party API": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Method": "POST",
        "RequestBody.$": "$.body",
        "Authentication": {
          "ConnectionArn": "CONNECTION_ARN_PLACEHOLDER"
        },
        "ApiEndpoint.$": "$.url",
        "QueryParameters.$": "$.appCode"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "MaxAttempts": 30,
          "JitterStrategy": "FULL",
          "Comment": "Main Retrier",
          "IntervalSeconds": 20,
          "BackoffRate": 1
        }
      ],
      "End": true,
      "InputPath": "$"
    }
  }
}

Why Temporal is Different

In Temporal, workflows are implemented entirely in code using the SDK. You don’t need to predefine a workflow structure or declare a state machine. The only setup required is creating a namespace—everything else happens dynamically at runtime.

This model shifts the ownership of orchestration from infrastructure (like JSON/YAML flow definitions) to application code. It provides flexibility for teams that want to iterate on workflow logic through code deployments and enables versioning, type safety, and testing in the developer's native language.

Both models — predefined state machines in Step Functions and code-first workflows in Temporal — represent different philosophies in how orchestration can be managed.

Temporal lets you write workflows as regular code — with built-in durability, retries, and state recovery.

Our use case was a perfect fit, just like it was with Step Functions: wait, then trigger logic. But instead of using the standard Workflow.Sleep() pattern, we took it a step further.

Saving on Actions: Using `StartDelay` Instead of a Timer

The typical Temporal pattern looks like this:

await Workflow.Sleep(TimeSpan.FromSeconds(X));
await Activity.CallAsync(() => DoWork());

But that Sleep adds an extra action. In Temporal Cloud, that would cost more. During our POC with Temporal Cloud, we realized this optimization could significantly reduce cost.

So instead, we used StartDelay when starting the workflow:

await _client.StartWorkflowAsync(
    (TemporalCallApiWorkflow wf) => wf.RunAsync(timerRequest.Data.Url, requestBody),
    new WorkflowOptions(
        id: requestId,
        taskQueue: _configuration.GetTemporalWorkflowOptions().TaskQueue
    )
    {
        StartDelay = TimeSpan.FromSeconds(timerRequest.WaitTimeSeconds)
    }
);

Wrapped with exponential backoff:

await CommonUtils.RetryCallWithExponentialBackoff(
    async () => await _client.StartWorkflowAsync(...),
    "TemporalWorkflowExecution",
    _logger,
    ExecutionRetries
);

This pattern reduces actions per workflow from 3 to 2, saving ~33% per execution in the cloud model, and even better—on our self-hosted setup, it reduces the performance overhead by 33% per workflow.

We configure the system with a unified structure that makes environment switching easy:

{
  "TargetHost": "temporal-frontend.temporal.svc.cluster.local:7233",
  "Namespace": "default",
  "WebhookBaseUrl": "http://my-app.test.svc.cluster.local",
  "TaskQueue": "my-api-caller-task-queue",
  "IsShadowMode": false
}

Our Infra Stack

Cluster: EKS
Database: Aurora PostgreSQL (persistence + visibility: the two required Temporal DBs)
Ingress: Internal-only access inside AWS, plus an NLB to allow external access to Temporal for triggering workflows. Exposing the frontend service via ALB didn’t work
Web UI: VPN-only access
Observability: Datadog dashboard using the official integration
Config: Git-managed values.yaml via ArgoCD

Stress Testing: What Broke, What We Fixed

We used Maru to simulate production load, and the Scaling Temporal doc to interpret the results via our Datadog dashboard.

429 "Too Many Requests"

We hit the frontend.namespaceCount limit early.

→ Fixed using:

server:
  dynamicConfig:
    frontend.namespaceCount: 
    - value: 9999999
      constraints: {}

Pod Resource Starvation

Frontend, history, and worker pods couldn’t keep up.

→ Tuned:

frontend:
  service:
  resources:
    requests:
      cpu: 1
      memory: 3Gi
    limits:
      memory: 3Gi

Aurora Lagging

→ Upgraded the instance class. CPU usage remains high under stress, but it’s no longer a bottleneck.

History Shards

We're still on the default history.shards value. Planning to scale it later—even though it requires a DB reset. Since we're still in shadow mode, as I'll explain in the next paragraph, that's acceptable for now.

Tuning the number of shards is a critical scaling lever in Temporal.

Highly recommend reading: Choose the Number of Shards in Temporal History Service

During stress testing, we saw shard latency spikes, signaling a need to increase shard count. We're waiting to validate this under real traffic in shadow mode before committing to a reset.

Migration Strategy: Shadow Mode

We’re not cutting over cold turkey. Instead, we’re shadowing Step Functions:

Temporal runs the full logic
Sends results to a mock endpoint, including the expected trigger timestamp
If the delay > 2 seconds, we log it and investigate
We improve iteratively and expand the experiment
Once confidence is high, we flip the traffic

Zero risk. High signal.

Easy Rollback: Step Functions as a Safety Net

What made the decision easier was knowing we had Step Functions as a fallback.

They were already in place, already battle-tested. So we had no fear testing Temporal in production scenarios—because rollback was simple and safe.

Deployment Flow

All changes are GitOps-managed (separate gitops and gitops-production repos)
ArgoCD watches values.yaml
Manual sync for now — we like the control

Key Takeaway

Start with stability. Move to cost once you scale.

Once you decide to go self-hosted, don't fear it—but do it right.

Take time to deeply understand the product’s architecture. Run stress tests. Iterate.

Be ready with a fallback and observability before fully cutting over.

This approach gave us both confidence and clarity at each step.

This is an ongoing story — happy to share more details or dive deeper into any part of the article. I'll keep you posted about next steps and some more challenges I had during the process.

Feel free to reach out!