A Temporal workflow can run for minutes, days or months, and the code behind it will change while it runs. Temporal's answer is that workflow code must be deterministic: whenever a worker picks a workflow up, it replays the code against the recorded history, and the code has to make the same decisions it made the first time. In an earlier post I replayed one saved history against edited versions of a workflow to see what that check accepts.
This time I wanted the operational version of the question. If you deploy new workflow code while a hundred workflows are halfway through, which ones break, when do you find out, what does a broken workflow look like, and which of the usual fixes actually fixes it?
The setup
Everything ran locally on the Temporal CLI's development server, version 1.10.1 with Server 1.32.2, and the Python SDK 1.36.1.
The workflow has eight steps. Each step is an activity, and after each one the workflow sleeps for 15 seconds, so a single run takes about two minutes. Every activity appends a line to a log file before returning, which stands in for a real side effect and lets me count exactly what ran.
The new version adds one thing, an audit activity right after step 4:
for n in range(1, STEPS + 1):
await step(n)
if n == 4:
await audit()
await workflow.sleep(NAP)
That's a common kind of change: a new call in the middle of an existing flow. For each scenario I started 100 workflows over 60 seconds, so they were spread across every stage, then changed the workers 70 seconds after the first start. Afterwards I read every workflow's history and its side effects.
A plain restart broke everything past the change
Stopping the old worker and starting the new one split the workflows cleanly by how far they'd got:
- 58 hadn't reached step 4 yet. They carried on with the new code, ran the audit once and completed.
- 42 had already finished step 4. Every one of them broke.
The error was the same for all 42:
[TMPRL1100] Nondeterminism error: Activity machine does not handle this event: HistoryEvent(id: 44, TimerStarted)
The history says that after step 4 the workflow started a timer. The new code, replaying that history, wanted to schedule the audit activity instead. The two can't be reconciled, so the worker refuses to continue.
None of this happened at the moment of the deploy. Each workflow failed when its next timer fired and the new worker had to replay it, between 0.2 and 14.6 seconds after the switch.
A broken workflow doesn't look broken
I expected these workflows to fail. They didn't. All 42 stayed in the Running state for the rest of the run, about six minutes, and would have stayed there. Temporal treats a nondeterminism error as a failed workflow task, not a failed workflow, and keeps retrying the task in case a fixed worker turns up. By the end each one was on its 9th or 10th attempt.
The history doesn't make it obvious either. Each broken workflow had exactly one WorkflowTaskFailed event, however many retries followed. The retries showed up only in the attempt count of the pending task and as warnings in the worker's log. A dashboard that counts failed workflows would have shown zero.
A rolling deploy was worse
Next I ran the old and new workers side by side for a minute before stopping the old one, the way a rolling deploy would. I expected the new worker to take over gradually. It ran nothing at all, not one workflow task and not one activity.
Workflows stay on the worker that already has them in memory, so the old worker kept running all of them, on the old code, straight past step 4. When it stopped, the new worker had to replay every workflow that was still running, and 83 of them broke, between 60 and 75 seconds after the new worker had started. Only the 17 that finished during the overlap completed.
So the overlap didn't soften anything. It just let more workflows get past the point where the new code differs.
patched() is what works
The fix Temporal documents is to guard the change:
if n == 4 and workflow.patched("audit-after-4"):
await audit()
When a workflow runs that code for the first time, patched() returns true and records a marker in the history. When a worker replays a history that has no marker, it returns false, and the workflow follows the old path.
With the same restart, all 100 completed. The 58 that hadn't reached step 4 ran the audit. The 42 that had passed it skipped it, as their history said they should. No workflow task failed.
Rolling back made a different set stuck
Suppose you shipped the unpatched version anyway, saw the errors and rolled back. I tried that, going back to the old code 80 seconds after the bad deploy.
The 42 stuck workflows recovered. Their histories matched the old code again, and they completed. But by then 58 workflows had run the new audit step, and 8 of those had finished. The other 50 now had an audit in their history that the old code didn't expect, and they broke instead:
[TMPRL1100] Nondeterminism error: Timer machine does not handle this event: HistoryEvent(id: 44, ActivityTaskScheduled)
Across the deploy and the rollback, 92 of the 100 workflows hit a nondeterminism error at some point.
Patching after the fact didn't save them either
The obvious next move is to fix forward: deploy the patched version on top of the broken one. I ran the same bad deploy and replaced it with the patched code 80 seconds later.
The result matched the rollback, number for number. The 42 old-shaped workflows had no marker, so patched() sent them down the old path and they completed. The 50 that had run the unpatched audit also had no marker, because the unpatched code never wrote one. So patched() returned false for them as well, the code skipped an audit their history said had happened, and they broke.
There's no version of the code that can satisfy both groups. Neither history says which code produced it, and patched() only knows about markers that patched code wrote.
Resetting gets them out, and runs things twice
The way out for those 50 is a reset. Temporal can rewind a workflow to an earlier workflow task, throw away everything after it and run the rest again with the current code. I reset each one to the task that had scheduled the unpatched audit, with the patched worker running.
All 50 completed within 80 seconds. But a reset means re-running, and the log showed it: the 50 audits ran a second time, and the steps after step 4 ran again too. 175 step executions were duplicates of ones that had already happened. A reset is only safe if everything after the reset point can run twice.
In every other scenario, nothing ran twice. A stuck workflow simply stopped, which is the one comforting part of all this.
The numbers in one place
| Scenario | Completed | Stuck at the end |
|---|---|---|
| Restart onto unpatched code | 58 | 42 |
| Old and new workers overlapping for 60 s | 17 | 83 |
Restart onto patched() code |
100 | 0 |
| Unpatched deploy, then rollback | 50 | 50 |
| Unpatched deploy, then patched code | 50 | 50 |
| ...then a reset of the 50 | 100 | 0, with 225 side effects repeated |
What I got wrong
I expected the rolling deploy to be the gentle option, with the new worker picking up work bit by bit while the old one was still around. The new worker didn't run a single task or activity during the overlap. I didn't dig into why the activities stayed on the old worker too.
I also assumed a rollback was the safe undo. It is only if nothing ran on the new code in between, and with timers firing every 15 seconds, plenty did within 80 seconds.
And my first analysis reported that the workflows had failed two hours before the deploy. I'd converted the event timestamps with a function that treats them as local time, and my laptop is two hours ahead of UTC. Reading the raw seconds fixed it.
Caveats
This is the Python SDK on a local dev server, with one kind of change: an activity added in the middle. Removing a step would break in its own way, and the earlier post showed that reordering activities or changing a timeout doesn't trip the check at all. Each scenario ran once. I didn't try Temporal's worker versioning, which pins running workflows to the build they started on and is meant for exactly this problem.
What I'd take from it
- Every change to a workflow's sequence of steps needs
workflow.patched()from the first deploy. It's the only option here that lost nothing and repeated nothing. - Don't rely on a rollback or a later patch to undo an unpatched deploy. Both rescue one group of workflows by breaking another.
- Alert on workflow task failures or nondeterminism warnings, not on failed workflows. Broken workflows stay "Running".
- Running old and new workers side by side doesn't move workflows to the new code. It moves them further down the old path.
- If you need a reset, assume everything after the reset point runs again.
Top comments (1)
The 42 could have failed in CI instead of in production. Temporal's Python testing docs suggest this as a CI check: fetch the histories of recent open and closed workflows with
client.list_workflows(...).map_histories(), run them through aReplayerbuilt with the new workflow code, and fail the build on any error. Your unpatched audit would have hit the same TMPRL1100 against every history already past step 4, before any worker picked one up.The same check answers the rollback question. Before rolling back, replay the open histories against the previous build. Anything that already ran the new audit fails that replay, so you'd know it was 50 workflows, not zero, before choosing between a rollback and a reset.