DEV Community

swaroop kolasani
swaroop kolasani

Posted on

Debugging an Idempotency Bug in Temporal Worker Deployment Version Deletion

#backend #debugging #github #softwareengineering

I recently worked through Temporal issue #11539, which involved deleting Worker Deployment Versions when the parent deployment and version workflow had drifted out of sync.

The code fix was small. The interesting part was reproducing the inconsistent state and finding the correct place to make deletion idempotent.

The problem

A Worker Deployment stores version summaries in the parent deployment workflow.

Normally:

Worker Deployment
    |
    | DeleteWorkerDeploymentVersion
    v
Worker Deployment Version workflow
    |
    | delete succeeds
    v
Parent removes version from State.Versions
Enter fullscreen mode Exit fullscreen mode

The bug appears when the parent still lists a version even though the version workflow is already completed or missing:

DescribeWorkerDeployment
    -> version still listed

DescribeWorkerDeploymentVersion
    -> version not found

Version workflow
    -> completed or missing
Enter fullscreen mode Exit fullscreen mode

When DeleteWorkerDeploymentVersion runs, Temporal tries to send a delete-version update to that workflow.

Because the workflow can no longer accept the update, the server returns errors such as:

workflow execution already completed
Enter fullscreen mode Exit fullscreen mode

or:

workflow not found for ID: ...
Enter fullscreen mode Exit fullscreen mode

That error propagates back through the activity.

The parent only removes the version summary after the activity succeeds:

delete(d.State.Versions, args.Version)
Enter fullscreen mode Exit fullscreen mode

So the stale entry remains, and every later delete fails for the same reason.

This matters because the stale version can still count toward deployment limits and can prevent the entire Worker Deployment from being deleted.

Reproducing the bug

I reproduced the issue locally on Temporal main.

Initial state:

parent deployment workflow: RUNNING
version workflow:           RUNNING
parent versionSummaries:    contains v1
Enter fullscreen mode Exit fullscreen mode

I then tested two scenarios.

1. Version workflow completed

I terminated only the version workflow while keeping the parent running.

parent deployment: still lists v1
version workflow:  TERMINATED
Enter fullscreen mode Exit fullscreen mode

Deleting the version failed, with the server logs showing:

workflow execution already completed
Enter fullscreen mode Exit fullscreen mode

The parent summary remained.

2. Version workflow missing

I then removed the version workflow completely.

DescribeWorkerDeployment
    -> v1 still listed

DescribeWorkerDeploymentVersion
    -> not found

workflow describe
    -> workflow not found
Enter fullscreen mode Exit fullscreen mode

Deleting the version failed again, this time with:

workflow not found for ID: ...
Enter fullscreen mode Exit fullscreen mode

The important result was that retention was not required to reproduce the bug.

The failure exists whenever the parent version summary outlives the version workflow.

Choosing the fix

There were several possible places to handle the problem.

The frontend could swallow the NotFound error, but that would only hide the failure. The parent workflow would still keep the stale entry.

The parent could periodically reconcile missing version workflows, but that would introduce a broader cleanup mechanism for something deletion can handle directly.

The better question was:

Where does NotFound mean the delete operation has already reached its desired state?

The answer was Activities.DeleteWorkerDeploymentVersion.

If the version workflow is already gone, the version-side goal of deletion is already satisfied.

So deletion can safely treat that specific condition as success:

outcome, err := updateWorkflow(...)

if err != nil {
    var notFound *serviceerror.NotFound
    if errors.As(err, &notFound) {
        return nil
    }
    return err
}
Enter fullscreen mode Exit fullscreen mode

I intentionally left the generic updateWorkflow helper unchanged.

NotFound does not mean success for every Temporal update. It only has that meaning here because this is a delete operation.

Once the activity succeeds, the existing parent workflow logic can remove the stale version summary normally.

Regression testing

Before changing the production code, I added tests for:

version workflow completed  -> delete should succeed
version workflow missing    -> delete should succeed
unrelated service error     -> delete should still fail
Enter fullscreen mode Exit fullscreen mode

The first two failed before the fix, while the unrelated error still behaved normally.

I also added a functional regression test that:

  1. Creates a deployment and version.
  2. Closes the version workflow.
  3. Keeps the parent deployment running.
  4. Calls the public delete API.
  5. Verifies that the version disappears from the parent's summaries.

After adding the NotFound handling, the regression tests passed without changing unrelated deletion behavior.

What happened upstream

I opened PR #11638 with the idempotent deletion change and regression coverage.

While it was under review, a Temporal maintainer merged PR #11696, which applies the same core behavior: treat a missing Worker Deployment Version as already deleted.

My PR was therefore not the commit that landed upstream, but the issue was ultimately resolved using the same underlying approach.

Takeaway

The fix itself was only a few lines.

The real work was:

reproduce inconsistent state
        ↓
trace the failed state transition
        ↓
find the correct idempotency boundary
        ↓
write a failing regression test
        ↓
make the smallest semantic change
Enter fullscreen mode Exit fullscreen mode

The bug was not really about a missing workflow.

It was about two durable pieces of state completing the same delete operation at different times.

Once the child-side state is already gone, retrying deletion should converge toward the desired final state instead of requiring the deleted object to still exist.

References

Top comments (0)