DEV Community: swaroop kolasani

Debugging an Idempotency Bug in Temporal Worker Deployment Version Deletion

swaroop kolasani — Fri, 28 Aug 2026 16:08:05 +0000

I recently worked through Temporal issue #11539, which involved deleting Worker Deployment Versions when the parent deployment and version workflow had drifted out of sync.

The code fix was small. The interesting part was reproducing the inconsistent state and finding the correct place to make deletion idempotent.

The problem

A Worker Deployment stores version summaries in the parent deployment workflow.

Normally:

Worker Deployment
    |
    | DeleteWorkerDeploymentVersion
    v
Worker Deployment Version workflow
    |
    | delete succeeds
    v
Parent removes version from State.Versions

The bug appears when the parent still lists a version even though the version workflow is already completed or missing:

DescribeWorkerDeployment
    -> version still listed

DescribeWorkerDeploymentVersion
    -> version not found

Version workflow
    -> completed or missing

When DeleteWorkerDeploymentVersion runs, Temporal tries to send a delete-version update to that workflow.

Because the workflow can no longer accept the update, the server returns errors such as:

workflow execution already completed

or:

workflow not found for ID: ...

That error propagates back through the activity.

The parent only removes the version summary after the activity succeeds:

delete(d.State.Versions, args.Version)

So the stale entry remains, and every later delete fails for the same reason.

This matters because the stale version can still count toward deployment limits and can prevent the entire Worker Deployment from being deleted.

Reproducing the bug

I reproduced the issue locally on Temporal main.

Initial state:

parent deployment workflow: RUNNING
version workflow:           RUNNING
parent versionSummaries:    contains v1

I then tested two scenarios.

1. Version workflow completed

I terminated only the version workflow while keeping the parent running.

parent deployment: still lists v1
version workflow:  TERMINATED

Deleting the version failed, with the server logs showing:

workflow execution already completed

The parent summary remained.

2. Version workflow missing

I then removed the version workflow completely.

DescribeWorkerDeployment
    -> v1 still listed

DescribeWorkerDeploymentVersion
    -> not found

workflow describe
    -> workflow not found

Deleting the version failed again, this time with:

workflow not found for ID: ...

The important result was that retention was not required to reproduce the bug.

The failure exists whenever the parent version summary outlives the version workflow.

Choosing the fix

There were several possible places to handle the problem.

The frontend could swallow the NotFound error, but that would only hide the failure. The parent workflow would still keep the stale entry.

The parent could periodically reconcile missing version workflows, but that would introduce a broader cleanup mechanism for something deletion can handle directly.

The better question was:

Where does NotFound mean the delete operation has already reached its desired state?

The answer was Activities.DeleteWorkerDeploymentVersion.

If the version workflow is already gone, the version-side goal of deletion is already satisfied.

So deletion can safely treat that specific condition as success:

outcome, err := updateWorkflow(...)

if err != nil {
    var notFound *serviceerror.NotFound
    if errors.As(err, &notFound) {
        return nil
    }
    return err
}

I intentionally left the generic updateWorkflow helper unchanged.

NotFound does not mean success for every Temporal update. It only has that meaning here because this is a delete operation.

Once the activity succeeds, the existing parent workflow logic can remove the stale version summary normally.

Regression testing

Before changing the production code, I added tests for:

version workflow completed  -> delete should succeed
version workflow missing    -> delete should succeed
unrelated service error     -> delete should still fail

The first two failed before the fix, while the unrelated error still behaved normally.

I also added a functional regression test that:

Creates a deployment and version.
Closes the version workflow.
Keeps the parent deployment running.
Calls the public delete API.
Verifies that the version disappears from the parent's summaries.

After adding the NotFound handling, the regression tests passed without changing unrelated deletion behavior.

What happened upstream

I opened PR #11638 with the idempotent deletion change and regression coverage.

While it was under review, a Temporal maintainer merged PR #11696, which applies the same core behavior: treat a missing Worker Deployment Version as already deleted.

My PR was therefore not the commit that landed upstream, but the issue was ultimately resolved using the same underlying approach.

Takeaway

The fix itself was only a few lines.

The real work was:

reproduce inconsistent state
        ↓
trace the failed state transition
        ↓
find the correct idempotency boundary
        ↓
write a failing regression test
        ↓
make the smallest semantic change

The bug was not really about a missing workflow.

It was about two durable pieces of state completing the same delete operation at different times.

Once the child-side state is already gone, retrying deletion should converge toward the desired final state instead of requiring the deleted object to still exist.

References

Before You Upgrade Hardware, Fix the Software

swaroop kolasani — Wed, 01 Apr 2026 06:27:39 +0000

Better software algorithms can significantly improve effective memory efficiency, but only until the workload reaches a real hardware bottleneck.

The Misconception

Recent work such as Google's TurboQuant shows that software can significantly reduce memory pressure for specific workloads like LLM inference. At the same time, companies across the AI stack are investing in physical infrastructure such as power and chips to sustain growing compute demand. Meta has expanded its energy strategy, including major nuclear power agreements for AI-related infrastructure, while NVIDIA remains tied to the semiconductor path through advanced chip production and packaging. Together, these trends raise a broader question: if software can make systems more efficient, how often are we upgrading hardware before we have truly exhausted software optimization?

The Impact

The decision to optimize software or upgrade infrastructure is not only a technical choice. It affects cost, scalability, engineering time, and system reliability. When teams upgrade hardware too early, they often spend more without understanding the real bottleneck. Poor algorithms, inefficient memory use, weak caching, unnecessary data movement, or bad execution placement remain hidden behind larger machines. The system appears faster, but the underlying inefficiency remains unresolved.

The opposite mistake is also costly. If teams continue forcing software optimization after the workload has already reached a true hardware limit, they waste time chasing marginal gains. At that point, the system becomes harder to maintain, more fragile, and often less predictable under real load. What begins as optimization turns into complexity without meaningful return.

The real impact, then, is strategic: knowing when software can still recover efficiency, and when infrastructure upgrades are the only rational next step. Reaching that decision requires evaluating a wide range of technical scenarios, because the right answer depends on the workload, the bottleneck, and the tradeoffs the system can tolerate.

The Bottleneck

Before upgrading infrastructure, the first question is whether the workload is truly hardware constrained or simply inefficient. In many cases, software can recover substantial performance by reducing active memory pressure, improving execution strategy, or restructuring the workload itself. Compression and quantization can reduce memory use, better caching and locality can reduce wasted movement and repeated work, and stronger algorithms or data structures can change the resource profile of the system more than a hardware upgrade would.

Some of the most meaningful gains, however, come from architecture rather than low-level optimization. Software efficiency is not only about making a single process use less memory; it is also about deciding where work should run. Systems often become more efficient by separating latency-sensitive tasks from heavy background computation, reducing local resource pressure, and moving burstable workloads into environments better suited to them. Ephemeral cloud burst is one example of this approach: instead of permanently upgrading local hardware, a system can offload short-lived, compute-intensive, or memory-heavy work to temporary remote machines, using software to place the workload where the right resources already exist only when they are needed.

But software optimization has limits. Every efficiency technique introduces a tradeoff: compression adds processing overhead, caching consumes memory, recomputation trades storage for compute, and offloading introduces latency and synchronization cost. These strategies remain effective only while the system can tolerate those tradeoffs. In demanding or real-time workloads, that tolerance is often narrow. Interactive systems, games, rendering pipelines, and latency-sensitive applications cannot absorb unlimited overhead in exchange for lower local resource usage.

This is the real bottleneck, once a workload consistently hits limits in memory capacity, bandwidth, compute throughput, or latency tolerance, the issue is no longer inefficiency alone. It is a resource ceiling. At that point, the question is no longer how to force more efficiency out of the same hardware, but whether the workload now requires a different machine, a different execution environment, or a different system architecture.

The Last Option: Hardware

Hardware should be the last option, not the first reaction. Once software inefficiencies have been removed, architecture has been improved, and workload placement has been made efficient, the remaining limit is no longer design waste but physical constraint. That is the point where more RAM, more bandwidth, more compute, or a different class of machine becomes necessary.

Upgrading hardware at this stage is not an admission that software failed. It is an acknowledgment that software has already delivered its meaningful gains. The real mistake is upgrading before reaching this point. Hardware is not the enemy of efficiency; premature hardware dependency is.

Even this final option has constraints. Upgrading infrastructure or moving workloads into the cloud does not remove bottlenecks entirely; it often replaces local capacity limits with distributed systems limits such as latency, bandwidth, synchronization overhead, and data locality. The real problem is not whether to optimize software or upgrade hardware but knowing exactly where the bottleneck has moved next.