Expiry looks like housekeeping. A timestamp passes, a background worker finds the stale row, and the system moves it to a terminal state.
That model is safe only when your database is authoritative for the outcome. The moment another system can complete the work, a timeout becomes much weaker evidence. It tells you that your local clock ran out. It does not prove that nothing happened elsewhere.
A recent committed C# change made this distinction concrete. The implementation and names are private, but the lesson is broadly useful: before an expiry worker made an irreversible local transition, it first reconciled with the external authority.
The Failure Hidden Inside a Timer
Consider a generalized hosted operation:
- Your application creates an intent with immutable expected values.
- An external system accepts the operation and returns a reference.
- The user or caller leaves your process.
- A callback or browser return normally confirms the result.
- Your local intent eventually reaches its expiry time.
The dangerous assumption is step five: “No callback arrived, therefore the external operation did not complete.”
Callbacks are delivery mechanisms, not proof of non-completion. A browser can close. A webhook can be delayed. A network path can fail after the external commit but before your acknowledgement. The local row can remain stale while the external outcome is already final.
If a cleanup worker then marks that row expired, the data looks tidy while the business truth becomes harder to recover.
Authority Is an Architectural Relationship
The central design question is not “Which service runs the expiry job?” It is “Which system owns the fact we are about to assert?”
Your database may be authoritative for your workflow state. The external system may still be authoritative for whether its operation completed. Those are different facts.
That gives us a useful rule:
Before an irreversible transition, reconcile with the system that owns the outcome.
In practical terms, a worker can select expired candidates, find the external reference, locate a verifier for that kind of operation, and ask for the current outcome. A confirmed result should only be accepted when it matches the local intent’s frozen invariants, such as identity and expected value. Then the confirmation must be persisted before cleanup continues.
The expiry worker does not invent new truth. It asks the authority and applies an already-defined verification contract.
Preserve “Uncertain” as a Real State
Distributed workflows rarely have only two honest answers. They usually have three:
- Confirmed and matching: advance the workflow and persist the result.
- Clearly not completed: follow the normal expiry path.
- Unavailable, conflicting, or ambiguous: preserve evidence and route to review.
Collapsing the third answer into “failed” is where reliability problems become data-integrity problems.
Suppose the authority times out during reconciliation. Failing the entire sweep may create a retry storm and block unrelated candidates. Expiring the operation anyway is worse: absence of an answer has been treated as a negative answer.
A safer compromise is failure isolation. Log the verification problem, retain the external reference and relevant audit evidence, move the candidate to a reviewable state, and let the sweep continue. Uncertainty remains visible and recoverable.
This pattern also makes operational ownership clearer. The manual-review queue is not an embarrassment. It is the explicit cost of refusing to manufacture certainty.
The Trade-Off Is Real
Reconciliation adds work:
- another network dependency in a scheduled process;
- extra latency and provider load;
- rate-limit and back-off concerns;
- longer retention for ambiguous records;
- an operational queue that someone must own;
- more state-transition and concurrency tests.
Those costs should be designed, not ignored. Batch candidates. Bound concurrency. Use cancellation and timeouts. Record the last reconciliation attempt. Back off repeated uncertainty. Make confirmation idempotent. Prevent a concurrent callback and the sweep from applying contradictory transitions.
The return is not merely “fewer bugs.” It is a stronger integrity boundary. An automated cleanup task can no longer silently overwrite a result owned elsewhere.
Test the Missed-Acknowledgement Paths
A happy-path expiry test proves very little. The valuable regression tests exercise the boundary:
Completed, but the acknowledgement was missed
The external verifier reports a matching completion. The worker persists it and does not expire the operation.
The authority is unavailable
Verification throws or times out. The sweep continues, but the operation retains its evidence and moves to review rather than a clean terminal failure.
The authority does not confirm completion
The worker follows the documented evidence-preserving policy. Depending on the remaining evidence, that may mean normal expiry or review. The important part is that a single inconclusive query does not erase stronger durable evidence.
I would add two more tests where risk justifies them: a callback racing the sweep, and a repeated reconciliation proving idempotency.
Where Else This Applies
The same authority check appears far beyond one kind of integration:
- a deployment controller deleting resources after a local timeout;
- a message dispatcher retrying when the broker may already have accepted the message;
- a provisioning job rolling back while the cloud control plane is still converging;
- a refund workflow closing locally before the financial system settles;
- a reservation expiring while the downstream allocation already exists.
Whenever another system can cross the irreversible boundary, your cleanup job needs more than a clock.
A Practical Review Checklist
Before shipping an expiry or cleanup worker, ask:
- Which system is authoritative for the outcome?
- Can completion occur without our acknowledgement arriving?
- What durable reference lets us reconcile later?
- Which invariants must match before we accept confirmation?
- What state preserves uncertainty without blocking the whole batch?
- Are confirmation, retry, and concurrent callbacks idempotent?
Timeouts are useful scheduling signals. They are not universal evidence of failure. At an irreversible boundary, ask the authority, match the facts, and preserve ambiguity.
Where does one of your cleanup jobs currently make a decision that belongs to another system?
Top comments (0)