The Retry That Restored Access

Retries keep distributed systems moving through timeouts and temporary failures. In an access system, though, an old retry can be more dangerous than a failed request.

This is Part 2 of Security Infrastructure in Practice, a series about what happens when security design meets production systems.

Access had been removed from a deactivated account. The audit trail showed a successful removal. A few minutes later, the membership was back !!!

No administrator had restored it. An older provisioning job had timed out, waited, and replayed the grant after the deactivation.

The sequence looked like this:

1. A worker receives an event that adds a user to a group.
2. The destination accepts the request, but the response times out.
3. The user is deactivated in the authoritative identity system.
4. The original worker wakes up and retries the old group addition.

Every component did something understandable. The final state is still wrong.

Events Age, Even When Queues Do Not

A queue preserves work. It does not guarantee that the work remains valid.
If an event says “add this user to this group,” the worker needs to know whether a newer identity state has replaced that instruction. A timestamp helps, but clocks and delayed producers make ordering messy. I prefer a generation assigned by the authoritative identity record.

{
"subject_id": "user-1842",
"generation": 41,
"requested_change": "add_group",
"group_id": "finance-readers"
}

When the user is later deactivated, the authoritative record moves to generation 42. Before applying generation 41, the worker reloads the current state.

async def process(job):
desired = await identity_store.get(job.subject_id)

if job.generation < desired.generation:
    return "superseded"

return await reconcile(desired)

The stale job becomes harmless. It may still be acknowledged and recorded, but it cannot overwrite the newer decision.

Retry the Goal, Not the Verb

“Try the POST again” is procedural recovery. It assumes the original operation is still the right one.
State-based recovery asks a different question: what should this identity look like now?

@dataclass(frozen=True)
class DesiredIdentity:
subject_id: str
generation: int
active: bool
groups: frozenset[str]

The worker reads the destination, compares it with desired state, and calculates the smallest safe correction. If the destination already applied the timed-out request, no second create is needed. If the identity is now inactive, the plan removes access instead.

Timeout Does Not Mean Failure

This is the most dangerous retry assumption.
A client can time out after the server commits a write. The client knows that it did not receive a response. It does not know that the operation failed.
For account creation, search using an immutable external identifier before retrying:

async def recover_create(subject_id, desired):
existing = await destination.find_by_external_id(subject_id)

if existing:
    return await update_to_desired_state(existing, desired)

return await destination.create(
    external_id=subject_id,
    profile=desired.profile
)

Email is not a reliable external identifier. It can change and may be reassigned. Use a stable subject identifier from the system that owns identity.

Keep Deactivation Ahead of the Queue

Provisioning systems often prioritize creation because delayed onboarding is visible. Delayed removal is quieter.
Quiet does not mean safe.
Give deactivation work reserved capacity. Preserve an inactive record or tombstone long enough to reject old events. If you delete the subject immediately, a delayed create can look like a brand-new identity.

def priority(desired, observed):
if observed.active and not desired.active:
return 0 # highest priority
if observed.missing and desired.active:
return 10
return 20

Recovery Needs a Stop Condition

Some failures will not improve with time. The destination may reject a schema value. Two identity sources may disagree about ownership. A direct administrator change may be intentional.
After a bounded number of attempts, stop. Quarantine the item with the desired state and the latest observation. Give the responsible team a concrete conflict to resolve.
An infinite retry loop is not resilience. It is a way to hide a decision the system cannot make.

Measure Convergence

API success rate is a weak health metric for identity provisioning. A connector can report successful calls while the wrong accounts remain active.
Track how long destinations take to match desired state. Separate deactivation drift from ordinary profile drift. Look at the age of the oldest unresolved high-risk identity rather than relying on an average.

A retry system is safe when repeated work moves the destination toward the latest desired state. If replaying an old message can move it backward, the retry mechanism is part of the access-control problem.

How does your provisioning system prevent delayed work from restoring old access?