Why we stopped retrying on timeout and built reconciliation instead

Early in StareBrain’s action dispatch layer, we had a simple rule: if an action times out, retry it.

It felt safe. It felt standard. It was wrong.

The problem isn’t the retry. The problem is what a timeout actually tells you — which is almost nothing.

What a timeout means (and doesn’t mean)

When a dispatch call times out, you know one thing: you didn’t get a response in time. That’s it. You don’t know whether the action reached the provider. You don’t know whether it executed. You don’t know whether it’s sitting in a queue about to execute in the next 200ms.

Retrying in that state doesn’t solve the ambiguity. It doubles it. Now you have two dispatches in flight and still no confirmed outcome for either.

For a calendar booking, that means two bookings. For a payment, two charges. For a message send, two messages. The user asked for one thing.

So we stopped treating timeout as a signal to retry and started treating it as a signal to reconcile.

The state model underneath

We track every action on two independent axes:

`enum class DeliveryState { PENDING, CONFIRMED, FAILED }
enum class AcceptanceState { PENDING, ACCEPTED, REJECTED }

data class ActionRecord(
    val actionId: String,
    val dispatchId: String,
    val providerObjectId: String?,
    val delivery: DeliveryState,
    val acceptance: AcceptanceState,
    val dispatchedAt: Instant,
    val confirmedAt: Instant?,
    val eligibleForReconciliationAt: Instant
)`

Delivery tracks whether the action reached the provider and got a response. Acceptance tracks whether the provider processed it correctly.

These are orthogonal. A delivery can be confirmed (provider responded) while acceptance is still pending (provider is async and hasn’t evaluated it yet). Acceptance cannot move out of PENDING until delivery is CONFIRMED — that’s a structural invariant enforced in code:

`fun ActionRecord.tryAccept(verdict: AcceptanceState): ActionRecord {
    check(delivery == DeliveryState.CONFIRMED) {
        "Cannot evaluate acceptance before delivery is confirmed."
    }
    return copy(acceptance = verdict)
}`

A timeout leaves delivery in PENDING. That record is not a failure — it’s unresolved. We give it a permanent status to match:

`enum class ActionStatus {
    CONFIRMED,
    FAILED,
    DENIED_UNRESOLVED
}`

DENIED_UNRESOLVED means: the action was not confirmed, and we cannot establish what happened programmatically. It never decays into success on timeout. It stays unresolved until a human review or a successful reconciliation pass clears it.

The consistency window

One subtlety: “not found” after a timeout doesn’t mean “didn’t happen.” Some providers have replication lag. A record written during a slow dispatch might not be queryable for several seconds after the call returns.

We gate reconciliation behind a minimum age:

`fun ActionRecord.isEligibleForReconciliation(): Boolean {
    return Instant.now().isAfter(eligibleForReconciliationAt)
}`

eligibleForReconciliationAt is set at dispatch time: dispatchedAt + consistencyWindowDuration. A reconciliation job that runs before this gate doesn’t query the provider — it waits. “Not found inside the window” is not a failure signal.

Provider capability tiers

Not every provider supports the same query surface, which changes what reconciliation is possible:

`enum class ProviderCapabilityTier { FULL, RECONCILABLE, PARTIAL, BLIND }`

• FULL: provider returns an artifact ID, supports idempotency keys, and has a query endpoint. Full reconciliation possible.

• RECONCILABLE: provider returns an artifact ID and has a query endpoint, but no idempotency key support. Reconciliation possible, but a clean timeout without a returned artifact ID leaves no anchor to query against — the client reference must be transmitted in the dispatch request AND be queryable by the provider for this tier to work.

• PARTIAL: provider returns an artifact ID only. Can confirm delivery after the fact, but cannot query outcome.

• BLIND: none of the above. Cannot reconcile programmatically. Surfaces as a hard integration requirement, not something to paper over with retry logic.

A BLIND provider on timeout is a human review case. We surface this explicitly rather than letting it silently retry and double-execute.

What changed in practice

Switching from retry-on-timeout to reconciliation-on-timeout meant:

• Zero duplicate bookings from network blips
• A clear paper trail for every unresolved action
• Provider integrations graded by what’s actually possible, not what we wished was possible

The tradeoff: more infrastructure, more state to manage, more explicit handling of ambiguity. Worth it. The alternative — pretending timeout means failure and retrying — was a silent correctness bug waiting to surface at the worst moment.

If your agent dispatches actions with real-world side effects, timeout is not a retry signal. It’s a reconciliation signal. Build for it.