The retry worked.
The database is consistent, the customer received one result, and the commercial operation was applied once. From the application's perspective, the recovery behaved exactly as intended.
But there is still a question the final state cannot answer:
How many times did the work actually happen?
Imagine an AI product that generates research reports.
A customer starts one report:
POST /reports
↓
execution_789
↓
model provider
The application sends the provider request. The provider receives it and may begin processing it.
Then something goes wrong on the response path.
CALLER PROVIDER
request ------------------------>
work begins
...
X <---------------- response
TIMEOUT
From the caller's perspective, the operation did not complete successfully. There is no usable response, so the application retries.
CALLER PROVIDER
retry -------------------------->
work executes
response <-----------------------
SUCCESS
The second attempt succeeds. The report is generated. The final business transition happens once.
Maybe our application also uses an idempotency key to make sure repeated processing doesn't create a second report, consume the same commercial allowance twice, or apply the same settlement twice.
That is exactly the kind of correctness we want.
But it doesn't tell us what happened during the first provider attempt.
The first request may never have reached the provider. It may have reached the provider without starting meaningful work. The provider may have partially processed it. Or the operation may have completed and produced a response that the caller never received.
The timeout alone cannot distinguish those cases.
We may have successfully protected the business effect from duplication while still being uncertain about whether downstream work was duplicated.
That distinction matters in any distributed system. What makes it particularly interesting in AI infrastructure is that an execution attempt can involve variable, economically relevant resources: model inference, long context, search, external APIs, tools or additional agent steps.
So the question isn't whether retries are bad. Retries are one of the mechanisms that make distributed systems reliable.
The question I'm interested in is narrower:
Where does the correctness guarantee of an idempotent retry actually stop?
A Timeout Describes What the Caller Observed
It's easy to look at a timeout and mentally translate it into:
The operation failed.
At the application boundary, that description may be perfectly reasonable. We did not receive the response we expected, so the attempt cannot be treated as a successful completion.
But a timeout is not a complete description of what happened remotely.
Consider a slightly different view of the same interaction:
CALLER PROVIDER
request ------------------------>
receives request
performs work
produces response
X <---------------- response
network failure
TIMEOUT
The caller knows that it did not obtain a successful response within the expected conditions.
It does not necessarily know whether the remote operation never started, started and stopped somewhere in the middle, completed successfully, or completed successfully but lost its response on the way back.
This uncertainty is a familiar problem in distributed systems. AWS describes a similar case in its Builders' Library: after a network timeout, the caller may not know whether a remote resource was created, which makes a blind retry potentially unsafe without an idempotent contract.
But retrying solves a different problem from discovering what happened before the retry.
Suppose the second attempt succeeds:
execution_789
│
├── attempt_01 → TIMEOUT
└── attempt_02 → SUCCESS
We now know that the logical execution eventually produced the result we wanted.
We still don't know, from those statuses alone, how much work occurred during attempt_01.
That gives us the first distinction worth preserving:
A failed observation at the caller is not necessarily evidence of zero work at the provider.
That doesn't mean the provider definitely performed billable work. It means the evidence we currently have does not justify either conclusion.
Sometimes the most accurate state is simply:
attempt_01
technical_status: TIMEOUT
downstream_work: UNKNOWN
That uncertainty becomes important once we start reasoning about what the retry actually protected.
Because an idempotency guarantee has a scope.
Idempotency Has a Scope
When we say that a system is idempotent, the statement can hide an important question:
Idempotent where?
Suppose our report API accepts an idempotency key:
POST /reports
Idempotency-Key: report_123
Our application associates that key with one logical execution:
report_123
↓
execution_789
If the client repeats the request, the application can recognize that it refers to work already initiated instead of blindly creating another business operation.
That can protect several things we care about. We might avoid creating two reports for the same request, prevent the same entitlement mutation from being applied twice, or ensure that commercial usage is settled once even if the request is delivered more than once.
This is the general reason idempotency keys are useful for retried mutating operations. For example, Stripe documents idempotency keys specifically so a client can repeat a request after a connection error without accidentally creating a second object or applying the same update twice.
Conceptually:
CLIENT
↓
API ← idempotency protected here
↓
WORKFLOW
↓
MODEL PROVIDER ← separate operation
↓
SETTLEMENT ← separately protected
The important part is that the guarantee at one boundary does not automatically become a guarantee at every boundary below it.
Protected here does not automatically mean deduplicated everywhere.
Imagine that execution_789 reaches the model provider for the first time:
execution_789
↓
provider_attempt_01
↓
request sent
↓
provider receives it
↓
?
↓
TIMEOUT
Our application still knows that this belongs to execution_789. The idempotency key can continue protecting the logical operation from being recreated at the API boundary.
But if we decide to retry the provider call, we have crossed into another boundary:
execution_789
│
├── provider_attempt_01 → TIMEOUT
└── provider_attempt_02 → SUCCESS
Whether provider_attempt_02 is deduplicated against the first attempt depends on the semantics available at that downstream operation.
If the downstream API supports a compatible idempotency mechanism and we use it correctly, repeated requests may be recognized as the same operation.
If it doesn't, the second request represents another attempt to perform the downstream operation.
And if the first attempt had already performed some or all of the work before we lost certainty, both attempts may have consumed resources even though our application eventually commits only one business result.
This is why I find “the system is idempotent” too broad to be useful on its own.
A more useful question is:
Which operation is idempotent, at which boundary, and which effect is being protected?
An idempotency key on our public API can protect our public API semantics. A deduplication mechanism around commercial settlement can protect settlement. A downstream provider may expose its own idempotency semantics.
Those guarantees can complement each other, but they are not interchangeable.
The architecture may therefore look less like one global idempotency property and more like several explicit boundaries:
CLIENT
│
▼
┌─────────────────────┐
│ API IDEMPOTENCY │
│ protects creation │
│ of logical work │
└─────────────────────┘
│
▼
WORKFLOW EXECUTION
│
▼
┌─────────────────────┐
│ PROVIDER OPERATION │
│ its own semantics │
└─────────────────────┘
│
▼
┌─────────────────────┐
│ SETTLEMENT │
│ separately guarded │
└─────────────────────┘
Once I started thinking about retries this way, another distinction became harder to ignore.
The identity of the logical work is not necessarily the identity of every attempt used to complete it.
One Logical Execution Can Have Multiple Attempts
From the application's perspective, execution_789 represents one piece of logical work:
execution_789
↓
generate report
The customer asked for one report. The application intends to produce one report. Eventually, one report is delivered.
But the path to that result may contain more than one attempt:
execution_789
│
├── attempt_01
│ ↓
│ TIMEOUT
│
└── attempt_02
↓
SUCCESS
↓
report_123
There is no contradiction here.
One logical execution and multiple execution attempts describe different things.
The logical execution represents the work the application is trying to accomplish. An attempt represents one try at performing some part of that work.
Keeping those concepts separate matters because collapsing them can make a successful recovery look simpler than it actually was.
If we preserve only:
execution_789
status: SUCCESS
that may be completely correct for answering:
Did the report eventually succeed?
It tells us much less about:
What happened while getting there?
Preserving the attempts gives us more structure:
execution_789
attempt_01
status: TIMEOUT
attempt_02
status: SUCCESS
outcome
report_123
Now we can see that the successful outcome involved more than one attempt.
But even this does not tell us what the first attempt consumed.
A timeout remains an observation about the attempt's visible result from our side. It is not a measurement of the work performed remotely.
So we should resist the shortcut:
attempts = 2
therefore:
work = 2 × normal execution
That conclusion does not follow.
The first attempt may have failed before meaningful downstream work began. It may have performed only part of the work. It may have completed almost everything. The downstream system may have deduplicated the second request. Or some intermediate result may have been reused.
Retry count is not economic evidence.
To understand the economic impact, we need evidence about what actually happened during those attempts.
For an AI workflow, that evidence might include measured model usage, tool invocations, search operations, external API activity or other resource consumption associated with the attempt.
Suppose this time we actually have enough evidence to establish:
execution_789
attempt_01
status: TIMEOUT
observed_resource_cost: $0.16
attempt_02
status: SUCCESS
observed_resource_cost: $0.18
outcome
report_123
Now we can make stronger statements.
The successful attempt has $0.18 of observed resource cost. Across the two associated attempts, we observed $0.34.
And $0.16 of that observed consumption occurred on an attempt that did not produce the successful response our application used.
But even here, the $0.16 is not automatically waste.
Maybe the first attempt was an unavoidable consequence of operating across an unreliable network. Maybe it performed useful intermediate work. Maybe the retry was necessary to preserve the reliability expected by the customer.
The evidence tells us that resources were consumed.
It does not yet tell us whether that consumption was avoidable, productive or economically justified.
Otherwise retry analysis can quietly turn into:
failed attempt
↓
wasted work
↓
optimization target
without enough evidence to support the conclusion.
Sometimes the problem comes before optimization.
It's simply knowing what happened.
Unknown Is Not Zero
Now remove the resource evidence from the first attempt:
execution_789
attempt_01
status: TIMEOUT
provider_usage: ?
attempt_02
status: SUCCESS
provider_usage: observed
What should we record for attempt_01?
One convenient answer would be:
provider_usage: 0
After all, the attempt failed and we have no usage measurement.
But zero is not the absence of information.
Zero is a claim.
It says that we know no relevant resource consumption occurred.
If the only thing we actually know is that the caller timed out, that claim may be stronger than our evidence allows.
The opposite shortcut is problematic too. We should not automatically assume that the first attempt consumed the same resources as a complete execution:
provider_usage: estimated full execution
A timeout does not establish that either.
So when execution evidence is incomplete, the more accurate representation may be:
attempt_01
status: TIMEOUT
provider_usage: UNKNOWN
Or, depending on what the system knows:
provider_usage: PENDING
or:
provider_usage: ESTIMATED
Those states mean different things.
UNKNOWN says we do not currently have enough evidence to establish the consumption.
PENDING could mean that stronger evidence may still arrive later.
ESTIMATED means we are intentionally using an approximation rather than an observed measurement.
The exact vocabulary is system-specific. I don't think every runtime needs these exact states.
The principle is more important:
Unknown is more accurate than zero when the evidence does not establish zero.
This may sound like a small modeling detail, but it changes what we can safely conclude later.
Imagine aggregating one thousand timed-out attempts.
If every unknown attempt was recorded as zero, the final total may look precise:
failed_attempt_cost = $0
But the precision is artificial.
The system has converted missing evidence into an economic conclusion.
Preserving uncertainty gives us a less convenient number, but a more defensible history:
observed_cost: $X
unknown_attempts: N
That doesn't solve the economic question.
It tells us where the evidence stops.
And once that distinction is visible, we can separate two questions that are easy to collapse into one.
Retry Correctness and Retry Economics Ask Different Questions
Suppose our workflow eventually reaches this state:
execution_789
attempt_01
status: TIMEOUT
observed_resource_cost: $0.16
attempt_02
status: SUCCESS
observed_resource_cost: $0.18
outcome
report_123
commercial_settlement
10 credits
From the application perspective, the retry may have behaved exactly as intended.
One report was delivered. The business state was committed once. The same commercial operation was not settled twice. Repeated processing did not create the duplicate side effects we were trying to prevent.
That is a correctness question:
Retry correctness asks whether repeated processing preserved the intended application semantics.
But the execution history gives us another question:
Retry economics asks what resource-consuming work actually occurred while reaching that state.
In this example, we have evidence of $0.34 across the associated attempts.
That does not imply that the customer should be charged twice.
It does not imply that the commercial system should consume 20 credits instead of 10.
And it does not imply that the first $0.16 was unnecessary.
Those are separate decisions.
What the evidence tells us is narrower:
BUSINESS RESULT
one report
COMMERCIAL SETTLEMENT
10 credits
OBSERVED RESOURCE CONSUMPTION
$0.34 across two attempts
All three can be true at the same time.
The commercial view asks what should happen to the customer's allowance or entitlement. The runtime evidence tells us what work we observed underneath.
Keeping those views separate matters because the existence of resource consumption does not determine the charging policy.
This is the part I find especially interesting about retry correctness in variable-cost AI systems.
We can successfully protect the state we expose to the customer while the execution layer underneath follows a more complicated path.
Idempotency can keep the application and commercial effects consistent.
It does not make the execution history disappear.
I explored the broader economic side of this distinction in the Licenzy Guide Your AI Workflow Failed. The Cost Didn't Disappear..
The Guide asks what failed work means economically once resources may already have been consumed.
The engineering question here comes one step earlier:
Did our retry and idempotency model preserve enough evidence to know what work actually happened across the attempts?
Reliability Has an Economic Footprint
Now scale the same idea beyond one execution.
Imagine two AI workflows over the same period:
WORKFLOW A
successful outcomes: 100
execution attempts: 105
WORKFLOW B
successful outcomes: 100
execution attempts: 165
At the surface, both produced the same number of successful outcomes.
Both might also be functionally correct. Their retries may have preserved application state correctly, their commercial settlement may be consistent, and every customer may have received exactly the result the product promised.
But the execution histories underneath those outcomes are different.
That difference does not tell us that Workflow B is less profitable.
It does not tell us that its retry policy is badly designed.
And it certainly does not tell us that 60 attempts were wasted.
We still don't know enough.
Maybe Workflow B depends on an external system with a higher transient failure rate. Maybe those retries are what allow it to maintain the reliability customers expect. Maybe many of the additional attempts terminate before expensive work begins. Or perhaps some of them repeat substantial model, search or tool execution.
The attempt count exposes a difference in execution behavior.
Understanding its economic significance still requires evidence about what those attempts actually did.
This is why I don't think the useful goal is simply to minimize retries.
Reliability has value.
A retry that consumes additional resources can still be the right engineering decision if it allows the system to recover from transient failures and deliver the intended result.
But in variable-cost AI workflows, reliability can also have an economic footprint.
A retry policy such as:
maxAttempts = 5
is obviously a reliability decision.
But if an attempt can trigger model inference, retrieval, search, tools or external APIs, that configuration can also influence the amount of work required to deliver a successful outcome.
That doesn't make the policy wrong.
It means that understanding the execution requires more than the final success count.
Instead of asking only:
How many retries did we have?
I think the more useful questions become:
What work actually occurred across those attempts?
Which resource consumption can we observe?
Where is the evidence incomplete?
How much execution work was required to produce the successful outcomes we eventually delivered?
Those questions come before deciding whether the retries were good, bad, necessary or avoidable.
They help establish what happened.
What Should Survive a Retry?
I don't think the answer is to persist an enormous retry schema for every operation.
The useful information depends on the questions the system needs to answer.
But if retries can matter economically, preserving only:
retry_count = 2
doesn't tell us very much.
A minimal conceptual history might instead preserve something like:
execution_id: exec_789
attempt_01
status: TIMEOUT
resource_evidence: UNKNOWN
attempt_02
status: SUCCESS
resource_evidence: OBSERVED
outcome_id: report_123
commercial_settlement:
one operation
The exact fields are not the point.
The important distinction is between the logical execution, the attempts used to perform it, the evidence available for those attempts, the outcome eventually produced and the commercial treatment applied afterward.
That gives us a history we can investigate without pretending to know more than the evidence supports.
If stronger evidence arrives later, the first attempt does not need to remain economically invisible just because it originally timed out.
And if stronger evidence never arrives, UNKNOWN can remain unknown instead of quietly becoming zero.
I've been thinking about this while exploring runtime correctness in Licenzy.
Idempotency is an essential part of making repeated operations safe. But variable-cost AI execution has made me interested in a second property alongside that correctness: not only whether the system eventually reached the intended state, but whether we preserved enough evidence to understand the work required to get there.
Final Thoughts
The retry can work exactly as designed while the execution underneath still contains more than one attempt.
That doesn't mean every retry duplicated the cost of a full execution. It doesn't mean failed attempts were wasted. And it doesn't tell us what the customer should be charged.
It means those questions cannot be answered from retry correctness alone.
An idempotency guarantee has a scope.
Protecting one business effect does not automatically tell us what happened at every downstream boundary.
And when the evidence is incomplete:
Unknown is not zero.
Retries, timeouts and idempotency are not new problems created by AI. What makes their intersection with AI infrastructure interesting to me is that repeated execution can involve variable amounts of model inference, search, tools, external APIs and other resource-consuming work.
Reliability can have an economic footprint without becoming an economic mistake.
So I don't think the next question is how to eliminate retries.
It's this:
If retries are part of how we buy reliability, do we have enough evidence to understand what that reliability actually required?
Top comments (0)