A Paused AI Workflow: Retry, Resume, or Keep Holding?

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

When an automated workflow pauses, choosing its next action requires more than deciding whether to try again. Missing evidence, expired authorization, a temporary service failure, and an unknown execution outcome require different recovery paths.

I built Paused Workflow Recovery Benchmark to measure how LLMs apply an explicit recovery policy to 30 synthetic scenarios.

There are six action labels, with five cases per label:

Action	Purpose under the supplied policy
RECONCILE	Establish an unknown execution outcome or recognize verified completion before another attempt.
BLOCK	Keep execution blocked when the governing current, verified risk decision is DENY.
REFRESH_EVIDENCE	Obtain adequate current, verified evidence for the exact action, network, and account.
REAUTHORIZE	Obtain valid authorization when risk evidence is adequate but authorization is missing, expired, revoked, or mismatched.
RETRY	Retry a temporary failure when risk and authorization are valid and evidence establishes that no execution was accepted or sent.
RESUME	Continue when risk and authorization are valid, no execution was sent, and the pause condition has been resolved.

The policy specifies precedence between these conditions. It also prevents informal operator messages and untrusted notes from overriding governing evidence.

Each case starts an independent conversation. The model receives the policy and scenario, but not the expected answer. Scoring trims surrounding whitespace, converts the response to uppercase, and compares the complete response with the expected label. There is no keyword extraction or LLM judge.

The task returns accuracy: correct answers divided by 30. The Kaggle benchmark uses Average of task scores; with one task, the overall score is that accuracy.

Models Tested

I started with Gemini 3.7 Flash, Claude Haiku 4.5, and Claude Sonnet 4.5, then added GPT-6 Astra, Claude Opus 4.8, and Gemini 3.1 Pro Preview. This gives a comparison across three providers and several model tiers.

These are the identifiers recorded in the runs:

google/gemini-3.7-flash
anthropic/claude-haiku-4-5@20251001
anthropic/claude-sonnet-4-5@20250929
openai/gpt-6-astra
anthropic/claude-opus-4-8@default
google/gemini-3.1-pro-preview

There was one completed run per model, evaluated on September 26, 2026.

I also attempted Claude Opus 5 and Grok 4.6. Those runs encountered errors and are excluded from the accuracy comparison. Grok returned a 404 model-not-found error; the exact final cause of the Opus 5 failure was not confirmed. An errored run is not a zero-accuracy result.

Findings

Model	Correct / total	Accuracy	Observed run cost
Claude Opus 4.8	30/30	100%	$0.1069
Gemini 3.1 Pro Preview	30/30	100%	$0.1138
GPT-6 Astra	30/30	100%	$0.1305
Gemini 3.7 Flash	30/30	100%	$0.0312
Claude Haiku 4.5	28/30	93.33%	$0.0136
Claude Sonnet 4.5	28/30	93.33%	$0.0408

Costs are the values displayed for these specific runs. They are not general pricing estimates or guarantees of future cost.

The two mismatches also exposed a benchmark ambiguity

Haiku and Sonnet both answered RESUME on T1 and T3, where the expected label was RETRY. All their other answers matched.

T1 describes a temporary connection failure before any request was sent. T3 describes a temporary preflight outage and a journal confirming zero requests sent. Both say risk and authorization remain valid. Neither explicitly states whether the temporary fault is still unresolved.

That omission matters to the distinction between RETRY and RESUME. The observed mismatches therefore reveal a limitation in the scenario wording as well as a difference in model outputs. Because the answers contain only labels, I cannot infer the models' reasoning.

I am keeping v1 unchanged. A clarified case set should be published and rerun as a separate version, so readers can distinguish the original evaluation from a revision made after inspecting results.

Perfect scores limit what this set can distinguish

Four models reached 30/30. On this sample, Gemini 3.7 Flash matched the more expensive successful runs at a lower observed cost. That is useful for this particular task, but the score ceiling prevents a finer comparison among those four models.

All six completed runs produced valid labels. None recommended RETRY or RESUME on the 20 cases whose targets required an execution hold: BLOCK, REFRESH_EVIDENCE, REAUTHORIZE, or RECONCILE.

That is a narrow, defined observation. The task does not execute actions, verify signatures, or establish end-to-end agent safety. It is a small synthetic test of applying a supplied policy, with one run per model.

What I would measure next

I would first clarify unresolved versus resolved temporary faults in a separately versioned set. I would then add paired scenarios that change one governing fact at a time, especially unknown execution outcomes combined with expired authorization. Repeated runs and independently reviewed labels would help separate wording sensitivity from consistent policy errors.

The practical lesson from this evaluation is to make recovery states and their precedence explicit—and to audit the benchmark's own wording before interpreting a score difference as a model weakness.

My Benchmark

• Public Kaggle benchmark and leaderboard
• Task v1, scoring code, and model outputs
• Kaggle Benchmarks SDK

AI assistance was used to help prepare the evaluation code and write-up. The reported results were checked against the recorded model outputs.