Migrating an Async Job Polling Loop Between Provider APIs

#ai #machinelearning #tutorial #llm

A polling loop looks like four lines and encodes at least six assumptions: which field holds the status, which values mean stop, where the outcome lives, how often to ask, when to give up, and what a failed poll means. Changing provider invalidates most of them independently, which is why the search-and-replace version of this migration produces a loop that spins forever on a job that finished.

What a polling loop actually asserts

Write the assumptions down before touching the code, because each one is a separate question with a separate answer in the new API:

The status lives at a known path in the response object. The field name differs — one published batch API calls it status, another calls it processing_status — and so does its position if one wraps the object in a data envelope.
Some subset of its values is terminal. Everything else means keep asking.
When it is terminal, the outcome is readable from the same object. This one is frequently false, and it is the assumption that breaks most expensively.
The polling interval is acceptable to the provider. Poll rate counts against a rate limit, and a tight loop over a long job can consume more request quota than the work itself.
There is a bound on total time. Without one the loop is an unattended process that can outlive the job, the deployment and the useful lifetime of the answer.
A poll that errors is different from a poll that says “still running”. Conflating them turns a transient 503 into a declared job failure.

The terminal-state set is provider-specific

The subtle failure is not a renamed field, which throws immediately and is fixed in a minute. It is a terminal-state set that is a different size.

OpenAI’s Batch API moves a job through validating, in_progress and finalizing before reaching completed, and can also end at failed, expired or cancelled; the per-request outcomes are counted in request_counts and the bodies live in the file named by output_file_id, with failures in error_file_id (OpenAI batch API reference). Anthropic’s Message Batches use processing_status with in_progress, canceling and ended, where ended is the single terminal value and the outcome of each request is read from the results stream at results_url (Anthropic message batches reference).

A loop ported from the first to the second by renaming the field will terminate correctly and then report success for a batch in which every request errored, because on that shape the job status says only that processing stopped. A loop ported the other way will treat finalizing as an unknown state. Neither is caught by a test that only exercises the happy path.

Status values and field names are the fastest-moving part of any API. Read the current reference for both sides on the day you write the loop; the shapes above are what those two references describe at the time of writing and are named here to show the structural difference, not as values to copy.

Interval, jitter and the deadline

Providers publish a recommended poll interval and it varies with the expected job duration: seconds for a job that finishes in under a minute, tens of seconds or minutes for one with a multi-hour window. Porting the old interval to a new API with a different job profile is how you either burn rate limit or add ten minutes of latency to a job that finished in thirty seconds.

The shape that works across both is capped exponential backoff with full jitter and a separate wall-clock deadline:

interval(n) = random_between(base, min(cap, base * 2**n))

base     start at the provider's documented minimum, or 1-2s
cap      the longest gap you can tolerate before noticing completion
jitter   randomised across the whole range, so N workers started by
         one batch submission do not stay in lockstep
deadline wall-clock, independent of attempt count

The deadline being wall-clock rather than a maximum attempt count is the part most often got wrong. With exponential growth, “stop after 20 attempts” is a different amount of time for every base and cap you pick, so a tuning change to the interval silently changes the timeout. Set the deadline from the provider’s documented job window, and set it as a duration.

Full jitter matters more here than in ordinary retry logic. Submitting a hundred jobs in one loop and polling each on a fixed interval produces a hundred synchronised requests every interval forever; randomising across the whole window spreads them. The same reasoning as backing off correctly on a 429, applied to the status endpoint rather than the inference one.

Rewriting the loop

Extract the provider-specific parts into a small interface before changing anything: submit, poll once, classify a status into pending / done / failed / cancelled, and fetch the outcome. The loop itself — timing, deadline, error handling — becomes provider-agnostic and stops being rewritten each time.
Write the classify function as an exhaustive match over the new provider’s documented values, with an explicit default arm that logs and treats the unknown value as pending. Never treat unknown as done.
Split terminal from successful. On a shape whose single terminal state carries no outcome, the classify function must return done-with-unknown-outcome and the next step must read the per-request results. Make that a distinct function so it cannot be skipped.
Set base, cap and deadline from the new API’s documented job window, not from the old numbers. Record in a comment where each came from, so the next person can tell a tuned value from an inherited one.
Handle poll errors separately from job failure. A 5xx, a timeout or a 429 on the status endpoint increments a consecutive-error count and retries; only a documented failure status fails the job. Give the error count its own limit so a permanently broken endpoint still terminates.
Persist the job record before the first poll and update it on every terminal transition, so a process restart resumes polling instead of losing the job. A loop that only exists in memory is a job that vanishes on deploy.

The failure modes to test for

Four tests, each cheap with a stubbed client, each covering a failure that only appears in production otherwise. First, an unknown status value: the loop must keep polling and log, not crash and not succeed. Second, a terminal status with failed sub-requests: the job must be reported as partially failed rather than succeeded. Third, three consecutive 5xx responses followed by a success: the loop must survive them, and the successful poll must not be counted against the error budget. Fourth, a job that never terminates: the deadline must fire and produce a distinguishable timeout outcome rather than a generic failure, because the operational response to it is different — you go and look at the provider’s status page.

If the new provider also offers webhooks, the right end state is usually both: webhooks as the fast path and polling as a low-frequency reconciliation sweep for jobs whose notification never arrived. That means the same terminal-state classification is used from two entry points, which is a further argument for it being one function rather than a branch inside the loop — see mapping the webhook payload fields for the notification half.