Building a Reliable Python Client for an Async API

Peng-AI-Coder — Mon, 26 Oct 2026 03:42:48 +0000

Most API integrations start with an assumption that holds until it doesn't: if the call came back fine, the data is fine.

For a synchronous endpoint that is usually true. For an asynchronous one it is false in a way that is hard to notice, because nothing fails loudly. A create call returns a job ID, the job reaches SUCCEEDED, and the record you get back says the work did not happen. No exception, no 5xx, no retry — just an empty field where you expected text.

The client I want to describe wraps an API that behaves exactly like that: it takes a public Instagram video URL, transcribes the audio somewhere else, and hands back records. Most of the interesting work was not calling the API. It was deciding what each of the four layers means, and keeping the client from collapsing them into one boolean.

What follows is the design, the code, and the parts that are still guesses.

A 200 means one thing only

There are four states between your request and your data, and only the first is about HTTP:

The request was accepted.
A remote job exists, and you know its ID.
That job reached a terminal status.
The record it produced contains what you asked for.

Each can succeed while the next fails. The request is accepted and the connection drops before you learn the job ID. The job ends SUCCEEDED and the dataset is empty. The dataset has an item and the item carries an error field instead of text. A client that returns True from step one has answered a question nobody asked.

What it took to keep those four states apart, in about 1,200 lines of Python, is the rest of this article.

The project these notes come from

apismith-labs/instagram-reels-transcript-api is an open-source repository of clients, runnable examples, and integration notes for a transcription service. It is not a hosted API. Collection and transcription run in an Apify Actor; the repository is the client side of that boundary, with implementations in Python, Node.js, Go, Java, Rust, and cURL.

The Python part is the case study here. examples/python/reliable_actor.py runs the three stages — create, poll, read — with separate retry rules for each, and examples/python/client.py wraps them in three functions: one URL, a list of URLs submitted in a single job, and one account's post metadata. Both files are standard library only, deliberately: an HTTP SDK would hide the per-stage control this article is about.

Two disclosures, because they change how you should read the rest. I maintain the repository and the Actor it calls, so this is a case study of my own code, not a neutral review. And the prose is largely machine-written: I set the direction and stand behind the technical claims, an AI assistant drafted the text, and I reviewed it against the code before publishing. Code examples were checked against the repository's offline test suite, not against live runs, and I flag where that limit bites.

Separate transport success from business success

The classifier is the smallest interesting part of the client. It takes the raw dataset items and returns one status:

def classify_items(items: Any) -> tuple[str, str]:
    if not isinstance(items, list): return "unknown", "Dataset response is not an item list"
    if not items: return "empty", "Dataset contains no items"
    if not all(isinstance(item, dict) for item in items): return "unknown", "Dataset contains a non-object item"
    errors = [str(item.get("errMsg")).strip() for item in items if item.get("errMsg") is not None and str(item.get("errMsg")).strip()]
    texts = [item.get("text") for item in items]
    if errors:
        return "actor_error", _redact_text("; ".join(errors))[:500]
    if any(isinstance(text, str) and text.strip() for text in texts): return "success", ""
    if any("text" in item and item.get("text") is None for item in items): return "no_transcript", "Transcript text is null"
    if any("text" not in item for item in items): return "no_transcript", "Transcript text is absent"
    if any(isinstance(text, str) and not text.strip() for text in texts): return "no_transcript", "Transcript text is empty"
    return "unknown", "Transcript text has an unsupported type"

Three decisions are packed into that ordering.

The error branch runs before the text branch. A record can carry a per-item error field and a populated text field at the same time. Check text first and that record becomes a success. Check the error first and an item that reports a failure is treated as one, even when it shipped something that looks usable.

Feed the client a synthetic item with both fields set and it returns both:

status    : actor_error
text      : 'Synthetic text that is present anyway'
succeeded : False

result.text is truthy. result.status is actor_error. Which makes this line wrong:

if result.text:          # true for an item that reported failure
    save(result.text)

and this one right — both are fragments, with save and alert standing in for your own functions:

if result.succeeded:     # status == "success"
    save(result.text)
else:
    alert(result.status, result.error)

Whether the upstream service ever emits both fields together is something I have not observed against a live job, and the repository's field reference lists it as unverified. The ordering is still the right call, because the error costs are asymmetric: a false success puts bad data into your pipeline, while a false failure puts a row in a queue you were going to look at anyway.

Absent, null, empty, and wrong type are not one thing. text: null, a missing text key, and "" are no_transcript; the value 0 is unknown. no_transcript on a music-only video is an outcome you can drop. unknown means the response shape changed and should go to a human.

A dataset with no items is a result, not an absence. The job ran and produced nothing, so it gets its own status instead of being folded into "success with zero results".

One invariant is worth copying: if the classifier says success but no non-empty string can be found, the client raises ProtocolError rather than returning a success with text=None. A client that contradicts itself should fail at the contradiction, not three layers deeper in your code.

Do not retry a POST you cannot confirm

Retrying a GET is a technicality. Retrying a POST that creates a job is a decision about duplicate work and money, and the two do not belong in the same retry helper.

The failure that forces the split is boring: the POST times out. Not rejected — timed out. Your process knows only that no response arrived, and the server may or may not have created the job. From your side the two cases are identical.

So the client classifies the failure before it decides anything. This is that branch, taken as-is from examples/python/reliable_actor.py; it sits inside execute_input, which supplies row_id, source_url, and a local save helper that appends one journal record:

        except ApiError as exc:
            # The Actor/platform's 429 creation semantics are unverified. Do not risk a second paid POST.
            if exc.status == 429:
                state = "creation_unknown"
                exc = ApiError("HTTP 429 on Run creation; acceptance semantics are unverified; POST was not retried", 429)
            else:
                state = "creation_unknown" if exc.ambiguous or exc.status is None else "transport_error"
            save("creation_unknown" if state == "creation_unknown" else "create_failed", state, exc)
            return {"rowId": row_id, "sourceUrl": source_url, "status": state, "error": safe_error(exc), "runId": None}

Simplified excerpt; see the repository for the complete implementation. The enclosing function — journal setup, budget checks, build verification — is omitted, and nothing in this branch creates a replacement job: runId is None because no ID arrived, and the 429 arm replaces the upstream message with one that records the POST was not retried.

A visible client error — 400, 401, 403 — means the request was rejected and no job exists. Retrying the identical request cannot help, so it is not retried.
Anything ambiguous — timeout, dropped connection, unparsable body, 5xx — means a job may exist. The state is creation_unknown, surfaced as its own exception type so a caller cannot catch it as a generic transport failure and loop.
A failure after creation — you already have the job ID — means the job exists. Retry the read, never the create.

The 429 case deserves a note, because the tempting reading is "rate limited, so it was rejected, so I can retry". That assumes the platform rejects before doing any work; nobody verified that for this API, so the client does not act on it. A 429 on create is recorded as unknown and not retried. Over-conservative? Possibly. The code says out loud that it lacks the evidence, which is the property I want in a client I have to trust at 3 a.m.

That leaves reconciliation as a human step. There is no idempotency key to send and no client-side auto-recovery; the journal keeps the input URL and a row ID but no job ID, because none arrived. Finding the job means matching run history by input and start time, then resuming it. Pretending otherwise in code is how you get two jobs and one invoice.

The client turns this into three exception types, and the split is the useful part:

Exception	What happened	What the caller should do
`ConfigurationError`	Nothing left the machine	Fix the input and call again
`CreationUnknownError`	A job may exist; you have no ID	Reconcile before any other call
`TranscriptTransportError`	A job exists; polling or reading failed	Resume that job

If those collapse into one Exception, every caller will handle them the same way, and the only handling that makes sense for all three is retry — which is wrong for two of them.

Make partial and unknown outcomes representable

An API that accepts N inputs and returns a list invites one assumption: that element i of the output belongs to input i. That is not a contract unless the API says so, and this one does not.

The status vocabulary has eight values, and unknown is the one people forget. On top of those, a bulk run gives each input its own row status; two of those exist only to describe a mapping problem — missing_result and duplicate_result. They exist so that "I don't know" survives contact with the caller.

Attribution starts by deciding what counts as the same video:

def _bulk_identity(url: str) -> tuple[str, str]:
    """Return conservative host-normalized URL key and exact path shortcode."""
    parsed = urlparse(url)
    return "https://instagram.com" + parsed.path, parsed.path.rstrip("/").rsplit("/", 1)[-1]

Only the host is normalised. The path has to match exactly, and the shortcode is a fallback that is used only when a record has no URL at all. If a record has both a URL and a shortcode and they disagree, the client records a conflict instead of picking a winner.

Each input then lands in exactly one state:

one matching record → classified like a single result;
no matching record, dataset complete → missing_result;
no matching record, dataset truncated → unknown, because absence proves nothing;
more than one matching record → duplicate_result;
a record matching no input → kept in extra_results rather than dropped.

Keeping the unmatched records matters. If the service starts returning canonical URLs that differ from the ones you submitted, every input becomes missing_result and every record lands in extra_results — together those two facts say exactly what changed. Drop the extras and you have a page of missing results and no explanation.

The truncation rule is worth its own sentence. The client reads one page with a limit; if exactly limit items come back, the read is treated as incomplete and the aggregate is unknown — even when every row it saw succeeded. A full page cannot be told apart from a cut-off one, so it is never reported as complete.

A failed job is a separate case again. When the job ends FAILED, TIMED-OUT, or ABORTED, there are no per-input results at all: every input becomes unknown carrying the job's error. Not missing_result — nothing was missing; there was nothing there to miss.

Validate at the boundary

_REEL_URL = re.compile(r"^https://(?:www\.)?instagram\.com/(?:reel|reels|p)/[A-Za-z0-9_-]+/?$")

def _validate_url(url: str) -> str:
    if not isinstance(url, str) or not url or url != url.strip() or not _REEL_URL.fullmatch(url):
        raise ConfigurationError("url must be a bare HTTPS Instagram Reel URL, not a Markdown link")
    parsed = urlparse(url)
    if parsed.scheme != "https" or parsed.hostname not in {"instagram.com", "www.instagram.com"} or parsed.query or parsed.fragment:
        raise ConfigurationError("url must not contain query parameters or a fragment")
    return url

The pattern is a module-level constant; it is shown next to the function here for readability.

Two gates, and the order is intentional. The regex is strict enough that the urlparse block is largely redundant — a query string cannot survive the $ — but the cheap gate produces the readable error and the structural gate keeps the guarantee if the pattern is ever loosened. Validation is also pure: no network, no token, no filesystem. That is what lets the same function run in a CLI check, in a loop over a CSV, and in the bulk validator that prefixes urls[3]:.

In practice the rule that bites is the query string, because share links carry tracking parameters. ?igsh= is the common one, appended by the Instagram app when you use Copy link; not every copied URL has one, and forwarded links can carry others. The client rejects all of them before a request leaves the process.

It rejects them with the wrong message, though: ?igsh= fails the regex first, so the error blames Markdown when the problem is a tracking parameter. Small thing, unless it costs you twenty minutes. Normalise at the edge instead, once, and hand the client something clean. One caveat before you copy anything: This helper is an additional example for this article, not part of the repository's current client API. It is a normaliser, not a validator, and the client does not clean URLs for you — whatever you pass in is what gets posted — so it rewrites the host and drops the query, fragment, and trailing-slash variance, but it accepts any path it is handed. Check the result against _validate_url if the input came from somewhere you do not control.

from urllib.parse import urlsplit, urlunsplit

def canonical(url: str) -> str:
    parts = urlsplit(url)
    host = parts.hostname or ""
    if host not in {"instagram.com", "www.instagram.com"}:
        raise ValueError(f"not an Instagram URL: {host!r}")
    path = parts.path.rstrip("/") + "/"
    return urlunsplit(("https", "instagram.com", path, "", ""))

https://www.instagram.com/reel/ABC/?igsh=xyz becomes https://instagram.com/reel/ABC/, which the client accepts. It drops the query, the fragment, the www, and the trailing-slash difference — and nothing else. The path stays, because the path is the identity used later for attribution; normalise it away and you have made every result unmatchable.

Test the failure paths without creating real jobs

If the interesting behaviour is the failure path, the test suite has to be able to make things fail. This one injects the transport: get_transcript(..., transport=...) accepts anything with a request(method, path, body, **kwargs) method, so tests hand it a fake that records calls and answers from a queue.

    def request(self, method: str, path: str, body=None, **kwargs):
        record = {"method": method.upper(), "path": path, "body": body, "query": kwargs.get("query")}
        self.requests.append(record)
        if not self.responses:
            raise AssertionError("No synthetic transport response queued")
        response = self.responses.pop(0)
        if isinstance(response, Exception):
            raise response
        return response

    @property
    def post_count(self):
        return sum(r["method"] == "POST" for r in self.requests)

Excerpt from tests/python/offline_support.py. Two properties make it useful. It fails loudly when the response queue runs dry, so a client that makes one call too many breaks the test instead of quietly succeeding. And it counts by method, so the assertion can be about what was sent, not only about what came back. That is the difference between a test that proves a value and a test that proves a behaviour:

    def test_unknown_creation_raises_distinct_no_blind_retry_exception(self):
        transport = FakeTransport([ApiError("synthetic timeout", ambiguous=True)])
        with self.assertRaises(api_client.CreationUnknownError):
            api_client.get_transcript(URL, token="synthetic-token", transport=transport)
        self.assertEqual(transport.post_count, 1)

The same shape covers "no request at all" — invalid URL, missing token, bad charge cap, unreadable journal — each asserting post_count == 0. That one is easy to get wrong: a client that validates after constructing its transport still passes a test that only checks the exception type.

The limits are worth stating plainly. Everything these tests prove about the network is the network as I mocked it. They cannot tell you that the real service returns an error field alongside text, that a dataset is visible the instant a job goes terminal, or that a 429 means what you hope. Those need controlled live runs, and the repository documents them as unverified rather than assuming them.

Within that limit: the three modules covering this client — test_python_client, test_reliable_actor, test_python_bulk — run with no token and no network. 58 tests, all passing, CPython 3.13.12.

What I would carry to another API

Five things, roughly in the order I would do them:

Model the layers separately. Request accepted, resource created, job finished, data usable. If one variable represents all four, three of them will be wrong somewhere in the code.
Keep uncertainty as a value. unknown and creation_unknown exist so "I don't know" is representable. A two-state model forces an answer you do not have, and the answer you will pick under pressure is the optimistic one.
Retry reads, not creates. GET is cheap and idempotent; a create POST is neither. Different policies, different code paths, no shared helper.
Never infer a one-to-one mapping. If the API does not promise that output i belongs to input i, match on identity and report unmatched records in both directions.
Test the failure paths first. The happy path you will exercise by hand anyway. The fake transport is cheap; a duplicate job is not.

None of this is novel. It is the standard checklist for asynchronous clients, and it is worth writing down because most clients skip it until an empty row shows up in a table nobody was watching.

Source

The code above comes from apismith-labs/instagram-reels-transcript-api — examples/python/client.py, examples/python/reliable_actor.py, and the tests under tests/python/. Both client files are standard library only, so the whole thing is readable in one sitting. The repository also keeps an explicit list of what it has and has not verified, which is where the caveats in this article come from.

DEV Community: Peng-AI-Coder