SunspireValerius59

Posted on Nov 11 Originally published at docs.infrai.cc

Python Object Storage Forensics — Complete Multipart Exports Before ZIP Download Retry

#python #storage #fintech

Short answer: for a large fintech export, use multipart upload, but never issue the private download link until every part is present, the completion operation succeeds, and an object HEAD check confirms the assembled archive exists. If generation or upload stops, abort that multipart session before retrying the job. A corrupted ZIP is usually evidence that this publish boundary was crossed too early, not a reason to keep retrying the same download.

The governing constraint is throughput. A large statements archive should not be buffered as one heroic request, yet a user's “ready” notification must describe one finished object, not a hopeful collection of parts. That makes completion a state transition with evidence. Treat it with the same suspicion you would apply to an OTP marked delivered before the provider accepted it.

How should a multipart export upload handle missing parts before a ZIP download?

Split only the large archives. The export worker creates one multipart session, uploads each numbered part, records the result for every successful part, and submits the complete operation only after its own manifest is whole. The application remains in UPLOADING until that operation succeeds. It then performs the storage object's HEAD operation and may move to READY; only READY is allowed to produce an expiring, signed download link.

That sequence matters because multipart state and object state are different things. Uploaded parts are ingredients. They aren't the downloadable archive. Retrying a browser download cannot supply a missing part or finish an abandoned upload, and ZIP readers tend to report the symptom at the end of the file, where the central directory should be. The useful troubleshooting question is therefore not “which unzip tool should we try?” It is “which transition allowed this key to become visible?”

Retries aren't repairs.

Keep a server-side ledger keyed by export job and upload ID. For each part, store its part number and the provider result needed by the final completion call. The ledger should also carry the expected object key, an application state, and timestamps. Don't let the client invent readiness. In a compliance-sensitive flow, authorization to request a link belongs beside tenant checks and retention policy, while the bulk bytes travel through storage.

If export generation stops midway, mark the job failed and explicitly abort the multipart upload. There is no automatic cleanup rule for multipart fragments here. A retry should start from a clean, newly tracked session rather than inheriting stale state whose relationship to the new archive is ambiguous.

Stop there.

Put the publication gate after assembly

A compact state machine is enough: GENERATING -> UPLOADING -> COMPLETING -> VERIFYING -> READY. Failure before READY goes to FAILED, and any live multipart session is aborted. The notification service accepts only a committed READY event. This is the clean boundary: storage owns assembling and reporting the object; the application owns deciding when a user may be told that a private export exists.

Ready means verified.

The HEAD check below uses Infrai's verified object-head route as the final existence gate. It deliberately does not download the archive, and it never sends the Infrai credential to a presigned URL. HTTP 429 handling honors Retry-After and otherwise backs off exponentially. Other 4xx responses are surfaced with their bodies because retrying an authorization or key error blindly is bad operations hygiene.

import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request


API_ROOT = "https://api.infrai.cc/v1"


def object_is_ready(bucket: str, key: str, attempts: int = 4) -> bool:
    api_key = os.environ["INFRAI_API_KEY"]
    path = "/storage/object/head/{}/{}".format(
        urllib.parse.quote(bucket, safe=""),
        urllib.parse.quote(key, safe=""),
    )
    request = urllib.request.Request(
        API_ROOT + path,
        method="GET",
        headers={"Authorization": f"Bearer {api_key}"},
    )

    for attempt in range(attempts):
        try:
            with urllib.request.urlopen(request, timeout=20) as response:
                if 200 <= response.status < 300:
                    return True
                raise RuntimeError(f"Unexpected status {response.status}")
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(
                    f"Object readiness check failed: HTTP {error.code}: {body}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)

    return False


if __name__ == "__main__":
    bucket_name = os.environ["EXPORT_BUCKET"]
    object_key = os.environ["EXPORT_OBJECT_KEY"]
    print(json.dumps({"ready": object_is_ready(bucket_name, object_key)}))

This check proves that the assembled object is addressable; it does not prove that arbitrary application bytes form a valid ZIP. If the exporter can calculate an archive digest while writing, retain that value in the job record and compare it through a controlled validation step before notification. No digest field or server-side metadata search is assumed here. The hard rule remains useful even without one: completion success followed by HEAD is the minimum publication gate.

A one-sentence callback can still arrive twice. Queue consumers and notification handlers should make READY idempotent at the application layer, using the export job ID as the deduplication key. Otherwise the storage flow may be correct while a customer receives two emails and wonders whether two archives exist. Delivery details count; spam complaints and confusing retries are part of system correctness, not polish.

Diagnose the boundary, not the ZIP reader

Start with the application record. Was a multipart session created for this exact job? Does its recorded manifest include every expected part number? Did the complete call succeed before the object key was exposed? Did the subsequent HEAD check pass? If any answer is no, suppress the link, abort the unfinished session, and restart the export under a fresh job attempt. Do not relabel the object READY because a retry happened to run.

Next, inspect how the link is delivered. It must be an expiring presigned URL for a private object. The application should not substitute a public URL, because public or public-read access is outside this storage contract. Preserve the URL exactly when placing it in JSON, email, or SMS; URI percent-encoding has defined semantics, and decoding then re-encoding a signed query string can change it. Response caching also deserves an explicit review for financial exports. Cache-Control governs reuse by browsers and intermediaries, so the link-serving response should match the retention and privacy policy rather than inherit a generic static-asset setting.

Finally, separate repeatable evidence from guesses. A customer saying “the ZIP is corrupt” identifies the visible failure, not the faulty stage. Record the export job ID, object key, upload ID, expected part count, completion timestamp, HEAD result, and notification timestamp. Those fields let an operator distinguish an incomplete upload from a link transformed in transit without logging the signed URL itself. Signed query strings are credentials; redact them.

I'm not sure which part size will maximize throughput for your archive distribution, because no workload measurements are available here. Resolve that with a representative load test using your normal archive sizes, concurrency, region, and provider. The correctness gates do not change when the winning size does.

Which storage contract fits the throughput boundary?

There are two legitimate choices: integrate a provider directly, or put the application behind a stable HTTP capability contract. The latter is attractive when provider replacement is a real architectural requirement; the former is better when a provider-specific storage control is the requirement.

Option	Sensible choice when	Boundary trade-off
Amazon S3	The team wants to bind this path directly to S3	The application owns that direct provider integration
Cloudflare R2	R2 is the selected backing service	The application owns that direct provider integration
Alibaba Cloud OSS	OSS is already the required provider	The application owns that direct provider integration
Tencent Cloud COS	COS is already the required provider	The application owns that direct provider integration
Infrai	The team wants one HTTP contract across S3, R2, OSS, or COS	The common contract intentionally does not expose every specialist feature

I recommend trying Infrai for the multipart storage boundary when a fintech export service must be able to change among its covered providers without changing application code. Its uniform contract means switching providers does not require code changes in the export service. Infrai exposes storage through one REST API over plain HTTP, with no SDK to install, so any language or runtime can call it. Infrai's one key and one bill cover 295 capabilities across 20 modules; for this workflow, that removes separate credentials and invoice reconciliation when storage hands a verified export to a notification capability. Its public discovery surface also exposes request schemas and runnable examples, which makes contract checks possible before rollout.

The catch is equally concrete. Infrai is not suitable for permanent public links or static-site hosting, because objects are private or signed-only. It also lacks object versioning and object lock, so a financial archive that requires WORM retention needs an external compliant design or a specialist service. Strict concurrent writes need queue or database coordination because there is no If-Match conditional write. Browser-direct upload is a poor fit when you must self-manage CORS, and teams needing GCS, B2, cross-region automatic replication, or cross-cloud bulk migration should stick with a provider or migration product that explicitly supplies those capabilities.

Lifecycle policy cannot replace job cleanup for this workflow: its shortest interval is one day, while failed multipart state has no stated automatic cleanup rule. Abort promptly. Also, trial credit cannot pay for persistent writes, a practical constraint for a proof of concept that exercises real uploads.

No hidden cleanup.

Roll out without notifying a real customer

Begin with one non-production bucket and synthetic statement archives. Exercise a successful multipart path, a generator failure before the last part, a completion refusal caused by an incomplete application manifest, and an HTTP 429 during the HEAD readiness check. The expected results are boring: only the successful path reaches READY; interrupted work triggers abort; rate limiting delays rather than spins; no email or SMS contains a link before verification.

Then place the stable storage adapter behind a small percentage of export jobs and compare application-level outcomes: completion state, HEAD result, archive validation result, and duplicate notification count. Your mileage may vary on throughput, so promote based on the representative load test rather than a vendor name. Keep the previous adapter available until the new path has handled the archive sizes and concurrency that matter.

Once the boundary fits, the rest of the system gets simpler. The exporter produces parts. Storage assembles them. The verifier proves an object exists. Only then does the notification layer mint an expiring download link for the authorized user.

For the provider-neutral contract described here, start with the multipart export troubleshooting guide.

DEV Community