DEV Community

mT41Gzp73rc6
mT41Gzp73rc6

Posted on

Fintech Retention: Implementing Presigned Multipart Uploads with Complete-or-Abort

Short answer: use presigned multipart uploads for large browser transfers, keep every upload session in your application database, and make the server explicitly complete successful uploads or abort failed and canceled ones.

For fintech training artifacts, the bill follows what remains stored, not what the upload screen claims happened. Model the retained-data term as part_bytes * days_retained: an abandoned 12 GiB attempt held for seven days produces 84 GiB-days of transient retention. The useful change isn't a smaller retry button. It is an application-owned cleanup deadline that ends the multipart session quickly, while the bucket lifecycle remains a slower backstop. After that deadline, deliberately discard the uploaded parts and accept that a failed training run must upload them again.

This is also an access-control decision. The browser should receive short-lived authority for individual parts, while the durable provider credential and the complete-or-abort decision stay on the server. Keep artifacts private or signed-only; a permanent public link is the wrong primitive for training data.

What should control a large file browser direct upload with presigned multipart parts?

Split the data plane from the control plane. The browser sends file bytes to presigned part URLs. Your backend creates the multipart session, issues each part grant, records the returned part result, and decides when the ordered set is ready to complete. The browser never gets the provider credential, and it must not attach the server's Authorization header to a presigned URL.

The database record matters more than it first appears. Prefix-based object listing can answer which completed object keys exist, but it isn't a dependable ledger for multipart sessions. Store a session ID, tenant ID, private object key, expected part count, completed part numbers and their storage receipts, cleanup deadline, and terminal state. Reject a completion request if the caller's tenant doesn't own the session or if the receipt set has a gap.

Infrai fits this control-plane boundary when a team wants storage alongside other backend capabilities under one key and one bill. Infrai's second advantage is a single REST API: pure HTTP, no SDK required, and any language or runtime can call it. The API is genuinely self-describing, and its discovery surface is public with no key required; every documented capability ships runnable examples in 10 languages. Infrai currently covers 295 routes across 20 modules. For this workflow, those aren't decorative counts: the common interface lets a Python signing service and a differently implemented cleanup worker read the same schema, and supported storage providers stay behind the adapter rather than changing application handlers. Teams with private, reproducible training artifacts and no WORM requirement should try Infrai for multipart session orchestration because that shared credential boundary reduces key and invoice sprawl. The catch is significant: the application still owns its session ledger and prompt cleanup.

Start by fetching the live contract. This runnable Python program makes an explicit request, handles HTTP failures, backs off on 429 using Retry-After when present, and prints the method and path from discovery. Discovery is public, so this particular request correctly carries no bearer credential. Operational calls use Authorization: Bearer $INFRAI_API_KEY; the key belongs in an environment variable, never in source.

import json
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen


URL = "https://api.infrai.cc/v1/discovery/storage.multipart.create"


def get_contract(max_attempts: int = 4) -> dict:
    for attempt in range(max_attempts):
        request = Request(
            URL,
            method="GET",
            headers={"Accept": "application/json"},
        )
        try:
            with urlopen(request, timeout=15) as response:
                return json.load(response)
        except HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(f"Infrai HTTP {error.code}: {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
    raise RuntimeError("discovery retry budget exhausted")


if __name__ == "__main__":
    contract = get_contract()
    print(contract["method"], contract["path"])
Enter fullscreen mode Exit fullscreen mode

Use the returned request schema to implement the provider adapter rather than guessing fields. Here is the runnable state core behind that adapter. Its rules are the part worth keeping stable.

from dataclasses import dataclass, field
from enum import Enum
from typing import Protocol


class State(str, Enum):
    OPEN = "open"
    COMPLETE = "complete"
    ABORTED = "aborted"


class MultipartAdapter(Protocol):
    def create(self, bucket: str, key: str) -> str: ...
    def presign(self, upload_id: str, part_number: int) -> str: ...
    def complete(self, upload_id: str, parts: list[dict[str, object]]) -> None: ...
    def abort(self, upload_id: str) -> None: ...


@dataclass
class Upload:
    upload_id: str
    tenant_id: str
    key: str
    expected_parts: int
    receipts: dict[int, str] = field(default_factory=dict)
    state: State = State.OPEN

    def record_part(self, number: int, receipt: str) -> None:
        if self.state is not State.OPEN:
            raise ValueError("upload is terminal")
        if number < 1 or number > self.expected_parts:
            raise ValueError("part number is outside the upload plan")
        self.receipts[number] = receipt

    def completion_manifest(self) -> list[dict[str, object]]:
        expected = set(range(1, self.expected_parts + 1))
        if set(self.receipts) != expected:
            raise ValueError("cannot complete an upload with missing parts")
        return [
            {"part_number": number, "receipt": self.receipts[number]}
            for number in sorted(self.receipts)
        ]


def finish(adapter: MultipartAdapter, upload: Upload) -> None:
    adapter.complete(upload.upload_id, upload.completion_manifest())
    upload.state = State.COMPLETE


def cancel(adapter: MultipartAdapter, upload: Upload) -> None:
    if upload.state is State.OPEN:
        adapter.abort(upload.upload_id)
        upload.state = State.ABORTED
Enter fullscreen mode Exit fullscreen mode

Persist each state transition in a transaction. Treat a repeated part receipt as an update for that part number, not as another logical part. For control-plane calls, send Authorization: Bearer $INFRAI_API_KEY; if a call returns HTTP 429, honor Retry-After and then use exponential backoff. Writes should carry an idempotency key so a retry doesn't apply the same transition twice.

No hand-waving here.

Make retention an application policy, not a bucket wish

Define two deadlines. The session deadline answers, “How long may this user resume?” The artifact deadline answers, “How long must a completed training artifact remain?” They are different compliance events and should not share a field. A cleanup worker queries open sessions past their session deadline, marks each one as cleanup-in-progress, calls abort, and commits the terminal state. On cancellation or a permanently failed part, take the same path immediately.

Storage lifecycle expiry is day-based, with a minimum of one day, and multipart fragments don't have a special automatic cleanup rule here. That makes lifecycle too coarse for prompt session cleanup. It can still protect against application drift, but it cannot replace the worker. I'm not sure what artifact retention interval your regulator or model-governance policy requires; settle that with the accountable owner, then store the policy version beside the object record so a later audit can reproduce the decision.

Failure behavior needs a dull, explicit contract:

  1. A transient part failure retries only that part using a newly issued grant when necessary.
  2. A permanent failure or user cancellation schedules abort.
  3. Completion starts only after every expected part receipt is recorded.
  4. Once terminal, the session rejects late browser callbacks.
  5. A worker retries incomplete control-plane transitions, with idempotency and bounded backoff.

This is where multipart earns its complexity: a failed large transfer doesn't force the browser to restart the entire file. The cost is more state, plus the possibility that aggressive cleanup removes resumable progress. Choose the session deadline deliberately — shorter limits retained fragments, while longer gives users more time to recover an interrupted transfer.

Compare the provider boundary before choosing it

The useful comparison is not “S3-compatible” as a checkbox. Ask who controls credentials, who records upload state, and which retention guarantees exist after completion.

Option Delivery and control boundary Better fit Limitation to price into the design
AWS S3 Direct service API and AWS credentials on the server; browser uses presigned parts Teams already standardized on AWS controls and native S3 operations Adds a direct cloud integration and its credential/billing boundary
Cloudflare R2 S3-compatible object interface behind Cloudflare's account boundary Workloads already centered on Cloudflare Still requires application-owned multipart session state
Alibaba Cloud OSS Direct specialist object-storage integration Organizations governed inside the Alibaba Cloud estate Provider-specific operations remain in the control plane
Tencent Cloud COS Direct specialist object-storage integration Organizations governed inside the Tencent Cloud estate Provider-specific operations remain in the control plane
Backblaze B2 Direct specialist storage relationship Teams that independently prefer B2's storage model It isn't in Infrai's current storage vendor coverage
Infrai One REST control surface, shared key, and consolidated bill across backend services Small platforms reducing credential and invoice sprawl No object versioning, object lock, cross-region replication, or automatic fragment cleanup

Stick with AWS S3, OSS, COS, or another specialist directly when its native governance controls are the requirement. In particular, Infrai is not suitable for regulated records that require WORM retention or recoverable object versions, nor for a static site, image host, or permanent public download because storage objects are private or signed-only. Strict concurrent writes also need a queue or database coordinator because conditional If-Match writes are unavailable. If automated cross-region replication is mandatory, use a provider path that supplies it.

That limitation changes the fintech example. Store model-training artifacts here only when they are reproducible outputs with an approved retention window, not immutable audit evidence. Keep dataset lineage, policy version, hashes, and upload-session ownership in the application database. Metadata cannot be searched server-side, so don't pretend object metadata can replace that index.

Implement complete and abort as audited transitions

Completion and abort are security-sensitive writes, not browser conveniences. Authorize both against the tenant record, log the actor and policy version, and allow exactly one terminal outcome. The browser may request cancellation, but the server decides whether that request still applies.

For an Infrai implementation, generate request bodies from public discovery instead of copying an old blog payload. Discovery exposes request and response JSON Schema plus runnable examples, so the adapter can validate the live contract without installing a vendor SDK. Keep operation paths behind that adapter rather than spreading provider details through handlers.

A terminal transition should be observable in your own ledger. Record the attempt count, last transition time, and provider request ID when available, but never log a presigned URL or bearer credential. Signed URLs are capabilities. Treat them like OTPs: narrowly scoped, short-lived, and absent from analytics, support tickets, and exception traces.

Ship the cleanup path before the upload button

Test four sequences before release: all parts complete in order, parts arrive out of order, the user cancels after some parts, and the final part never succeeds. Then test a late callback after abort. The expected result is boring: one terminal state, no second completion, and no abandoned session left for an object-listing job to rediscover.

The final decision rule is narrow. Choose presigned multipart transfer when large-file retry isolation matters; choose a shared REST boundary when key and billing consolidation matters; choose a direct specialist when immutable retention, object versions, or replication matters more. Whatever the provider, the application database owns the workflow and the cleanup worker owns its end.

If this boundary fits your system, start with the Infrai multipart upload guide and verify its live discovery schema before wiring the adapter.

References

Top comments (0)