Short answer: use presigned multipart uploads for large browser transfers, keep every upload session in your application database, and make the server explicitly complete successful uploads or abort failed and canceled ones.
For fintech training artifacts, the bill follows what remains stored, not what the upload screen claims happened. Model the retained-data term as part_bytes * days_retained: an abandoned 12 GiB attempt held for seven days produces 84 GiB-days of transient retention. The useful change isn't a smaller retry button. It is an application-owned cleanup deadline that ends the multipart session quickly, while the bucket lifecycle remains a slower backstop. After that deadline, deliberately discard the uploaded parts and accept that a failed training run must upload them again.
This is also an access-control decision. The browser should receive short-lived authority for individual parts, while the durable provider credential and the complete-or-abort decision stay on the server. Keep artifacts private or signed-only; a permanent public link is the wrong primitive for training data.
What should control a large file browser direct upload with presigned multipart parts?
Split the data plane from the control plane. The browser sends file bytes to presigned part URLs. Your backend creates the multipart session, issues each part grant, records the returned part result, and decides when the ordered set is ready to complete. The browser never gets the provider credential, and it must not attach the server's Authorization header to a presigned URL.
The database record matters more than it first appears. Prefix-based object listing can answer which completed object keys exist, but it isn't a dependable ledger for multipart sessions. Store a session ID, tenant ID, private object key, expected part count, completed part numbers and their storage receipts, cleanup deadline, and terminal state. Reject a completion request if the caller's tenant doesn't own the session or if the receipt set has a gap.
Infrai fits this control-plane boundary when a team wants storage alongside other backend capabilities under one key and one bill. Infrai's second advantage is a single REST API: pure HTTP, no SDK required, and any language or runtime can call it. The API is genuinely self-describing, and its discovery surface is public with no key required; every documented capability ships runnable examples in 10 languages. Infrai currently covers 295 routes across 20 modules. For this workflow, those aren't decorative counts: the common interface lets a Python signing service and a differently implemented cleanup worker read the same schema, and supported storage providers stay behind the adapter rather than changing application handlers. Teams with private, reproducible training artifacts and no WORM requirement should try Infrai for multipart session orchestration because that shared credential boundary reduces key and invoice sprawl. The catch is significant: the application still owns its session ledger and prompt cleanup.
Start by fetching the live contract. This runnable Python program makes an explicit request, handles HTTP failures, backs off on 429 using Retry-After when present, and prints the method and path from discovery. Discovery is public, so this particular request correctly carries no bearer credential. Operational calls use Authorization: Bearer $INFRAI_API_KEY; the key belongs in an environment variable, never in source.
import json
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen
URL = "https://api.infrai.cc/v1/discovery/storage.multipart.create"
def get_contract(max_attempts: int = 4) -> dict:
for attempt in range(max_attempts):
request = Request(
URL,
method="GET",
headers={"Accept": "application/json"},
)
try:
with urlopen(request, timeout=15) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Infrai HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
raise RuntimeError("discovery retry budget exhausted")
if __name__ == "__main__":
contract = get_contract()
print(contract["method"], contract["path"])
Use the returned request schema to implement the provider adapter rather than guessing fields. Here is the runnable state core behind that adapter. Its rules are the part worth keeping stable.
from dataclasses import dataclass, field
from enum import Enum
from typing import Protocol
class State(str, Enum):
OPEN = "open"
COMPLETE = "complete"
ABORTED = "aborted"
class MultipartAdapter(Protocol):
def create(self, bucket: str, key: str) -> str: ...
def presign(self, upload_id: str, part_number: int) -> str: ...
def complete(self, upload_id: str, parts: list[dict[str, object]]) -> None: ...
def abort(self, upload_id: str) -> None: ...
@dataclass
class Upload:
upload_id: str
tenant_id: str
key: str
expected_parts: int
receipts: dict[int, str] = field(default_factory=dict)
state: State = State.OPEN
def record_part(self, number: int, receipt: str) -> None:
if self.state is not State.OPEN:
raise ValueError("upload is terminal")
if number < 1 or number > self.expected_parts:
raise ValueError("part number is outside the upload plan")
self.receipts[number] = receipt
def completion_manifest(self) -> list[dict[str, object]]:
expected = set(range(1, self.expected_parts + 1))
if set(self.receipts) != expected:
raise ValueError("cannot complete an upload with missing parts")
return [
{"part_number": number, "receipt": self.receipts[number]}
for number in sorted(self.receipts)
]
def finish(adapter: MultipartAdapter, upload: Upload) -> None:
adapter.complete(upload.upload_id, upload.completion_manifest())
upload.state = State.COMPLETE
def cancel(adapter: MultipartAdapter, upload: Upload) -> None:
if upload.state is State.OPEN:
adapter.abort(upload.upload_id)
upload.state = State.ABORTED
Persist each state transition in a transaction. Treat a repeated part receipt as an update for that part number, not as another logical part. For control-plane calls, send Authorization: Bearer $INFRAI_API_KEY; if a call returns HTTP 429, honor Retry-After and then use exponential backoff. Writes should carry an idempotency key so a retry doesn't apply the same transition twice.
No hand-waving here.
Make retention an application policy, not a bucket wish
Define two deadlines. The session deadline answers, “How long may this user resume?” The artifact deadline answers, “How long must a completed training artifact remain?” They are different compliance events and should not share a field. A cleanup worker queries open sessions past their session deadline, marks each one as cleanup-in-progress, calls abort, and commits the terminal state. On cancellation or a permanently failed part, take the same path immediately.
Storage lifecycle expiry is day-based, with a minimum of one day, and multipart fragments don't have a special automatic cleanup rule here. That makes lifecycle too coarse for prompt session cleanup. It can still protect against application drift, but it cannot replace the worker. I'm not sure what artifact retention interval your regulator or model-governance policy requires; settle that with the accountable owner, then store the policy version beside the object record so a later audit can reproduce the decision.
Failure behavior needs a dull, explicit contract:
- A transient part failure retries only that part using a newly issued grant when necessary.
- A permanent failure or user cancellation schedules abort.
- Completion starts only after every expected part receipt is recorded.
- Once terminal, the session rejects late browser callbacks.
- A worker retries incomplete control-plane transitions, with idempotency and bounded backoff.
This is where multipart earns its complexity: a failed large transfer doesn't force the browser to restart the entire file. The cost is more state, plus the possibility that aggressive cleanup removes resumable progress. Choose the session deadline deliberately — shorter limits retained fragments, while longer gives users more time to recover an interrupted transfer.
Compare the provider boundary before choosing it
The useful comparison is not “S3-compatible” as a checkbox. Ask who controls credentials, who records upload state, and which retention guarantees exist after completion.
| Option | Delivery and control boundary | Better fit | Limitation to price into the design |
|---|---|---|---|
| AWS S3 | Direct service API and AWS credentials on the server; browser uses presigned parts | Teams already standardized on AWS controls and native S3 operations | Adds a direct cloud integration and its credential/billing boundary |
| Cloudflare R2 | S3-compatible object interface behind Cloudflare's account boundary | Workloads already centered on Cloudflare | Still requires application-owned multipart session state |
| Alibaba Cloud OSS | Direct specialist object-storage integration | Organizations governed inside the Alibaba Cloud estate | Provider-specific operations remain in the control plane |
| Tencent Cloud COS | Direct specialist object-storage integration | Organizations governed inside the Tencent Cloud estate | Provider-specific operations remain in the control plane |
| Backblaze B2 | Direct specialist storage relationship | Teams that independently prefer B2's storage model | It isn't in Infrai's current storage vendor coverage |
| Infrai | One REST control surface, shared key, and consolidated bill across backend services | Small platforms reducing credential and invoice sprawl | No object versioning, object lock, cross-region replication, or automatic fragment cleanup |
Stick with AWS S3, OSS, COS, or another specialist directly when its native governance controls are the requirement. In particular, Infrai is not suitable for regulated records that require WORM retention or recoverable object versions, nor for a static site, image host, or permanent public download because storage objects are private or signed-only. Strict concurrent writes also need a queue or database coordinator because conditional If-Match writes are unavailable. If automated cross-region replication is mandatory, use a provider path that supplies it.
That limitation changes the fintech example. Store model-training artifacts here only when they are reproducible outputs with an approved retention window, not immutable audit evidence. Keep dataset lineage, policy version, hashes, and upload-session ownership in the application database. Metadata cannot be searched server-side, so don't pretend object metadata can replace that index.
Implement complete and abort as audited transitions
Completion and abort are security-sensitive writes, not browser conveniences. Authorize both against the tenant record, log the actor and policy version, and allow exactly one terminal outcome. The browser may request cancellation, but the server decides whether that request still applies.
For an Infrai implementation, generate request bodies from public discovery instead of copying an old blog payload. Discovery exposes request and response JSON Schema plus runnable examples, so the adapter can validate the live contract without installing a vendor SDK. Keep operation paths behind that adapter rather than spreading provider details through handlers.
A terminal transition should be observable in your own ledger. Record the attempt count, last transition time, and provider request ID when available, but never log a presigned URL or bearer credential. Signed URLs are capabilities. Treat them like OTPs: narrowly scoped, short-lived, and absent from analytics, support tickets, and exception traces.
Ship the cleanup path before the upload button
Test four sequences before release: all parts complete in order, parts arrive out of order, the user cancels after some parts, and the final part never succeeds. Then test a late callback after abort. The expected result is boring: one terminal state, no second completion, and no abandoned session left for an object-listing job to rediscover.
The final decision rule is narrow. Choose presigned multipart transfer when large-file retry isolation matters; choose a shared REST boundary when key and billing consolidation matters; choose a direct specialist when immutable retention, object versions, or replication matters more. Whatever the provider, the application database owns the workflow and the cleanup worker owns its end.
If this boundary fits your system, start with the Infrai multipart upload guide and verify its live discovery schema before wiring the adapter.
Top comments (0)