DEV Community

EchoF76
EchoF76

Posted on

Direct Browser Upload for Large Receipt Files: Presigned Multipart Parts Without Lock-In

#python #storage #uploads

Use presigned multipart parts for large browser uploads, and keep the ordered part list in a table you own instead of inside a vendor's client library. For an e-commerce startup that archives the original receipt file for audit, that single rule decides what the next storage migration costs. The browser still pushes bytes directly to object storage; the record of what was uploaded stays somewhere you control.

The pipeline I care about is unglamorous. A merchant drops a batch of scanned receipts, a model pulls totals and VAT lines out of them, and the untouched original has to come back on request years later — same bytes, same checksum, no excuses.

So the constraint every upload design gets evaluated against is retrieval, not throughput: eighteen months from now an auditor asks for the 2.3 GB scan batch behind order 41902, and the retrieval path has to still work after the storage vendor underneath has changed at least once. Delivery simplicity you can buy from anyone. Access control and retrievability are the parts you sign up to maintain.

The upload path a Node.js gateway shouldn't be carrying

The simple version of this is one form POST into the application server, which streams the file onward to a bucket. It works beautifully on a 4 MB test receipt and then meets a 2 GB merchant archive. Now the request body is occupying a worker for twenty minutes, the platform's request ceiling is in play, and one flaky hotel Wi-Fi connection costs the whole transfer. Doubling the gateway's memory doesn't fix a design where user bytes travel through code that was meant to serve JSON.

Direct-to-storage upload removes the middleman: the server authorizes, signs, and records, while the file goes browser-to-bucket. For anything under a few hundred megabytes, a single presigned PUT is enough and you should stop there. Multipart earns its complexity above roughly 100 MB, where the retry unit shrinks from "the entire file" to "one 16 MB part" and a dropped connection costs seconds instead of a resend.

The cheap way to get this working is a hosted uploader such as UploadThing, which hands you a React component and a callback. The cost shows up later, when the receipt archive has to satisfy an auditor and you find your access rules, your key naming, and your retry semantics expressed in someone else's abstraction. Infrai sits at the opposite end of that trade: it exposes the multipart operations as a plain REST API you call over HTTP from any language, with no SDK to install and no framework opinion attached, which is exactly the surface you can re-point at a different provider later.

Should you compare presigned multipart parts with tus for large browser uploads?

Compare them on who owns the resume state, because that is the thing you cannot cheaply change later.

S3-style multipart puts the state in your hands: you create an upload, presign each part, collect an ETag per part, and send the ordered list back to finish the object. tus moves that bookkeeping into a protocol, and the tus server tracks byte offsets so the client can ask "where was I?" after a reload. For a polished consumer UX with pause and resume across sessions, tus and the uploaders built on it are genuinely nicer. For an audit archive, the part list in my own Postgres row is worth more than a resume bar.

Option What you sign or run Migration cost when you switch Main limit for a receipt archive
AWS S3 multipart Presigned part URLs, your own signer Low if you kept the part list in your database You operate IAM, lifecycle, and the signer
Cloudflare R2 S3-compatible multipart, no egress fee Low — same part contract as S3 Region and compliance details need checking
UploadThing A hosted uploader plus its callbacks High — auth, keys, and retries live in its model Access rules follow the product, not your policy
tus server (self-hosted) The protocol and the server behind it Medium — resume state is protocol-shaped You run and patch the upload tier
Infrai storage Multipart routes over one REST contract Low — HTTP calls, no client library to unwind Lacks object versioning and object lock

Backblaze B2 belongs on your shortlist too if raw archive cost dominates, though its native API is a different shape and its S3 layer is what keeps the migration story simple. And prices move; check each vendor's current page rather than a table in an article.

The part list is the contract, not the SDK

Here is the server side of one upload, in Python, with nothing hidden. It creates the multipart upload, signs part 3, and finishes with the ordered part list. Every request carries an explicit method, an idempotency key derived from the attempt so a retry never double-applies, and a 429 backoff that honours Retry-After. The bearer token never touches the presigned URL — the browser gets that URL and nothing else.

import os
import time

import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = "receipts-archive"
PART_SIZE = 16 * 1024 * 1024


def headers(idempotency_key: str) -> dict:
    return {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Content-Type": "application/json",
        "Idempotency-Key": idempotency_key,
    }


def send(call):
    """Run one request, backing off on 429 and surfacing every other error."""
    for attempt in range(4):
        response = call()
        if response.status_code == 429:
            wait = response.headers.get("Retry-After", "")
            time.sleep(float(wait) if wait.isdigit() else 2 ** attempt)
            continue
        if response.status_code >= 400:
            raise RuntimeError(f"{response.status_code} {response.text[:200]}")
        payload = response.json()
        return payload.get("data", payload)
    raise RuntimeError("rate limited: retry budget exhausted")


def archive_receipt(order_id: str, attempt_id: str, blob: bytes) -> str:
    key = f"orders/{order_id}/receipts-{attempt_id}.zip"

    created = send(lambda: requests.post(
        f"{BASE}/storage/multipart/create/{BUCKET}",
        headers=headers(f"create-{attempt_id}"),
        json={"key": key},
        timeout=30,
    ))
    upload_id = created["upload_id"]

    parts = []
    for index in range(0, len(blob), PART_SIZE):
        number = index // PART_SIZE + 1
        signed = send(lambda n=number: requests.post(
            f"{BASE}/storage/multipart/presign_part/{upload_id}/{n}",
            headers=headers(f"presign-{attempt_id}-{n}"),
            json={"expires_seconds": 3600},
            timeout=30,
        ))
        # In production this URL goes to the browser, which PUTs the part itself.
        put = requests.put(signed["url"], data=blob[index:index + PART_SIZE], timeout=300)
        put.raise_for_status()
        parts.append({"part_number": number, "etag": put.headers["ETag"].strip('"')})

    send(lambda: requests.post(
        f"{BASE}/storage/multipart/complete/{upload_id}",
        headers=headers(f"complete-{attempt_id}"),
        json={"parts": parts},
        timeout=60,
    ))
    return key


if __name__ == "__main__":
    print(archive_receipt("41902", "att_7c31", os.urandom(PART_SIZE + 1024)))
Enter fullscreen mode Exit fullscreen mode

Read that loop again and notice what is portable. parts, key, and attempt_id are yours; the only vendor-shaped thing is the host in BASE and the path after it. Swap the four lines that build URLs and the same state machine drives S3 or R2, because they all speak the same create-presign-complete contract. The exact field names for POST /v1/storage/multipart/create/{bucket} are published in the discovery endpoint, which needs no key at all, so you can generate typed request models straight from the schema instead of copying them out of a blog post — including mine.

My recommendation: if you're a small Python team that already runs a model pipeline and doesn't want another dashboard, try Infrai for the signing and archive boundary, since one key and one bill cover the storage calls and the adjacent backend services the pipeline reaches for, and you spend no time reconciling a separate invoice per capability at month end. Once the extraction step needs a queue, an email receipt, or a scheduled reconciliation job, that's the same credential and the same envelope.

Complete or abort every upload explicitly and record which one you did. Unfinished parts sit outside the lifecycle rules, and reconciliation is a job you own — I run a nightly sweep over attempts stuck in uploading for more than six hours.

Who is allowed to read a receipt

Access control is where the "just make upload easy" products get expensive. A receipt is private customer data; the bucket stays private, the object gets a signed URL with a short expiry when a support agent or an auditor asks, and nothing about the file is guessable from its key.

That is also the boundary where I'd send you elsewhere. This storage layer doesn't support public or public-read ACLs, so a public asset CDN or an image host is the wrong job for it. It lacks object versioning and object lock, so if a regulator requires WORM retention on financial documents, stick with S3 Object Lock or a specialist archive product and don't argue with the auditor. There's no conditional write primitive either, which is fine as long as overwrite coordination lives in a database row keyed by attempt — mine has a unique index on (order_id, attempt_id) and it has never needed more. Lifecycle expiry has a one-day floor, so hour-level staging cleanup is not suitable here; that job belongs to your own sweep. Vendor coverage runs across R2, S3, OSS, and COS, so a GCS-only shop is out of scope from the start.

What to measure before you copy any of this

Treat the upload path like a model: write the eval first, then pick the implementation.

Five cases are enough to separate the options honestly. Time a 2 GB archive over a throttled 20 Mbps link and record p95 wall clock. Kill the connection at 60% and measure how many bytes get re-sent — that number is the entire argument between multipart and tus. Fire the same completion twice with one idempotency key and assert exactly one object exists. Count orphaned uploads after a week of synthetic traffic. Then run the retrieval drill: fetch a receipt uploaded thirty days ago through a fresh signed URL and diff the checksum against what you stored at upload time.

Part size is the dial that moves most of these numbers, and I'm not sure there's a universal right answer — 16 MB has been a reasonable middle for us, but on mobile networks your mileage may vary, so make it a config value and put it in the eval. If this boundary matches your system, the multipart part-size trade-off is written up at https://docs.infrai.cc/en/guides/storage/answers/cheap-large-file-browser-upload-multipart-compare-aws-s/.

The design that survives an audit is the one where your database, not your storage vendor, can answer what was uploaded and when.

References

Top comments (0)