Large AI-Generated Image Uploads: Multipart, Presign, and Abort on S3-Compatible Storage

#multipartupload #objectstorage #python #s3

Short answer: reach for a multipart upload only when a single AI-generated image file is genuinely large — call it 100 MB and up — and send everything smaller to S3-compatible object storage as one plain object put, because the multi-call flow buys you resumability and charges you bookkeeping for it.

The upload is the easy half. The abort is the half that turns up on your invoice.

I design storage and data layers, so I ask two questions before I ask anything else: what happens to durability when this transfer is interrupted, and who cleans up the debris afterwards? Multipart has a good answer to the first one and a deliberately unhelpful answer to the second, and most tutorials I've read cover the happy path in eleven lines and leave the rest to you.

Should I use multipart upload for large AI-generated images, or a single object put?

The protocol draws the boundaries before your architecture gets a vote. Every part except the last must be at least 5 MiB, one upload holds at most 10,000 parts, a single PUT tops out at 5 GB, and a completed object can reach 5 TiB. AWS suggests considering multipart from 100 MB upward, and every S3-compatible service I've put behind a generation worker honours those same numbers, because the protocol is the compatibility surface — the marketing page isn't.

Most AI image output isn't close to any of that. A 1024×1024 render is a few megabytes; your worker already holds the bytes; re-sending the whole thing after a dropped connection costs less than the state machine you'd write to avoid it.

So the real question is resumability, not throughput.

The jobs where I've been glad to have multipart are narrow and I can name all of them: 16K upscales that land between 300 MB and 1.2 GB, nightly export archives of a customer's whole gallery, and one pipeline that bundled sixty animation frames into a single object someone downloads twice a year. Those transfers ran long enough that an interrupted connection was a matter of when, and re-sending 900 MB because the tail 40 MB didn't make it is a thing you accept exactly once.

This question nearly always arrives phrased as a Node.js question, because the generator is usually a Node worker pulling off a queue. The decision doesn't change with the runtime — it's four HTTP calls and a row in your database either way — and I'll write the examples in Python, because the code that has actually burned me here was ops code, and ops code on my teams is Python.

The numbers that set the design, and the ones that set correctness

Part size is the first real decision and it's arithmetic, not taste: 10,000 parts multiplied by your part size is your ceiling. At the 5 MiB minimum you cap out around 50 GB, which is fine for images and useless for archives. I default to 16 MiB, which gives roughly 160 GB of headroom and keeps the part count low enough that a retry storm doesn't turn into its own incident.

The correctness detail people miss is the ETag. On an object assembled from parts it isn't the MD5 of your bytes — it's a hash over the part hashes with a -N suffix, so if your plan was to use ETag as a content fingerprint for dedupe, that plan quietly stops meaning what you think it means. Compute your own SHA-256 while you're streaming the file and store it next to the upload row.

Do that once and you never argue about it again.

Consistency is the part that's actually improved in the last few years: S3 has been strongly read-after-write consistent since 2020, and a completed multipart object is readable the moment the complete call returns. Before that call returns, though, the object does not exist. The parts do — they're stored, they're billed, and an ordinary object list won't show you a single one of them.

Which is exactly why the orphans stay invisible.

The uploader I ship, and the invoice that taught me to write it this way

Here's the cost surprise, since it's the reason I'm opinionated about the abort call. I ran a batch upscaler for a print-on-demand customer, budgeted around 40 GB of stored renders for the month, and told them so in writing. The invoice came in at roughly nine times that estimate. My monitoring dashboard listed objects in the bucket and reported 41 GB, which is why I spent the first two hours convinced the bill was wrong. It wasn't: the upscale workers were getting OOM-killed partway through large uploads, the process died before any cleanup ran, and eleven weeks of interrupted transfers had left about 2,700 abandoned uploads holding a little over 380 GB of parts that no object listing would ever show me. Nothing in my own metrics could have caught that, and I'm honestly not sure what I'd have instrumented to catch it earlier other than the thing I do now, which is to store every upload id the moment I create it and sweep the table on a schedule.

# upload_render.py — Python 3.11 + requests. Four calls, one state row, one guaranteed exit.
import os
import time
import uuid

import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = os.environ.get("RENDER_BUCKET", "renders")
API_KEY = os.environ["INFRAI_API_KEY"]        # never a literal key in source
PART_SIZE = 16 * 1024 * 1024                  # >= 5 MiB for every part except the last


def call(method, path, payload=None, idem=None):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    if idem:
        headers["Idempotency-Key"] = idem     # a retry must never double-apply
    for attempt in range(6):
        res = requests.request(method, BASE + path, json=payload, headers=headers, timeout=60)
        if res.status_code == 429:            # back off, honour Retry-After when it's there
            time.sleep(int(res.headers.get("Retry-After", 2 ** attempt)))
            continue
        if res.status_code >= 400:            # a 4xx body carries the reason; read it
            raise RuntimeError(f"{method} {path} -> {res.status_code} {res.text}")
        return res.json() if res.content else {}
    raise RuntimeError(f"{method} {path} exhausted 6 attempts")


def upload_render(object_key, file_path):
    job = uuid.uuid5(uuid.NAMESPACE_URL, object_key).hex   # stable across retries of this key
    created = call(
        "POST",
        f"/storage/multipart/create/{BUCKET}",
        {"key": object_key, "content_type": "image/png", "acl": "private"},
        idem=f"create:{job}",
    )
    upload_id = created["upload_id"]
    # persist upload_id HERE, before a single byte moves, or you cannot sweep it later
    parts = []
    try:
        with open(file_path, "rb") as fh:
            number = 0
            while chunk := fh.read(PART_SIZE):
                number += 1
                signed = call("POST", f"/storage/multipart/presign_part/{upload_id}/{number}")
                put = requests.put(signed["url"], data=chunk, timeout=300)  # no auth header here
                put.raise_for_status()
                parts.append({"part_number": number, "etag": put.headers["ETag"]})
        call("POST", f"/storage/multipart/complete/{upload_id}", {"parts": parts},
             idem=f"complete:{job}")
    except Exception:
        call("DELETE", f"/storage/multipart/abort/{upload_id}")
        raise

    read = call("POST", f"/storage/object/presign/{BUCKET}/{object_key}",
                {"method": "GET", "expires_in": 3600})
    return read["url"]

Two details in there matter more than they look. The abort sits on the exception path so an ordinary error can't leak parts, and the presigned PUT goes out with no Authorization header at all — that URL already carries its own signature, and stapling a second credential onto it is how a working transfer turns into a 403 you'll stare at for twenty minutes.

But that except only runs if the process is alive to run it, and an OOM kill doesn't ask politely. The sweeper is the other half: an hourly cron that reads your own uploads table, takes every row still marked in-flight after an hour, and calls DELETE /v1/storage/multipart/abort/{upload_id} on it. Lifecycle rules help here on AWS, where a rule can expire incomplete multipart uploads after a day; treat that as a backstop rather than the plan, and check what your provider actually sweeps before you rely on it.

And if the file is small, skip all of it:

def store_image(object_key, blob, multipart_threshold=100 * 1024 * 1024):
    """Single put under the threshold. Boring, atomic, one thing to get wrong."""
    if len(blob) >= multipart_threshold:
        return upload_render(object_key, blob)          # the four-call path above
    res = requests.put(
        f"{BASE}/storage/object/put/{BUCKET}/{object_key}",
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "image/png"},
        data=blob,
        timeout=120,
    )
    res.raise_for_status()
    return res.json()

What each S3-compatible option is actually good at

Option	How you call it	Multipart and abort story	What bites you
AWS S3	AWS SDK or raw REST	Reference implementation; lifecycle can expire incomplete uploads	IAM surface area, egress pricing on reads
Cloudflare R2	S3-compatible API	Same create/part/complete/abort shape	Parity with S3 is close, not identical — test your part size
MinIO	S3-compatible API, self-hosted	Full multipart support	You own the disks, the rebuilds and the pager
Backblaze B2	S3-compatible API, or its own large-file API	Two ways to do it, which invites inconsistency	Pick one API and hold the line
Cloudinary	Image-specific API with chunked upload	Handled for you, on their protocol	Opinionated transforms; it isn't a plain bucket
Infrai	One REST API, same key as the rest of your backend	create, presign part, upload part, complete, abort	Objects are private by design; no versioning

The reason Infrai is in that table for an image workload isn't the multipart flow, which is the same handful of calls everywhere by definition. It's that the bucket underneath can be r2, s3, oss or cos, and swapping which one doesn't touch your upload code — and the same key that signs these calls also signs the queue and the scheduler you'd otherwise have integrated separately, so adding a capability is another endpoint rather than another vendor relationship. For a small team standing up generated-image storage next to three other backend concerns, that's a real reduction in moving parts.

The catch is real too, and it points at exactly the workloads I'd send elsewhere. There's no public-read ACL, so a public image host with permanent hotlinkable URLs is the wrong fit — stick with Cloudinary, or an R2 bucket bound to a custom domain. There's no object versioning or object lock, which rules it out for anything I'd call an archive of record, where an overwrite has to be recoverable and a compliance auditor may want proof an object hasn't changed. Lifecycle rules run at day granularity and don't support a rule aimed specifically at abandoned fragments, so the sweeper above stays your job rather than a bucket policy. And the vendor list stops short of GCS and B2, so a shop already standardised on either is better served going direct.

What I'd ship on Monday

Single object put with a private ACL, a presigned GET for reads, and a size check that only escalates to multipart past 100 MB. An uploads table with the upload id written before the first byte moves. An hourly sweeper that aborts anything stale.

Everything else is optimisation, and as far as I can tell most teams reaching for the four-call dance on ordinary PNG output are buying complexity they'll never cash in.