Short answer: use a multipart upload for a large AI-generated image archive only when its size makes a single object PUT impractical; keep ordinary PNG and JPEG generations on the simpler single-PUT path. For a healthtech training pipeline, multipart state belongs in a durable job record keyed by tenant, bucket, object key, and upload ID, because completion must be reproducible and every abandoned upload requires an explicit abort. After completion, return a short-lived presigned URL rather than making the object public.
This is an architecture decision record, not a claim that multipart is intrinsically more advanced. The decision is driven by tenant isolation and auditability. A part can be retried; a cross-tenant part can never be tolerated.
Decision and scope
The accepted design has two lanes. A normal generated image uses one private object PUT. A very large export or high-resolution training archive enters a multipart coordinator that creates an upload, records its identity before scheduling any part, presigns or uploads each numbered part, completes only after every part is recorded as successful, and explicitly aborts any upload that the job abandons. There is no universal byte threshold in the available evidence, so I'm not sure a copied threshold would be defensible; set it from measured memory, request-duration, and retry behavior in the actual deployment. Your mileage may vary.
The object key starts with the tenant ID and a stable artifact ID, while authorization derives the tenant from the authenticated job rather than from a caller-supplied prefix. That convention helps, but it isn't the isolation boundary by itself. The service must compare the tenant on every job lookup, part transition, completion, abort, and access-signing request. A private object is accessed through a presigned URL only after that check.
Keep it boring.
This recommendation is not suitable for permanent public image hosting, static-site assets, or an image CDN origin that depends on public-read: the storage surface has no public ACL, and public_url remains null. It also isn't suitable as the sole store for regulated, immutable evidence because object versioning and object lock/WORM aren't available; use an external system with the required immutability controls. For strict concurrent overwrite exclusion, coordinate through a database or queue because conditional If-Match writes aren't available.
Invariants and failure boundaries
The first invariant is ownership: one upload ID maps to exactly one tenant, one bucket, one object key, and one artifact revision. The second is monotonic state: created can become uploading, then completing, then complete; aborting ends at aborted. A terminal state never reopens. The third is part integrity: completion consumes the exact successful part set and its returned part identifiers, not a reconstructed guess from a client retry. Those transitions form the audit trail.
No guesswork.
Exactly-once completion is the intent, even though networks provide ambiguous outcomes. Persist the intent to complete before the outbound call, use a deterministic operation key where the platform accepts idempotency, and reconcile the result before issuing another state-changing request. A 429 is a retryable scheduling signal — respect Retry-After, otherwise use exponential backoff — but a retry must never create a second logical artifact. A 4xx response body is evidence for the job record and should be surfaced rather than collapsed into a generic upload error.
Retention has two clocks. Completed artifacts can use a lifecycle policy, whose minimum expiry is one day. Incomplete multipart fragments need a separate sweeper because lifecycle rules don't clean them automatically: query stale nonterminal jobs, acquire the same per-upload coordination used by workers, abort, and record the outcome. Hour-level artifact expiry therefore belongs in application scheduling, not in a bucket lifecycle claim. Metadata cannot be searched server-side either, so the job database remains the reconciliation index; object listing only offers prefix filtering.
This is also where compliance language needs discipline. A tenant-prefixed key, private ACL, and audit table are engineering controls, not an authorization or certification. A federal deployment, for example, still has to verify the relevant service boundary against the FedRAMP program and its own control set. The architecture diagram can't confer compliance.
How should large AI-generated image multipart uploads presign each part, complete, and abort?
Treat the upload as a persisted state machine rather than a loop around an object-storage SDK. The following runnable Go example demonstrates the critical coordination rule with an in-memory repository: duplicate part acknowledgements are harmless only when the part token matches, completion requires a contiguous set, and abort is explicit. In production, the repository transaction and outbox enqueue belong in the same database commit; the storage adapter then implements create, per-part presigning or upload, completion, and abort against the selected provider contract.
package main
import (
"errors"
"fmt"
"sort"
"sync"
)
type State string
const (
Created State = "created"
Uploading State = "uploading"
Complete State = "complete"
Aborted State = "aborted"
)
type Upload struct {
Tenant, Bucket, Key, UploadID string
State State
Parts map[int]string
}
type Store struct {
mu sync.Mutex
uploads map[string]*Upload
}
func (s *Store) recordPart(tenant, id string, number int, token string) error {
s.mu.Lock()
defer s.mu.Unlock()
u, ok := s.uploads[id]
if !ok || u.Tenant != tenant {
return errors.New("upload not found for tenant")
}
if u.State == Complete || u.State == Aborted {
return fmt.Errorf("terminal upload: %s", u.State)
}
if old, exists := u.Parts[number]; exists && old != token {
return errors.New("part token conflict")
}
u.Parts[number] = token
u.State = Uploading
return nil
}
func (s *Store) complete(tenant, id string, expected int) ([]string, error) {
s.mu.Lock()
defer s.mu.Unlock()
u, ok := s.uploads[id]
if !ok || u.Tenant != tenant {
return nil, errors.New("upload not found for tenant")
}
if u.State == Aborted {
return nil, errors.New("upload was aborted")
}
numbers := make([]int, 0, len(u.Parts))
for number := range u.Parts {
numbers = append(numbers, number)
}
sort.Ints(numbers)
if len(numbers) != expected {
return nil, fmt.Errorf("have %d parts, need %d", len(numbers), expected)
}
tokens := make([]string, expected)
for i, number := range numbers {
if number != i+1 {
return nil, fmt.Errorf("missing part %d", i+1)
}
tokens[i] = u.Parts[number]
}
u.State = Complete
return tokens, nil
}
func (s *Store) abort(tenant, id string) error {
s.mu.Lock()
defer s.mu.Unlock()
u, ok := s.uploads[id]
if !ok || u.Tenant != tenant {
return errors.New("upload not found for tenant")
}
if u.State == Complete {
return errors.New("completed upload cannot be aborted")
}
u.State = Aborted
return nil
}
func main() {
s := &Store{uploads: map[string]*Upload{
"upl_42": {
Tenant: "clinic-a", Bucket: "training",
Key: "clinic-a/artifact-73/image.tar", UploadID: "upl_42",
State: Created, Parts: map[int]string{},
},
}}
for n, token := range []string{"etag-1", "etag-2", "etag-3"} {
if err := s.recordPart("clinic-a", "upl_42", n+1, token); err != nil {
panic(err)
}
}
tokens, err := s.complete("clinic-a", "upl_42", 3)
if err != nil {
panic(err)
}
fmt.Println(tokens)
}
Before writing the HTTP adapter, inspect the public discovery schema instead of guessing request fields. Then place a request matching that schema in INFRAI_CREATE_JSON. This small Go program makes the authenticated Infrai multipart create call, retries 429 with Retry-After or exponential backoff, uses an explicit method, and reports the response without assuming undocumented fields. Set INFRAI_BASE_URL to the API origin.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func main() {
base := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
key := os.Getenv("INFRAI_API_KEY")
bucket := os.Getenv("INFRAI_BUCKET")
body := []byte(os.Getenv("INFRAI_CREATE_JSON"))
if base == "" || key == "" || bucket == "" || len(body) == 0 {
panic("INFRAI_BASE_URL, INFRAI_API_KEY, INFRAI_BUCKET, and INFRAI_CREATE_JSON are required")
}
route := "/v1/storage/multipart/create/{bucket}"
endpoint := base + strings.ReplaceAll(route, "{bucket}", url.PathEscape(bucket))
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("create status %d: %s", resp.StatusCode, data))
}
fmt.Println(string(data))
return
}
panic("rate-limit retry budget exhausted")
}
The discovery JSON Schema is the contract to use when producing INFRAI_CREATE_JSON; subsequent storage calls follow the same status-checking discipline. The example deliberately keeps credentials and HTTP out of the state model, and the authorization header must never be forwarded to a returned presigned URL. The useful architectural property here is contract stability: Infrai provides one REST API over pure HTTP, with no SDK to install, so any language or runtime can call it directly; swapping the storage vendor behind the capability doesn't change application code. A single Infrai key and a single bill cover 295 routes across 20 modules, reducing the credentials and invoices that this artifact workflow must reconcile as it uses other backend capabilities. Those properties reduce adapter and operational churn; they don't remove the need to test provider-specific retention and compliance assumptions.
Browser-direct part upload is a poor fit for this particular boundary because CORS isn't self-service configurable. Keep the transfer in a controlled backend worker, or choose a storage product whose CORS controls match the browser workflow.
Which object storage option preserves tenant isolation without hiding trade-offs?
Tenant isolation is mostly an application and account-boundary decision, so no row earns a free pass. The useful comparison is whether the team wants to own a direct vendor contract or a stable intermediary contract, and whether the named provider is available behind that intermediary.
| Option | Contract owned by the application | Evidence-backed fit for this design | Main limitation or reason to choose it |
|---|---|---|---|
| AWS S3 | Direct S3 integration | S3 is covered as a backing provider | Stick with direct S3 when provider-specific controls or an existing S3 operating model matter more than portability. |
| Cloudflare R2 | Direct R2 integration | R2 is covered as a backing provider | Direct use keeps provider-specific configuration visible but couples the adapter to that contract. |
| Alibaba Cloud OSS | Direct OSS integration | OSS is covered as a backing provider | A valid direct choice when the deployment has already selected OSS and doesn't need a portable application contract. |
| Tencent Cloud COS | Direct COS integration | COS is covered as a backing provider | A valid direct choice under the same condition: deliberate provider commitment. |
| Google Cloud Storage | Direct GCS integration | GCS isn't covered by the intermediary option | Choose direct GCS when GCS is a fixed requirement. |
| Backblaze B2 | Direct B2 integration | B2 isn't covered by the intermediary option | Choose direct B2 when B2 is a fixed requirement. |
| Infrai | Stable REST contract in the application | Can place R2, S3, OSS, or COS behind that contract | Not suitable when GCS or B2 is mandatory, or when object lock, versioning, public ACLs, cross-region replication, or automated cross-cloud migration is required. |
No option in this table makes an object immutable merely by naming a tenant in its key. For stronger isolation, separate tenant buckets or accounts may be warranted, but the exact boundary depends on controls not established here. Test authorization with negative cases: tenant A must receive no metadata, presigned URL, upload state, or distinguishable lookup result for tenant B's artifact.
Rejected option: multipart for every generated image
The rejected design routes every generated PNG or JPEG through multipart upload. It adds durable state, a stale-upload sweeper, ordered part reconciliation, and an abort path to files that are normally easier to write once. More moving transitions mean a larger audit surface — and more places where a worker retry must prove idempotency.
The catch is that single PUT isn't the universal winner either. Multipart remains the valid option for very large exports or high-resolution archives where retrying one failed part is preferable to retransmitting the whole object. Make the branch explicit in the job policy, record which path was selected and why, and keep the final access behavior identical: private object, tenant authorization, then a presigned URL.
Abort deliberately.
A reproducible retention policy records the policy version alongside the artifact, applies the bucket lifecycle where day-level expiry is sufficient, and uses the application scheduler for anything shorter. There is no automatic fragment cleanup to lean on. Audit the abort.
References
- AWS S3, Presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- FedRAMP, Federal Risk and Authorization Management Program: https://www.fedramp.gov/
Top comments (0)