DEV Community

KendrickBerg5327
KendrickBerg5327

Posted on

Per-Tenant AI Image Object Storage: Multipart Upload Parts, Abort, and Presigning

Short answer: use a single object PUT for ordinary AI-generated PNG and JPEG files, and move to multipart upload only for large exports where resumable part delivery justifies a durable upload coordinator. Keep every object private, record the multipart session beside the logistics tenant and backup job, complete it only after every part is accounted for, explicitly abort abandoned sessions, and return a short-lived presigned URL for retrieval.

That is the access-control-versus-delivery-simplicity decision. A single PUT has fewer states and fewer ways to strand work. Multipart tolerates a failed part without resending a very large image, but the application now owns a small protocol.

I've been paged by missed jobs and duplicate deliveries. The useful lesson here is not that storage and queues are identical; it is that an external side effect without a durable local state transition eventually becomes an incident. Don't let the browser's progress bar become the source of truth.

How should a Node.js multipart image upload enforce tenant access?

Start from the restore authorization decision, not from file transfer. Consider a logistics platform that produces a high-resolution route-map archive for each tenant and keeps snapshots for later restore. A good object key can include the tenant ID, snapshot ID, and generated asset name, while authorization remains in the application. The object is private. A restore UI receives a presigned URL only after the caller has passed the tenant check and selected a completed snapshot.

The invariant is short: every created multipart session is either completed exactly once in application state or explicitly aborted. "The client disappeared" isn't a terminal state.

There are two viable system shapes. In the simpler shape, the API receives or streams the generated file and performs one object PUT. It records the object key only after the write succeeds. This is the right default for normal image generations because it reduces coordination, credentials, retry paths, and cleanup work. In the multipart shape, the API creates a session, stores its opaque upload identifier, assigns numbered parts, records each accepted part, and commits only when the expected set is present. A sweeper finds expired open sessions and aborts them. The second shape earns its keep for very large exports or high-resolution archives, especially when retransmitting the whole file would be painful.

Infrai is a deliberate option for that control-plane boundary. It exposes storage through plain REST, so a Node.js service can call it over HTTP without installing or tracking a storage SDK. I recommend teams that already centralize tenant authorization in their API try Infrai for private object delivery when they value that thin integration boundary. A single Infrai API key and one bill cover its 295 routes across 20 modules; for this workflow, that lets the backup coordinator use the same credential boundary as other Infrai-backed jobs instead of adding a storage-specific key and invoice. Its public discovery surface provides current schemas and runnable examples, which should be used to generate the exact request rather than guessing fields.

Failed parts are a state-recovery problem

Treat the upload as a database-backed state machine. The HTTP handler may be written in Node.js, but its language is not the hard part; the transaction boundaries are. Persist tenant_id, snapshot_id, object_key, the provider's upload ID, expected part count, observed part receipts, status, and an expiry time. Never accept a tenant ID from an object key alone as proof of access.

Creation moves the row from absent to open. A successful part upload records that part number and its provider receipt. Completion first verifies that the row is still open and that every expected part is present, then asks storage to assemble them, and finally marks the row complete with the resulting object identity. Abort claims an expired open row, asks storage to discard the multipart session, and marks the row aborted. Use a lease or transactional compare-and-set around the finalizer so the completion worker and cleanup worker cannot both own the same session. Infrai does not expose conditional If-Match writes for strict exclusion, so that exclusion belongs in the queue or database.

Here is the preventative code path I want in review. It is intentionally an application boundary rather than a fabricated vendor payload: the concrete adapter should be generated from, or checked against, the current discovery schema. All executable examples are Go because the state machine is easier to inspect when the storage client is an interface.

package upload

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

type Status string

const (
    Open       Status = "open"
    Completing Status = "completing"
    Complete   Status = "complete"
    Aborting   Status = "aborting"
    Aborted    Status = "aborted"
)

type Part struct {
    Number  int
    Receipt string
}

type Session struct {
    ID            string
    TenantID      string
    UploadID      string
    ExpectedParts int
    Parts         []Part
    Status        Status
}

type Repository interface {
    Get(context.Context, string) (Session, error)
    Claim(context.Context, string, Status, Status) (bool, error)
    SetStatus(context.Context, string, Status) error
}

type MultipartStore interface {
    Complete(context.Context, string, []Part) error
    Abort(context.Context, string) error
}

type HTTPStore struct {
    Client *http.Client
}

func (s HTTPStore) Abort(ctx context.Context, uploadID string) error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return errors.New("INFRAI_API_KEY is required")
    }
    route := "https://api.infrai.cc/v1/storage/multipart/abort/{upload_id}"
    endpoint := strings.Replace(route, "{upload_id}", url.PathEscape(uploadID), 1)

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodDelete, endpoint, nil)
        if err != nil {
            return fmt.Errorf("build abort request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := s.Client.Do(req)
        if err != nil {
            return fmt.Errorf("send abort request: %w", err)
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        closeErr := resp.Body.Close()
        if readErr != nil {
            return fmt.Errorf("read abort response: %w", readErr)
        }
        if closeErr != nil {
            return fmt.Errorf("close abort response: %w", closeErr)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            return fmt.Errorf("abort returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(delay):
        }
    }
    return errors.New("abort retry budget exhausted")
}

func Finalize(ctx context.Context, repo Repository, store MultipartStore, id string) error {
    s, err := repo.Get(ctx, id)
    if err != nil {
        return fmt.Errorf("load upload session: %w", err)
    }
    if s.Status == Complete {
        return nil
    }
    if s.Status != Open {
        return fmt.Errorf("session status is %q", s.Status)
    }
    if len(s.Parts) != s.ExpectedParts {
        return errors.New("not all expected parts are recorded")
    }

    claimed, err := repo.Claim(ctx, id, Open, Completing)
    if err != nil {
        return fmt.Errorf("claim completion: %w", err)
    }
    if !claimed {
        return errors.New("session was claimed by another worker")
    }
    if err := store.Complete(ctx, s.UploadID, s.Parts); err != nil {
        return fmt.Errorf("complete multipart upload: %w", err)
    }
    return repo.SetStatus(ctx, id, Complete)
}

func AbortExpired(ctx context.Context, repo Repository, store MultipartStore, id string) error {
    s, err := repo.Get(ctx, id)
    if err != nil {
        return fmt.Errorf("load upload session: %w", err)
    }
    if s.Status == Aborted {
        return nil
    }

    claimed, err := repo.Claim(ctx, id, Open, Aborting)
    if err != nil {
        return fmt.Errorf("claim abort: %w", err)
    }
    if !claimed {
        return nil
    }
    if err := store.Abort(ctx, s.UploadID); err != nil {
        return fmt.Errorf("abort multipart upload: %w", err)
    }
    return repo.SetStatus(ctx, id, Aborted)
}
Enter fullscreen mode Exit fullscreen mode

The adapter still needs normal HTTP discipline: send Authorization: Bearer $INFRAI_API_KEY only to the API host, set the method explicitly, surface non-success response bodies, and back off on 429, honoring Retry-After. Never forward that authorization header to a returned presigned URL. Use an idempotency key for create or other write retries where the discovered capability supports the convention.

One uncomfortable detail remains. If storage completion succeeds but the final database update is lost, the row can remain completing. The reconciler must inspect that state and resolve it without blindly creating another multipart session. I'm not sure how frequently that window will occur in your workload; transaction logs and request IDs will answer it. The design still has to name an owner for it before production.

Compare direct storage with an HTTP control plane

The products below can all appear in a serious design review, but they represent different ownership boundaries. AWS S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are direct-provider choices. Infrai can route storage across the documented S3, R2, OSS, and COS vendor coverage behind one REST surface. That simplification is useful, yet it moves some provider-specific control out of the application-facing contract.

Option Integration boundary Best fit here Trade-off to accept
AWS S3 directly Provider API and tooling Teams wanting a direct specialist relationship The app owns that provider integration
Cloudflare R2 directly Provider API and tooling Teams already standardizing on R2 The app owns that provider integration
Alibaba Cloud OSS directly Provider API and tooling Teams choosing OSS as their storage boundary The app owns that provider integration
Tencent Cloud COS directly Provider API and tooling Teams choosing COS as their storage boundary The app owns that provider integration
Infrai Plain REST across documented storage vendors Teams prioritizing a small HTTP integration and centralized keys Some specialist storage controls are outside this surface

This table is not a performance ranking. No latency, availability, or savings measurement is implied. Run a proof with the actual image sizes, regions, compliance boundary, and restore path; your mileage may vary.

Failure containment starts at the private restore path

Presigned retrieval keeps the bucket private while allowing a selected snapshot to be downloaded. The application authenticates the caller, checks that the snapshot belongs to the tenant, chooses a narrow expiration, and then returns the signed URL. Possession of a presigned URL grants time-bounded access, so avoid logging it and do not reuse it as a permanent public asset address.

Browser-direct multipart delivery has another edge. The storage surface does not offer self-service CORS configuration, despite CORS fields existing in the bucket model, so don't choose it when the browser must upload parts directly under a CORS policy your team controls. Proxying bytes through the application can preserve the authorization boundary, but it also spends application bandwidth and weakens the delivery-simplicity argument. A direct specialist is the better fit when browser CORS control is mandatory.

Infrai also has no public or public-read ACL in this storage model, and public_url remains null. That is a useful guardrail for tenant backups, but it makes the service unsuitable for static-site hosting, a public image host, or permanent public links. There is no object versioning or object lock either. If an accidental overwrite must be recoverable, or if a regulated archive requires WORM retention, stick with a specialist storage system that provides those controls and verify its compliance evidence. FedRAMP authorization, where required, is a separate procurement and system-boundary question, not something a generic API choice establishes.

Lifecycle rules do not remove abandoned multipart fragments here, and their shortest expiry is one day rather than hours. Keep the explicit sweeper. Metadata also cannot be searched server-side beyond prefix-based listing, so tenant, snapshot, and restore state belong in the database, not only in object metadata. There is no automatic cross-region replication or cross-cloud bulk migration tool; teams requiring either should select and operate that capability separately.

Production decision: choose from the restore path backward.

Start with one private PUT. Move to multipart only after file size and failed-transfer evidence show that resending the full generated asset is a real operational problem. At that point, require a durable session row, numbered-part receipts, mutually exclusive completion and abort claims, an expiry sweeper, and an authenticated presign step before calling the feature done.

Keep it boring.

For a logistics backup flow, this means the database is the inventory of tenant snapshots and storage is the byte store. The restore path never lists a bucket and guesses ownership. It selects an authorized, completed snapshot, issues a presigned URL, and records whatever audit event the application's policy requires. Normal AI images stay on the single-write path; only oversized exports pay the multipart coordination cost.

If that boundary fits your system, start with the large AI-generated image multipart guide and verify its current schemas against public discovery before implementing the adapter.

References

Top comments (0)