thomasmoore5082

Posted on Aug 31

Node.js Multipart Strategy for Private Generated Image Files

#storage #node #devops

Short answer: for large AI-generated images, keep the bucket private, have the Node.js control plane use multipart upload only when the object warrants it, and publish the database record only after completion and an independent object check agree on the expected size. The tempting alternative is to call the upload complete and move on; under production concurrency, that makes a transport acknowledgement do the work of a durability contract.

The incident lesson is less dramatic than it sounds. A renderer can finish a batch, a worker can submit every part, and a dashboard can show a healthy request-success SLI while the application has still not earned the right to expose a new asset. The invariant is small: a generated image becomes available only after storage verification and the metadata transaction succeed in that order. Build around that invariant, and capacity planning, retry policy, privacy, and cleanup stop pulling in opposite directions.

What should a Node.js service verify before it publishes large AI-generated images to private object storage?

Treat upload completion as a state transition, not an event. The useful states are rendered, uploading, stored, and published; only the last state may be resolved into a signed read URL or an application-visible image record. If a process exits between two states, the durable state record tells a retry worker what it can safely do next.

For an object-store multipart protocol, the sequence is initiate, upload numbered parts, complete with the returned part identifiers, then inspect the resulting object. Amazon S3 documents that the final complete request can start with an HTTP 200 response while processing continues, so clients must read the complete response rather than equating the status line with a successful assembly. Do not acknowledge the job, mark the row stored, or publish a key until the completion response and a subsequent metadata read have both succeeded.

Small rule. Big consequence.

The independent read should compare a known byte length and, where the chosen storage protocol supports it, an explicit checksum. Do not use a multipart ETag as if it were always the object's MD5: multipart ETags are not a portable content-integrity contract. Store the content type, expected byte count, checksum algorithm, object key, and upload attempt ID beside the render. These fields make duplicate delivery boring instead of mysterious. A retry can discover that the exact key already has the expected properties and finish the metadata transition without writing another public-facing record.

Here is the preventative path in Go. The SDK call names are deliberately abstract because the reliability boundary is the ordering, not a library wrapper.

func persistRender(ctx context.Context, store ObjectStore, db RenderStore, r Render) error {
    if err := db.MarkUploading(ctx, r.ID, r.ObjectKey); err != nil {
        return fmt.Errorf("mark upload %s: %w", r.ID, err)
    }

    completed, err := store.MultipartPut(ctx, r.ObjectKey, r.Reader, r.Size, r.ContentType)
    if err != nil {
        return fmt.Errorf("upload %s: %w", r.ObjectKey, err)
    }
    if err := completed.Validate(); err != nil {
        return fmt.Errorf("complete %s: %w", r.ObjectKey, err)
    }

    head, err := store.Head(ctx, r.ObjectKey)
    if err != nil {
        return fmt.Errorf("inspect %s: %w", r.ObjectKey, err)
    }
    if head.Size != r.Size || head.ContentType != r.ContentType {
        return fmt.Errorf("stored properties differ for %s", r.ObjectKey)
    }

    return db.MarkStoredAndPublish(ctx, r.ID, r.ObjectKey, head.ETag)
}

MarkStoredAndPublish should be one database transaction. It does not make object storage transactional; it makes the application claim about that object conditional on evidence. That distinction is where many otherwise tidy upload paths fail.

Multipart is a retry mechanism, not a default file format

Multipart upload earns its extra control-plane calls when transfer time, loss rate, or object size makes restarting an entire object costly. S3-compatible multipart implementations commonly impose a maximum of 10,000 parts, a minimum 5 MiB part size except for the final part, a maximum 5 GiB part size, and a maximum 5 TiB object size. Those limits are architectural inputs, not trivia for a runbook.

Choose a part size from the tail of the object-size distribution, then bound in-flight bytes before raising concurrency. For example, 16 MiB parts with four parts concurrently buffered place up to 64 MiB of part data behind one active upload; eight simultaneous uploads imply up to 512 MiB before runtime overhead, image decoding, queues, or neighboring work. A streaming implementation can reduce retained memory, but it does not erase network buffers or the cost of having several transfers contend for the same worker. Put an explicit concurrency ceiling in the worker configuration and make it part of the pod memory budget.

A single PUT is often the better choice for small, server-produced previews. It has less state to reconcile, fewer credentials or part records to protect, and no incomplete-upload inventory to sweep. The catch is that one disrupted long transfer means starting again. Multipart lets a retry replace one failed part, but it creates an object lifecycle that needs ownership.

Choice	Operational benefit	Cost you must accept	Appropriate boundary
Single streamed PUT	One request path and simple cleanup	A failed long transfer restarts from byte zero	Small objects on a stable internal path
Worker-side multipart	Part-level retries and bounded retry work	Completion, verification, and abandoned-part cleanup	Large images already held by a backend renderer
Direct client multipart	The application avoids carrying media bytes	Authorization, client state, and abandoned uploads	User-provided media over unreliable networks
Self-hosted object storage	Control over placement and operations	Disk capacity, replication, upgrades, and an on-call rotation	Teams with storage operations capacity and placement requirements

This is the buy-versus-build table worth revisiting at planning time. A managed object store reduces the storage fleet burden, while self-hosting can satisfy placement or control requirements; neither decision removes the need for a publish-after-verify application contract.

Keep a private bucket private after the bytes arrive

A private bucket is not achieved by picking a private-looking key prefix. Deny anonymous reads at the bucket policy layer, scope the renderer identity to the required write prefix, scope the delivery service to the required read prefix, and issue short-lived signed read URLs only at the application boundary that has already authorized the caller. Separate original renders from derived display assets, because their retention, audience, and cache behavior are usually different.

Keep generated originals under keys that are hard to enumerate and have no business meaning that a URL could disclose. Key unpredictability is not access control, but it reduces accidental information leakage when logs, traces, or browser histories escape their intended scope. Log the object key carefully: an SLO dashboard needs an upload ID, size bucket, attempt number, and result class more often than it needs a user-correlated path.

Private delivery adds a capacity question that gets skipped in architecture diagrams. A signed URL is a read authorization with a lifetime; when a cache sits in front of it, the cache policy and URL expiry need to agree, or users see authorization failures that look like image failures. Set an explicit read-path SLO, track signed-URL authorization failures separately from storage-read failures, and test expiry behavior with a clock-controlled integration test.

The cleanup and telemetry that keep multipart from becoming storage debt

An initiated upload that never completes leaves parts behind until an abort path removes them. Configure a lifecycle rule to abort incomplete multipart uploads after a retention window that matches the longest credible retry horizon, and also run a reconciler that finds application records stuck in uploading. The lifecycle rule handles storage-side residue; the reconciler handles application-side uncertainty. They solve different problems.

The minimum useful signals are completion latency at p50, p95, and p99; completed uploads by final result; verification mismatches; retry count; age of the oldest incomplete upload; and bytes in flight per worker. Alert on a sustained verification mismatch rate and on stale upload age approaching the lifecycle window. Alerting on raw request volume alone tells an on-call engineer almost nothing about whether users can retrieve the images they were promised.

Test the state machine with forced interruption points: before any part, after a successful part, after completion but before inspection, and after inspection but before the metadata transaction. Then replay the same job ID. The expected outcome is one published record pointing to an object with the expected properties, or a retryable non-published record. No third state should be acceptable.

Before turning up concurrency, run this as a bounded capacity exercise rather than a happy-path benchmark. Feed the worker a representative mix of small previews and files from the upper end of the observed image-size distribution, then hold the arrival rate steady long enough for retries, queueing, and garbage collection to interact. Record the resident set size per active upload, the number of open file descriptors, part-completion latency, and time from render completion to a verified object. Repeat with an interrupted transfer, because a retry can overlap a new upload and create the real peak. The decision is not "can one upload finish?" It is whether the worker remains inside its memory and latency budgets while several uploads are doing ordinary, inconvenient things at once. If p99 verification latency consumes most of the time allowed before a user expects an image, reduce concurrency, enlarge parts only after checking the 10,000-part ceiling, or move the work to a separate pool with an SLO that does not compete with inference. This is also where an on-call team finds out whether the metrics answer the first useful question: are bytes waiting on rendering, transfer, multipart completion, verification, or the database transition? A single generic upload-failure counter cannot answer it.

I'm not sure a universal size threshold for multipart is useful, because link characteristics, renderer locality, and memory limits vary by deployment. Measure the p95 object size, upload duration, retry rate, and worker RSS under representative concurrency; that data resolves the choice better than a rule copied from another team's architecture.

Where a multipart private-bucket design is the wrong fit

Do not impose multipart on every AI image. For a modest preview generated and consumed inside one trusted service boundary, a single streaming PUT plus the same post-write inspection may meet the SLO with much less operational machinery. If image transformation, audience-specific delivery, and asset lifecycle are the dominant work, evaluate a media-management service as a product category, while accounting for its URL model, export path, policy controls, and migration effort before placing originals there.

Likewise, a private bucket does not address content correctness. Storage can preserve every byte of a black frame, an unexpected aspect ratio, or a render that violates a product rule. Put content validation before the published transition when the application needs it, and retain the object-validation step for the separate question of durable storage.

The practical conclusion is narrow: select multipart because the transfer failure budget requires part-level recovery, and select a private bucket because the access model requires it. Publish only after independent verification. Everything else should be justified by observed load and an owned operational boundary.

DEV Community