TL;DR
The engineering decision is conditional: multipart is justified for a large AI-generated image when retrying a whole object would harm the upload SLO; the application should create and complete the session, the uploader should send presigned parts directly to S3-compatible object storage, and abort plus lifecycle cleanup belong in the initial design. A single presigned upload remains the least complex choice for modest files and dependable links.
The useful unit here is not an object. It is an upload session with an owner, an expiry, and one terminal state. Treating it that way keeps the data plane out of Node.js while leaving authorization and reconciliation in a service that can be observed.
The incident lesson: bytes are cheap; unfinished state is not
A production review of an image pipeline should start with a boring question: what remains after the process that began an upload has gone away? A renderer can be stopped, a client can lose its network, or the caller can decide a generated image is no longer worth keeping. Parts already accepted by object storage are still storage state until the upload is completed or aborted. They do not become a useful object merely because most of the bytes arrived.
That is why proxying image bytes through a Node.js API is usually an awkward default. The API tier becomes responsible for ingress bandwidth, egress bandwidth, streaming backpressure, connection limits, and a longer request lifetime, while the object store was built to receive the bytes directly. A presigned part URL scopes a client to one intended operation for a limited period; the control service can retain the authority to create the upload, decide which key is allowed, and accept the final manifest.
The failure mode is quieter than a failed request. A service can meet its request-success target while accumulating sessions that never become objects. Capacity planning needs a separate measure: open multipart sessions by age, plus an aborted-to-created ratio. Alert on age before it becomes a cost surprise, and investigate a sustained rise in the ratio as a signal about client networks, job cancellation, or worker lifetime.
Short version: successful delivery is not enough.
Measure it.
Then clean it.
No guesswork.
How should a Node.js service handle multipart presigned uploads, parts, and aborts for large AI-generated images?
Keep three authority-bearing operations on the service: create, issue a signed URL for a numbered part, and complete or abort the upload. The client receives only the upload identifier, a bounded part plan, and short-lived URLs. On completion, it returns the part numbers and ETags it received from the storage service; the server sorts and validates that manifest before it requests completion. The final object becomes visible only after that completion operation succeeds.
Multipart upload has hard protocol limits in the S3 model: an object can have at most 10,000 parts, every part except the last must be at least 5 MiB, and multipart supports objects up to 5 TiB. A single PUT can be up to 5 GiB. Those limits do not automatically make a 50 MiB image a multipart candidate. The decision is driven by retry cost, expected link quality, client memory, and how much session lifecycle machinery the team is willing to own.
A small internal state record is enough to make the ownership explicit: created, uploading, completed, or aborted; bucket, key, expected byte count, expiry, and a caller identity. Completion and abort must be idempotent because a caller may retry after a timeout without knowing whether the prior control-plane request reached storage. Do not accept an arbitrary key or arbitrary part count from the uploader. Both are authorization inputs.
| Design | Retry unit | Control-plane work | Use it when | Avoid it when |
|---|---|---|---|---|
| One presigned PUT | Entire object | Minimal | Images are modest and retrying is acceptable | A failed late retry would consume too much time or bandwidth |
| Presigned multipart parts | One part | Session tracking and cleanup | Large images cross unreliable links | The application cannot operate session reconciliation |
| API proxy streaming | Application-defined | Highest | A gateway must inspect bytes before storage | The API tier would become the data-plane bottleneck |
A Go control path that makes cleanup explicit
The Node.js service can expose these operations, while a Go worker or shared control-plane package implements the state transition. The interface below deliberately names capabilities instead of binding the article to one SDK. It shows the contract that matters: create before signing, record each part, and pick exactly one terminal action.
package upload
import (
"context"
"fmt"
"sort"
"time"
)
type Part struct {
Number int
ETag string
}
type Session struct {
ID string
Bucket string
Key string
ExpiresAt time.Time
}
type MultipartStore interface {
Create(ctx context.Context, bucket, key, contentType string) (Session, error)
PresignPart(ctx context.Context, s Session, number int, expiresAt time.Time) (string, error)
Complete(ctx context.Context, s Session, parts []Part) error
Abort(ctx context.Context, s Session) error
}
type Sessions interface {
Save(ctx context.Context, s Session) error
MarkTerminal(ctx context.Context, id, state string) error
}
func Finish(ctx context.Context, store MultipartStore, sessions Sessions, s Session, parts []Part) error {
sort.Slice(parts, func(i, j int) bool { return parts[i].Number < parts[j].Number })
if len(parts) == 0 {
_ = store.Abort(ctx, s)
_ = sessions.MarkTerminal(ctx, s.ID, "aborted")
return fmt.Errorf("multipart completion requires at least one part")
}
if err := store.Complete(ctx, s, parts); err != nil {
_ = store.Abort(ctx, s)
_ = sessions.MarkTerminal(ctx, s.ID, "aborted")
return err
}
return sessions.MarkTerminal(ctx, s.ID, "completed")
}
This is intentionally incomplete as an HTTP handler, because the part that changes by deployment is identity: a browser, an AI worker, and an internal batch process should not receive the same signing authority. The invariant does not change. Persist the session before a URL is issued, constrain key prefixes and content type in the create operation, and require the final manifest to match the planned range. An ETag is a response value for assembling parts, not a portable promise that it equals a whole-file MD5; applications needing end-to-end integrity should choose and record an explicit checksum strategy.
There is one sharp edge in the control flow. If completion cannot be accepted, try to abort the session before returning the error to the caller, then let a bucket lifecycle rule clean up sessions that escape every application path. The explicit abort provides prompt reclamation and a useful metric. The lifecycle rule is the backstop because application code cannot observe every lost client.
Put a clock on the session before adding concurrency
Concurrency improves elapsed time only while the source can read parts without exhausting memory and the network has usable parallel capacity. Start with a fixed, conservative parallelism limit, collect part latency and retries, then tune against the upload SLO. More workers can make a constrained connection worse.
Expiry needs the same restraint. A URL should be valid long enough for its intended part and a normal retry, but not so long that a leaked URL remains useful for days. The signing service should refuse to issue more URLs after the session expiry and the reaper should use an age comfortably beyond the slowest legitimate transfer. Clock skew between the caller and service belongs in testing, along with duplicate completion requests, reordered parts, cancellation during upload, and a late abort racing a complete request.
For observability, record created sessions, completed sessions, aborted sessions, age of the oldest open session, total bytes planned, and bytes completed. Join these with client type and image size band. A rising retry count isolated to one client class is an engineering lead; a rising oldest-session age is a cleanup lead. They call for different owners.
The accounting view deserves its own dashboard. Suppose a worker starts eight parts, sends five, and is then reclaimed; the object listing remains empty, but those five parts still consume storage until an abort or lifecycle action removes them. A second worker may retry the job under a new session, doubling the stranded bytes. Graph open-session count beside planned bytes, completed bytes, and the age histogram, then set a review threshold tied to the storage budget rather than an arbitrary request count. This catches a capacity problem while it is still a small queue of abandoned state, before a monthly bill or a full bucket forces the conversation.
Where the simpler design wins
The catch is operational weight. Multipart is not suitable when generated images are small enough that a whole-object retry is harmless, when links are controlled, or when the team cannot own an idempotent session record and a cleanup process. Use one presigned PUT in those cases. The shorter code path is a feature.
Compliance requirements may also decide the architecture before performance does. For workloads subject to a federal authorization program or a regional data rule, select an endpoint and audit boundary that meet the applicable requirement, then validate the signed-request design within that boundary. Portability at the API layer does not remove the responsibility to verify residency, logging, access controls, and retention with the organization that assesses the workload.
The practical decision is sober: multipart buys targeted retries and direct-to-storage data flow; it costs state, cleanup, tests, and on-call attention. Make the upload session visible, give it a deadline, and the architecture remains understandable under cancellation.
References
- AWS S3 User Guide, Using presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- AWS S3 User Guide, Uploading and copying objects using multipart upload: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- Federal Risk and Authorization Management Program: https://www.fedramp.gov/
Top comments (0)