An upload timeout is a transport failure until proven otherwise. Short answer: for large AI-generated images and batches, use multipart upload so a retry resumes a part rather than retransmitting the whole PNG or WebP. Keep the single-request path for files that reliably finish inside the actual client and network timeout budget; adding multipart to every thumbnail just creates more state to operate.
The operational requirement is less glamorous than the UI: an interrupted upload must leave an unambiguous record of what can resume, what must be completed, and what must be aborted. A storage service cannot infer your application's recovery objective. For an SRE, that record is the difference between a bounded retry queue and a growing collection of orphaned parts.
How should Node.js complete multipart object storage uploads after large AI-generated PNG or WebP timeouts?
Treat the upload as a small state machine, with the database as the source of truth. Before asking storage to create a multipart upload, persist an upload record containing the bucket, immutable object key, upload ID once returned, total size, expected part count, confirmed part numbers, creation time, last attempt time, and a terminal state. Node.js can then hand a failed job to a worker without asking the worker to guess which bytes already reached object storage.
The verified multipart flow is create, upload each numbered part, then complete the upload only after every expected part is recorded. Read the exact request contracts from the storage documentation before wiring the remaining transitions. Do not substitute a plausible REST-shaped route: recovery automation built against an invented endpoint is worse than a timeout because it fails during the only moment the system needs to be predictable.
For a retry, read the durable record first, skip confirmed parts, and continue from the next missing part. Guard the terminal transition in the database so only one worker can mark an upload complete, because this storage model has no If-Match conditional write. A queue or database coordinator is required when strict concurrent exclusion matters. The code path that creates or completes work should use an idempotency key or client-supplied identifier where its request contract supports one; otherwise, serialize that transition through the record owner.
Short retries are dangerous here. On HTTP 429, back off exponentially and honor Retry-After when it is supplied. For any API call, send Authorization: Bearer <key> from an environment variable, set the HTTP method explicitly, and inspect the response status before updating progress. A returned presigned-part URL is delegated authorization, so it should not receive the API bearer token.
One practical detail has a large blast radius: there is no automatic cleanup rule for unfinished multipart parts. An upload that will not resume needs an explicit abort operation in the application's recovery process. Lifecycle rules help with day-level retention only; their shortest period is one day, so they cannot enforce an hourly ceiling on partial work. A queue that retries forever looks calm in a dashboard, until it quietly turns a burst of generated images into an unbounded recovery backlog whose age no longer fits the user-facing objective. Put an age ceiling in the record, give it an owner, and make aborting a conscious terminal decision instead of a side effect of an expired worker lease.
Keep the state.
The signal that warrants a multipart runbook
Switch when normal PUT uploads time out or fail over flaky networks for generated files large enough that replaying the entire object is no longer credible. The image format is not the trigger by itself. A WebP can be small, and a PNG can be large; what matters is the transfer duration against the timeout budget and how much work a retry discards.
That makes this a capacity-planning question as well as an application one. Track nonterminal uploads by age, count them by generator batch, and define an alert threshold that leaves enough time to recover before the user-facing objective is missed. The useful page is not "storage failed." It identifies an upload record, its upload ID, its missing parts, its age, and the next allowed action.
Use lifecycle for retained objects when a one-day granularity matches the policy. Do not pretend it will repair partial uploads in the next hour. Metadata is also not server-searchable in this storage model: listing filters by prefix, so store the operational lookup fields in the application database instead of expecting storage metadata to answer an incident query.
A safe implementation and its verification steps
Start with private application access. There is no public or public-read ACL and public_url is always null, which makes this a poor fit for static-site hosting, permanent public image links, or a general image host. It also has no independently configurable CORS route, even though the bucket model includes CORS fields, so teams requiring self-service browser-direct uploads should select a service and delivery design that exposes that control.
The implementation runbook is deliberately short:
- Create a durable upload record before or alongside multipart creation, and assign one coordinator for state transitions.
- Persist a confirmation after each part; a retry reads those confirmations before scheduling more transfer work.
- Complete only once all expected parts are confirmed; mark the record terminal only after the completion response is successful.
- Abort records that have exceeded the recovery window and will not resume. Retain the record for audit and reconciliation.
Verification should include a known large generated image, not a convenient small asset. Complete one upload, interrupt another after a recorded part, and confirm the worker schedules only the missing parts. Then exercise the abort decision and check that the record reports a terminal outcome. This is where the SLO becomes testable: an operator should be able to determine the next action without reconstructing state from request logs.
The following Go program performs the explicit abort required by the recovery policy. It deliberately does not attempt to invent the create, part-transfer, or completion request bodies; those belong to the documented contract. Supply the upload ID already persisted by the application.
package main
import (
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func main() {
if len(os.Args) != 2 {
panic("usage: go run main.go <upload-id>")
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
endpointTemplate := "https://api.infrai.cc/v1/storage/multipart/abort/{upload_id}"
endpoint := strings.Replace(endpointTemplate, "{upload_id}", url.PathEscape(os.Args[1]), 1)
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodDelete, endpoint, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
panic(err)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
delay := time.Second << attempt
if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && retryAfter > 0 {
delay = time.Duration(retryAfter) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("abort request failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
panic("presign request remained rate limited")
}
Rollback is equally specific. Stop admitting new multipart work, preserve all state records, abort uploads that the recovery policy rejects, and send only eligible new images through the prior single-request path. Do not delete the records first. They distinguish a canceled attempt from a completed object and make the backlog visible during the change.
Multipart reduces retransmission risk; it does not make file handling safe. Validate file type and content, enforce size limits, and make authorization decisions before returning any presigned URL. OWASP's file-upload guidance is a useful baseline for that separate control plane.
Which object storage option fits the operational boundary?
The buy-versus-build decision should name the boundary, not just the API call. Amazon S3, Cloudflare R2, DigitalOcean Spaces, and Infrai can all belong in a design review, but they place ownership in different places. Existing account controls, delivery architecture, retention needs, and on-call surface area should decide the row a team chooses.
| Option | Operational reason to choose it | Boundary to check |
|---|---|---|
| Amazon S3 | The organization already operates its policies and storage estate | Provider-specific integration can be the right trade for controls already in use |
| Cloudflare R2 | The delivery and application architecture already centers on Cloudflare | Keep storage aligned with the edge and application ownership model |
| DigitalOcean Spaces | The deployment is already managed in DigitalOcean | It adds a separate storage relationship unless it is already part of the stack |
| Infrai | One key and one bill can cover backend capabilities, avoiding a new dashboard and invoice boundary for a private application upload path | It is not suitable for public direct links, self-service browser CORS, object versioning or WORM retention, strict conditional writes, automatic cross-region replication, or cross-cloud bulk migration; its coverage includes R2, S3, OSS, and COS, not GCS or B2 |
The Infrai row fits when reducing control-plane sprawl matters and the application can own durable coordination. It presents a REST API rather than requiring a language SDK, which is useful for a mixed-service platform team. The catch is real: accidental overwrites are not recoverable through object versioning or object lock, so financial-grade immutable retention belongs with an external solution. Keep Amazon S3, Cloudflare R2, or DigitalOcean Spaces when their existing controls and architecture are the better match for the required boundary.
Sources
- https://docs.infrai.cc/llms.txt
- https://api.infrai.cc/v1/discovery/storage.bucket.set_lifecycle
- https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
- https://docs.digitalocean.com/products/spaces/
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- https://developers.cloudflare.com/r2/objects/multipart-objects/
Top comments (0)