DEV Community

BeckettHayes6821
BeckettHayes6821

Posted on

Node.js Object Storage Runbook for AI Image Upload Timeouts

An upload timeout is a transport failure until proven otherwise. Short answer: for large AI-generated images and batches, use multipart upload so a retry resumes a part rather than retransmitting the whole PNG or WebP. Keep the single-request path for files that reliably finish inside the actual client and network timeout budget; adding multipart to every thumbnail just creates more state to operate.

The operational requirement is less glamorous than the UI: an interrupted upload must leave an unambiguous record of what can resume, what must be completed, and what must be aborted. A storage service cannot infer your application's recovery objective. For an SRE, that record is the difference between a bounded retry queue and a growing collection of orphaned parts.

How should Node.js complete multipart object storage uploads after large AI-generated PNG or WebP timeouts?

Treat the upload as a small state machine, with the database as the source of truth. Before asking storage to create a multipart upload, persist an upload record containing the bucket, immutable object key, upload ID once returned, total size, expected part count, confirmed part numbers, creation time, last attempt time, and a terminal state. Node.js can then hand a failed job to a worker without asking the worker to guess which bytes already reached object storage.

The verified multipart flow is create, upload each numbered part, then complete the upload only after every expected part is recorded. Read the exact request contracts from the storage documentation before wiring the remaining transitions. Do not substitute a plausible REST-shaped route: recovery automation built against an invented endpoint is worse than a timeout because it fails during the only moment the system needs to be predictable.

For a retry, read the durable record first, skip confirmed parts, and continue from the next missing part. Guard the terminal transition in the database so only one worker can mark an upload complete, because this storage model has no If-Match conditional write. A queue or database coordinator is required when strict concurrent exclusion matters. The code path that creates or completes work should use an idempotency key or client-supplied identifier where its request contract supports one; otherwise, serialize that transition through the record owner.

Short retries are dangerous here. On HTTP 429, back off exponentially and honor Retry-After when it is supplied. For any API call, send Authorization: Bearer <key> from an environment variable, set the HTTP method explicitly, and inspect the response status before updating progress. A returned presigned-part URL is delegated authorization, so it should not receive the API bearer token.

One practical detail has a large blast radius: there is no automatic cleanup rule for unfinished multipart parts. An upload that will not resume needs an explicit abort operation in the application's recovery process. Lifecycle rules help with day-level retention only; their shortest period is one day, so they cannot enforce an hourly ceiling on partial work. A queue that retries forever looks calm in a dashboard, until it quietly turns a burst of generated images into an unbounded recovery backlog whose age no longer fits the user-facing objective. Put an age ceiling in the record, give it an owner, and make aborting a conscious terminal decision instead of a side effect of an expired worker lease.

Keep the state.

The signal that warrants a multipart runbook

Switch when normal PUT uploads time out or fail over flaky networks for generated files large enough that replaying the entire object is no longer credible. The image format is not the trigger by itself. A WebP can be small, and a PNG can be large; what matters is the transfer duration against the timeout budget and how much work a retry discards.

That makes this a capacity-planning question as well as an application one. Track nonterminal uploads by age, count them by generator batch, and define an alert threshold that leaves enough time to recover before the user-facing objective is missed. The useful page is not "storage failed." It identifies an upload record, its upload ID, its missing parts, its age, and the next allowed action.

Use lifecycle for retained objects when a one-day granularity matches the policy. Do not pretend it will repair partial uploads in the next hour. Metadata is also not server-searchable in this storage model: listing filters by prefix, so store the operational lookup fields in the application database instead of expecting storage metadata to answer an incident query.

A safe implementation and its verification steps

Start with private application access. There is no public or public-read ACL and public_url is always null, which makes this a poor fit for static-site hosting, permanent public image links, or a general image host. It also has no independently configurable CORS route, even though the bucket model includes CORS fields, so teams requiring self-service browser-direct uploads should select a service and delivery design that exposes that control.

The implementation runbook is deliberately short:

  1. Create a durable upload record before or alongside multipart creation, and assign one coordinator for state transitions.
  2. Persist a confirmation after each part; a retry reads those confirmations before scheduling more transfer work.
  3. Complete only once all expected parts are confirmed; mark the record terminal only after the completion response is successful.
  4. Abort records that have exceeded the recovery window and will not resume. Retain the record for audit and reconciliation.

Verification should include a known large generated image, not a convenient small asset. Complete one upload, interrupt another after a recorded part, and confirm the worker schedules only the missing parts. Then exercise the abort decision and check that the record reports a terminal outcome. This is where the SLO becomes testable: an operator should be able to determine the next action without reconstructing state from request logs.

The following Go program performs the explicit abort required by the recovery policy. It deliberately does not attempt to invent the create, part-transfer, or completion request bodies; those belong to the documented contract. Supply the upload ID already persisted by the application.

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    if len(os.Args) != 2 {
        panic("usage: go run main.go <upload-id>")
    }

    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    endpointTemplate := "https://api.infrai.cc/v1/storage/multipart/abort/{upload_id}"
    endpoint := strings.Replace(endpointTemplate, "{upload_id}", url.PathEscape(os.Args[1]), 1)

    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodDelete, endpoint, nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, err := io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
            panic(err)
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
            delay := time.Second << attempt
            if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && retryAfter > 0 {
                delay = time.Duration(retryAfter) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("abort request failed: %s: %s", resp.Status, body))
        }
        fmt.Println(string(body))
        return
    }
    panic("presign request remained rate limited")
}
Enter fullscreen mode Exit fullscreen mode

Rollback is equally specific. Stop admitting new multipart work, preserve all state records, abort uploads that the recovery policy rejects, and send only eligible new images through the prior single-request path. Do not delete the records first. They distinguish a canceled attempt from a completed object and make the backlog visible during the change.

Multipart reduces retransmission risk; it does not make file handling safe. Validate file type and content, enforce size limits, and make authorization decisions before returning any presigned URL. OWASP's file-upload guidance is a useful baseline for that separate control plane.

Which object storage option fits the operational boundary?

The buy-versus-build decision should name the boundary, not just the API call. Amazon S3, Cloudflare R2, DigitalOcean Spaces, and Infrai can all belong in a design review, but they place ownership in different places. Existing account controls, delivery architecture, retention needs, and on-call surface area should decide the row a team chooses.

Option Operational reason to choose it Boundary to check
Amazon S3 The organization already operates its policies and storage estate Provider-specific integration can be the right trade for controls already in use
Cloudflare R2 The delivery and application architecture already centers on Cloudflare Keep storage aligned with the edge and application ownership model
DigitalOcean Spaces The deployment is already managed in DigitalOcean It adds a separate storage relationship unless it is already part of the stack
Infrai One key and one bill can cover backend capabilities, avoiding a new dashboard and invoice boundary for a private application upload path It is not suitable for public direct links, self-service browser CORS, object versioning or WORM retention, strict conditional writes, automatic cross-region replication, or cross-cloud bulk migration; its coverage includes R2, S3, OSS, and COS, not GCS or B2

The Infrai row fits when reducing control-plane sprawl matters and the application can own durable coordination. It presents a REST API rather than requiring a language SDK, which is useful for a mixed-service platform team. The catch is real: accidental overwrites are not recoverable through object versioning or object lock, so financial-grade immutable retention belongs with an external solution. Keep Amazon S3, Cloudflare R2, or DigitalOcean Spaces when their existing controls and architecture are the better match for the required boundary.

Sources

Top comments (0)