Short answer: the storage API pattern I choose for private marketplace files uses backend-issued presigned URLs for direct browser uploads, server-side auth and validation, then a separate object HEAD check before readiness; callback webhooks are only for asynchronous processing.
The browser should move the bytes, but it shouldn't decide ownership. This keeps large-file traffic away from the application process while preserving an authorization point that can answer a plain question: may this user write this exact key?
One rule matters most: an upload response is not a readiness signal.
Keep that boundary.
Why is completion a separate signal?
A marketplace has at least two states that look deceptively similar: the browser finished sending bytes, and the application accepted an object for download. Collapsing them creates the storage equivalent of acknowledging a queue message before the side effect commits. A dropped callback, a stale client tab, or a repeated request can then make the database disagree with object storage. The safer state machine is pending -> uploaded -> ready, with the final transition owned by the backend after verification.
I don't let the client submit an arbitrary path. After application auth succeeds, the backend chooses a key that contains server-owned identifiers, such as the listing and upload IDs, and records that allocation before returning a signed request. A retry for the same logical upload must resolve to the same allocation. That's the idempotency reflex: duplicate delivery is ordinary input, not an exceptional event. Consider a seller who taps retry after the progress display stalls at the end: both attempts must resolve through one upload record, one allocated key, and one readiness transition. If each callback can invent state, the second arrival can overwrite the first decision or attach another seller's object. The database record prevents that ambiguity because every callback has to present an upload ID, pass ownership validation, and resolve to the key already reserved by the server.
Retries will happen.
For large files, direct transfer also protects application capacity. The application handles small control-plane messages rather than proxying every byte through a Node or Express worker. The actual throughput ceiling still depends on the selected storage vendor, region, network path, and multipart behavior; I'm not sure which one wins for your workload without a representative upload test from the browsers and regions you serve. Measure completion time, retry volume, and abandoned multipart uploads.
How should Node Express handle direct browser uploads, auth validation, and callback webhooks?
The same sequence applies even though the runnable client below is Go: authenticate in the application, reserve a safe key, request the signed upload, let the browser upload directly, then ask the storage control plane for object metadata before changing the database state. A Node/Express service should keep those boundaries in middleware and a storage adapter rather than mixing them into one callback handler.
Treat the presigned response as opaque provider data. The browser uses the returned signed request exactly as issued and must not attach the Infrai bearer credential to that storage URL. After the browser reports completion, the backend calls the object-head capability and validates the expected identity and application policy before publishing a download link. Because the supplied interface does not establish which response fields encode size or checksum, the exact field checks should come from live discovery for the capability instead of being guessed in application code.
This minimal Go program exercises only the two verified control-plane routes. It generates a server-owned key, sends an explicit method on each request, retries HTTP 429 with Retry-After support, supplies an idempotency key for the write-like presign operation, and surfaces every other non-success response. It prints the JSON unchanged, so it doesn't invent a response schema.
package main
import (
"bytes"
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func main() {
if len(os.Args) != 4 {
fmt.Fprintln(os.Stderr, "usage: uploadctl presign|head BUCKET SELLER_ID")
os.Exit(2)
}
method := http.MethodPost
key := os.Args[3]
idempotencyKey := ""
if os.Args[1] == "presign" {
var err error
key, err = objectKey(os.Args[3])
if err != nil {
panic(err)
}
idempotencyKey = "marketplace-upload-" + strings.ReplaceAll(key, "/", "-")
} else if os.Args[1] == "head" {
method = http.MethodGet
} else {
fmt.Fprintln(os.Stderr, "action must be presign or head")
os.Exit(2)
}
path := "/storage/object/presign/" + url.PathEscape(os.Args[2]) + "/" + url.PathEscape(key)
if method == http.MethodGet {
path = "/storage/object/head/" + url.PathEscape(os.Args[2]) + "/" + url.PathEscape(key)
}
body, err := call(method, path, idempotencyKey)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
func objectKey(sellerID string) (string, error) {
if sellerID == "" || strings.ContainsAny(sellerID, "/\\") {
return "", fmt.Errorf("invalid seller ID")
}
id := make([]byte, 16)
if _, err := rand.Read(id); err != nil {
return "", err
}
return "marketplace/" + sellerID + "/" + hex.EncodeToString(id), nil
}
func call(method, path, idempotencyKey string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if key == "" || baseURL == "" {
return nil, fmt.Errorf("INFRAI_API_KEY and INFRAI_BASE_URL are required")
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, baseURL+path, bytes.NewBufferString("{}"))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
if idempotencyKey != "" {
req.Header.Set("Idempotency-Key", idempotencyKey)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("request failed with status %d: %s", res.StatusCode, data)
}
return data, nil
}
return nil, fmt.Errorf("rate limit retry budget exhausted")
}
In a real service, persist the key returned by the allocation transaction, accept an upload ID from the authenticated client, load the corresponding key, and verify that the same user owns it. Pass that stored key to the head action; only the presign action creates a new allocation key. That database lookup is the callback authorization check. It also makes repeated completion callbacks harmless.
Keep notifications out of the synchronous readiness path unless the product truly requires asynchronous work. Virus scanning, thumbnails, and document processing are good notification consumers. Their handlers should be idempotent because duplicate delivery must not duplicate state transitions. A notification says work is available; the worker still checks current application state before committing results.
Which storage integration fits large-file marketplace throughput?
The first decision isn't a logo. It is whether one consistent control plane or a vendor-specific data path matters more. Infrai is a credible option when the team wants storage alongside other backend modules under one key and one bill, while its single plain REST API requires no SDK and works from any language or runtime. The self-describing discovery surface is public with no key required, and every documented capability has runnable examples in 10 languages. The broader platform exposes 295 routes across 20 modules. That breadth is useful when upload completion later feeds a queue or another backend capability without adding another SDK integration.
The catch is substantial for this use case: browser CORS cannot be configured through a separate self-service route, multipart fragments have no automatic cleanup rule, cross-region replication and bulk cross-cloud migration tools are outside the supported surface, and the available storage vendors cover R2, S3, OSS, and COS rather than GCS or B2. For a high-volume marketplace that needs tuned multipart recovery, independently managed CORS, or migration tooling, I would stick with a direct provider integration.
| Option | What it optimizes | When I would choose something else |
|---|---|---|
| AWS S3 direct | One provider-specific integration and direct control of that account | Choose an abstraction when one contract across backend capabilities matters more |
| Cloudflare R2 direct | A direct R2 integration | Choose another provider when R2 is not an approved storage target |
| Alibaba Cloud OSS direct | A direct OSS integration | Choose another provider when the workload and account live outside OSS |
| Tencent Cloud COS direct | A direct COS integration | Choose another provider when the workload and account live outside COS |
| Infrai | Broad backend capability coverage through a consistent REST surface | Choose direct storage for self-managed CORS, GCS or B2, cross-region replication, or bulk migration tooling |
None of these rows proves a throughput winner. Run the same multipart workload against the short list, including interrupted uploads and regional clients. Large-file averages hide the failure mode I care about: the tail where a nearly complete upload restarts and consumes user time again.
What should verification and rollback cover?
Verification begins before release. Confirm that unauthenticated users cannot obtain a signed request, one seller cannot request another seller's allocated key, and a completion callback cannot mark an unknown upload ready. Then test the unhappy but ordinary cases: two completion callbacks, a callback before the object is visible to the verification path, HTTP 429 from the control plane, and an object whose observed properties violate the application's recorded expectations. The safe response is to leave the database state pending or rejected; never manufacture readiness to make the workflow look healthy.
Keep overwrite prevention in the application. This storage surface has no object versioning, object lock, or conditional If-Match write. Use unique immutable keys, reserve them transactionally, and coordinate strict concurrent writes through a database or queue. WORM-style financial retention needs an external solution. Private delivery is also a hard boundary: public ACLs and permanent public links are not supported, so static-site hosting and public image-hosting workloads should use a different product.
Rollback should be boring. Disable new upload issuance with a feature flag, keep existing signed uploads recognizable by their persisted allocation records, and stop readiness transitions while operators reconcile pending objects with the HEAD check. Don't delete pending records during the rollback; they are the audit trail needed to decide whether an upload can safely resume. If asynchronous workers are enabled, pause consumption before changing notification configuration, then replay only jobs whose idempotency record is absent.
A small runbook is enough: watch pending age, verification rejection count, 429 rate, and duplicate callback count; define who can halt issuance; record the last safe deployment; and practice restoring the state transition without changing object keys. No heroics.
References
- RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110
- Federal Risk and Authorization Management Program: https://www.fedramp.gov/
Top comments (0)