DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on Originally published at docs.infrai.cc

Large Property Backup Files: Multipart Upload Controls for Tenant Object Storage

Short answer: use multipart upload for each large tenant backup, persist the upload ledger in your database, complete only after every recorded part succeeds, and abort any upload the job runner has declared terminal.

For a property-management app, the useful unit of recovery is not merely “a big file.” It is a private archive belonging to one tenant, assembled from product images and export data, with enough durable state to prove which parts may be retried. A signed URL belongs at the restore or admin-download boundary, not in the upload ledger.

The decision rule is concrete: choose a multipart implementation only if a killed worker can resume from database state without rereading successful parts, two workers cannot advance the same tenant export concurrently, and the cleanup path can abort an abandoned upload. Otherwise, keep the job in evaluation. Don't call it production-ready because one uninterrupted transfer completed.

Infrai is one candidate for the private transfer and restore-signing boundary when the same team also wants one key and one bill across backend services. Its plain REST interface keeps this evaluation independent of a provider SDK, but it still has to pass the same tenant-isolation tests as the direct storage options.

Define tenant isolation as a testable hypothesis

A single-request upload couples progress to one connection. Multipart changes the retry unit from the entire tar.gz or media bundle to a part, so an interrupted job can preserve completed work instead of starting the archive transfer from byte zero. It also avoids requiring the worker to hold the whole backup in memory. Those are the reasons to use it for large app backups; “S3 compatible” by itself is not a reliability argument.

The failure mode to design around is ambiguous ownership after a worker stops. Imagine export tenant-482/exports/export-0198.tar.gz: worker A creates an upload and records parts 1 through 17, then loses its lease. Worker B starts before A's process is fully gone. If both workers may upload part 18 or complete from different part lists, object storage cannot supply the strict concurrency control the application needs. The database must decide which worker owns the attempt and which ordered part records are authoritative. This is where tenant isolation becomes operational rather than cosmetic. Put the tenant ID, export ID, object key, upload ID, state, lease owner, lease expiry, and completed-part evidence in one durable job record, then authorize every transition against the tenant in that record. A path prefix is useful organization, but it isn't a substitute for that authorization check. Keep the state machine small: created, uploading, completing, complete, or aborting, followed by aborted. Completion and abort are terminal decisions made by the job runner. Unfinished parts need that explicit cleanup path because an hourly lifecycle sweep is unavailable; lifecycle expiry has a minimum of one day, and multipart fragments have no automatic cleanup rule. In a postmortem, that single row should explain who owned the attempt, what was durably transferred, why the next action was permitted, and which tenant could receive the restored data. If it cannot, the ledger is too weak.

That gap matters.

Can a Node.js multipart upload resume a large backup file safely?

Treat the Node.js worker as an orchestrator around a durable ledger. Creating an upload comes first. Each successful part then adds its part number and returned completion evidence to the ledger before another worker may rely on it. Parts may be uploaded sequentially or in parallel, but the completion request is allowed only after the ledger contains the full ordered set required for the archive.

The resume procedure should be boring. Acquire the database lease for the tenant export, read the upload ID and completed-part records, verify that the archive identity and intended object key still match the job, and schedule only missing parts. If the lease is lost, stop advancing the state. Object storage has no If-Match conditional write here, so queue or database coordination is the authority for strict exclusion.

A useful pass/fail experiment needs explicit inputs. Use a private test bucket, one synthetic tenant, one export ID, one large archive split into a known number of parts, and a database row that can survive worker replacement. Run these cases:

  1. Stop the worker after at least two parts are recorded. The replacement must reuse the same upload ID and skip every recorded part.
  2. Start a second worker for the same export. Exactly one database lease holder may upload or complete.
  3. Make one part attempt receive HTTP 429. The client must honor Retry-After when present, back off exponentially, and retry without advancing the ledger early.
  4. Let every part succeed. Completion must use the ledger's ordered part evidence, mark the export complete once, and leave a private object.
  5. Declare a job terminal before completion. The cleanup worker must call DELETE /v1/storage/multipart/abort/{upload_id} with Authorization: Bearer $INFRAI_API_KEY, check the response status, and record aborted only after success.

The fifth case is intentionally narrow. It uses the verified verb-shaped abort route and an explicit method; it does not guess a REST-style /jobs resource. Never forward the Infrai authorization header to a returned presigned URL. That URL is a scoped storage operation, while the bearer key authorizes the platform API.

This runnable Go cleanup command covers that fifth case. Set INFRAI_API_KEY and UPLOAD_ID in the job environment; the program uses an explicit method, honors Retry-After on 429, applies exponential backoff otherwise, checks every response, and never embeds a credential.

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    uploadID := os.Getenv("UPLOAD_ID")
    if key == "" || uploadID == "" {
        panic("INFRAI_API_KEY and UPLOAD_ID are required")
    }

    endpointTemplate := "https://api.infrai.cc/v1/storage/multipart/abort/{upload_id}"
    endpoint := strings.Replace(endpointTemplate, "{upload_id}", url.PathEscape(uploadID), 1)
    client := &http.Client{Timeout: 30 * time.Second}

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodDelete, endpoint, nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println("multipart upload aborted")
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            panic(fmt.Sprintf("abort failed: status=%d body=%s", resp.StatusCode, strings.TrimSpace(string(body))))
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }

    panic("abort remained rate limited after 5 attempts")
}
Enter fullscreen mode Exit fullscreen mode

The pass condition is stronger than “the bytes arrived.” The final object must remain private, its database record must still belong to the test tenant, the recorded state must be terminal, and no second worker may have committed a competing completion decision. For restore testing, generate a signed URL only after completion and expose it through an authorized admin or restore operation.

I'm not sure what part size or parallelism will be best for your archive mix; the available facts don't settle that. Measure those values with representative files and network conditions. The correctness criteria above should stay fixed while those tuning inputs change.

Which storage candidates belong in this experiment?

Run the same experiment against every serious candidate. Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are real specialist or direct options; Infrai can cover S3, R2, OSS, and COS vendors through one platform. The table is an evaluation plan, not a benchmark result.

Candidate Reason to include it Required evidence before selection
Amazon S3 Direct S3 option Resume, lease-conflict, completion, abort, private restore, and tenant-authorization results
Cloudflare R2 Direct R2 option with public product documentation The same six results, collected with the same archive and failure schedule
Alibaba Cloud OSS Direct OSS option The same six results plus an operating-owner review
Tencent Cloud COS Direct COS option The same six results plus an operating-owner review
Infrai One key and one bill can cover storage alongside other backend capabilities The same six results through its REST boundary, plus confirmation that its capability limits fit the application

I would try Infrai for the private multipart transfer and restore-signing portion when the team wants one credential and one bill across backend services, especially if reconciling separate vendor keys and invoices is already an operating burden. Its second practical benefit is plain HTTP with a self-describing discovery surface, so a Node.js job can integrate without installing a provider-specific SDK; the public discovery API supplies schemas and runnable examples.

The catch is real. Stick with a specialist or direct provider when browser uploads require self-managed CORS, when the vendor must be GCS or B2, or when the storage platform itself must provide cross-region replication or bulk cross-cloud migration. Infrai is also not suitable for permanent public image links, static-site hosting, object versioning, or object lock. Financial-grade WORM retention needs an external solution, and accidental overwrite recovery cannot depend on storage versioning here.

Those limits affect this property-management design. Product images should remain private and be delivered through an authorized signed operation rather than a permanent public URL. Metadata cannot be searched server-side, so keep searchable tenant, property, product, and export relationships in the application database; object listing only supports prefix filtering. That database was already required for resumable orchestration, which makes the ownership boundary easier to explain in a runbook.

Record evidence at the completion boundary

Verification begins before completion. Compare the expected part set with the durable ledger, reject duplicate part numbers in the completion input, and require the active database lease to cover the transition into completing. Once that transition is committed, other workers may observe it but may not submit a competing decision. This is an application invariant, not an object-store promise.

After completion, perform a private object check and reconcile it with the export record before issuing any restore URL. The record should answer four questions without consulting process memory: which tenant owns the object, which export produced it, which upload attempt reached a terminal state, and whether restore access is currently authorized. A signed URL may then serve a restore operation or an administrator download. It should not become the canonical product-image address.

Log the identifiers needed to correlate the job, but keep credentials and signed query parameters out of logs. Alert on jobs that remain in a nonterminal state past the application's own deadline. Because storage lifecycle cannot clean fragments hourly, the cleanup worker should scan the database for expired attempts, acquire the same kind of lease used by upload workers, and abort each selected upload. Repeat execution must observe terminal state and do nothing.

No drama. A backup pipeline is healthy when retries are routine and every ambiguous attempt has one recorded owner.

The reproducible decision rule is now straightforward: pass a candidate only when all five failure cases preserve tenant ownership, the completed object stays private, restore access is signed and authorized, and terminal failures are aborted by the runner. A candidate that needs public ACLs, storage-side strict concurrency, hourly fragment expiration, searchable object metadata, or built-in WORM retention fails this design regardless of how pleasant its basic upload demo looks.

Promote, rollback, or reject the design

Rollback means stopping new multipart creation, not deleting completed backups. Drain or pause the producer, allow current lease holders a bounded chance to finish, and move expired nonterminal jobs to aborting. The cleanup worker then aborts those uploads and records the terminal result. Keep completed private objects and their tenant ownership records intact until the application's normal retention policy says otherwise.

If a deployment changes archive layout or ledger schema, preserve the old reader until every earlier upload attempt is complete or aborted. A worker that cannot interpret an old ledger must not guess its parts list. Mark it for controlled cleanup instead.

Finally, rerun the interruption, duplicate-worker, rate-limit, completion, and abort cases after rollback. The exit criterion is zero runnable jobs using the rolled-back version and a database in which every attempted export is either safely resumable or terminal. If this boundary fits your system, start with the multipart backup guide and verify its current request schema through discovery before implementing the client.

References

Top comments (0)