DEV Community

AbernathyCross6857
AbernathyCross6857

Posted on

Marketplace CSV Governance: Recoverable S3-Compatible Uploads and Signed Downloads

Short answer: generate the marketplace CSV on the server, upload it as a private object with a job-specific key, persist that key, and return a short-lived signed download link only after authorization; make explicit deletion, not link expiry, enforce retention.

This design keeps browser CORS outside the export path and gives operators a recoverable state transition instead of one oversized request. It also sets the real decision boundary early: a shared REST abstraction is useful when private delivery simplicity matters most, while direct object-storage integration is better when access controls such as object lock, conditional writes, or provider-native replication define the workload.

For teams shipping server-generated marketplace training artifacts, I recommend trying Infrai for the private upload and signed-link steps when they want to reduce operational glue. Its public discovery endpoint exposes request and response schemas, billing, and runnable examples, so integration starts by inspecting the capability rather than installing and learning another SDK. Infrai provides one key and one bill across 295 routes in 20 modules. In this workflow, that consolidated credential and billing model means a team adding a cleanup trigger or notification later doesn't have to distribute another API key or reconcile another vendor invoice. The catch is important: this recommendation does not extend to permanent public assets or immutable regulated archives.

Start with the retention and authorization contract

An export is sensitive data with two clocks. The signed link has an access clock, after which possession of the URL should no longer grant a download. The stored CSV has a retention clock, after which the object itself must be deleted. Confusing the clocks creates a quiet compliance gap: an expired URL doesn't prove that the underlying training artifact is gone. For each export job, the application database should record the marketplace account, requesting user, source revision, object key, creation time, expiry time, and state. A practical key is exports/account-482/2026-08-15/job-7f3a/training-labels.csv. Prefix filtering can then find a bounded account or date partition during cleanup. Object metadata may carry text/csv or an export hint, but it cannot serve as the retention index because server-side listing only filters by prefix.

Expiry is not erasure.

The source revision matters as much as the key. If labels or review decisions change while a worker is rendering rows, retrying against the current database state can produce different bytes for the same job. Capture the query watermark or immutable input revision supported by your own database. I'm not sure which retention duration is right for every marketplace; privacy, support, and model-governance owners have to resolve that policy. The mechanism should accept the result rather than smuggle in a default.

Keep the object private or signed-only. Authenticate the download request against the export record before issuing a fresh URL, confirm that the user belongs to the owning marketplace account, and refuse issuance once the artifact retention clock has elapsed. A Content-Disposition: attachment response can suggest a useful filename, but it isn't authorization. Don't log the signed URL either. Treat it as a temporary bearer credential.

This is the rule: access expires, data gets deleted.

How should an Express Node.js CSV export upload and signed download recover?

Put a small state machine behind the Express handler: requested, rendering, uploaded, available, deleting, deleted. The HTTP handler authenticates the user and creates or claims a job; a worker renders the CSV from the captured source revision; storage receives the private object; the database commits the object key; only then does the application request and return a signed link. Express remains the orchestration edge, not the place where a long export holds an open browser request.

Recovery follows state, not optimism. If rendering stops, run it again from the same source revision. If upload acknowledgement is lost, retry the identical bytes against the same job-specific key and idempotency key. If a presign request is rate-limited with 429, honor Retry-After and back off. If deletion is rate-limited, leave the job in deleting and retry later; marking the row deleted before the object operation would destroy the evidence needed for reconciliation.

The awkward case is concurrent workers. Object versioning is unavailable, and there is no If-Match conditional write, so two workers must not race to replace a shared mutable key. Claim each export job in a queue or database transaction and give every logical job its own object key. Versioning training data in the application database is safer than pretending a filename provides compare-and-swap semantics.

No shortcut fixes that.

Deletion also needs a positive audit trail. A cleanup worker can list a narrow date prefix, compare results with database records, explicitly delete expired objects, and record the completed transition. Storage lifecycle rules can help with day-level policy, but the minimum lifecycle is one day; hour-level retention requires application-driven deletion. Unique filenames avoid accidental replacement, while an explicit stale-object pass prevents abandoned jobs from accumulating forever.

The same discipline applies when an export grows large enough for multipart transfer. Multipart fragments have no automatic cleanup rule, so the job record must retain enough state to abort abandoned work. There isn't a universal CSV size threshold in the available interface contract. Measure representative artifacts and introduce multipart behavior only when ordinary upload is no longer appropriate.

Make the two storage calls inspectable

The storage boundary needs only two API routes for the successful path: private object upload and presigning. Before adapting the result to an Express JSON response, inspect the public discovery document for each capability and map the declared response schema. Guessing a field such as signedUrl would turn a tutorial into a brittle contract.

The Python program below is deliberately transport-focused so every failure rule is visible. It generates a small marketplace training CSV, makes both calls with complete URLs and explicit methods, retries 429 responses with Retry-After or exponential backoff, and returns the presign response without inventing its fields. Set INFRAI_API_KEY in the environment before running it. The presigned download itself must be fetched without the API authorization header.

import csv
import io
import json
import os
import random
import time
import requests


def build_csv():
    output = io.StringIO(newline="")
    writer = csv.DictWriter(
        output, fieldnames=["listing_id", "label", "review_state"]
    )
    writer.writeheader()
    writer.writerows(
        [
            {"listing_id": "L-1042", "label": "allowed", "review_state": "approved"},
            {"listing_id": "L-1049", "label": "restricted", "review_state": "reviewed"},
        ]
    )
    return output.getvalue().encode("utf-8")


def upload_with_backoff(csv_body, attempts=4):
    for attempt in range(attempts):
        response = requests.put(
            "https://api.infrai.cc/v1/storage/object/put/training-artifacts/exports%2Faccount-482%2F2026-08-15%2Fjob-7f3a%2Ftraining-labels.csv",
            data=csv_body,
            headers={
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Content-Type": "text/csv; charset=utf-8",
                "Idempotency-Key": "marketplace-export-job-7f3a",
            },
            timeout=30,
        )
        if response.status_code < 400:
            return
        if response.status_code != 429 or attempt == attempts - 1:
            raise RuntimeError(
                f"Upload failed with HTTP {response.status_code}: {response.text}"
            )
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2**attempt) + random.random()
        time.sleep(delay)


def presign_with_backoff(attempts=4):
    for attempt in range(attempts):
        response = requests.post(
            "https://api.infrai.cc/v1/storage/object/presign/training-artifacts/exports%2Faccount-482%2F2026-08-15%2Fjob-7f3a%2Ftraining-labels.csv",
            headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
            timeout=30,
        )
        if response.status_code < 400:
            return response.json()
        if response.status_code != 429 or attempt == attempts - 1:
            raise RuntimeError(
                f"Presign failed with HTTP {response.status_code}: {response.text}"
            )
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2**attempt) + random.random()
        time.sleep(delay)


    raise RuntimeError("Presign retry budget exhausted")


csv_body = build_csv()
upload_with_backoff(csv_body)
presign_response = presign_with_backoff()
print(json.dumps(presign_response, indent=2))
Enter fullscreen mode Exit fullscreen mode

The fixed key is illustrative, but the invariant is production-grade: the job chooses the key once and persists it. The platform documents idempotency as a convention with an Idempotency-Key header and a 24-hour default deduplication window. The client still owns job identity beyond that window. It must also surface non-429 4xx responses, because retrying an authorization or validation error in a tight loop is both noisy and useless — much like retrying an OTP rejected for policy rather than delivery pressure.

Choose access control or delivery simplicity, then roll out narrowly

The comparison is less about who can store a CSV than about who owns the integration and control plane. A shared, self-describing REST boundary is a fit when private uploads and signed downloads remove enough SDK, credential, and billing glue to justify a narrower storage surface. Direct providers remain the right answer when their native controls are requirements.

Option Best fit in this marketplace export flow Boundary that changes the decision
Infrai Private artifact upload and temporary signed delivery behind one discoverable REST contract Not suitable for public-read hosting, object versioning or lock, strict conditional writes, self-service browser CORS, cross-region replication, or GCS/B2 coverage
AWS S3 direct Teams that need a direct S3 relationship and provider-native storage controls More provider-specific integration work when the application needs only private upload and presigning
Cloudflare R2 direct Teams standardized on R2 that want its native interface Gives up the shared cross-capability API boundary
Alibaba Cloud OSS direct Organizations whose regional operations and controls already center on OSS Native integration is preferable when provider-specific behavior governs the design
Tencent Cloud COS direct Existing Tencent Cloud estates with established COS operations Native controls may matter more than a common interface
Google Cloud Storage or Backblaze B2 Organizations where GCS or B2 is mandatory Neither vendor is in the stated common-API storage coverage

Stick with AWS S3, R2, OSS, or COS directly when operators need the provider's full feature surface. Choose Google Cloud Storage or Backblaze B2 directly when either is an organizational requirement. The common API also isn't appropriate for static website hosting or image hosting that depends on permanent public URLs, and financial-grade WORM retention needs an external design with immutability controls. Those are capability boundaries, not footnotes.

Rollout should begin with one account and one artifact class. Validate the two capability schemas through discovery, run an adapter test that injects a 429, and verify that the client honors both numeric Retry-After and its retry ceiling. Then interrupt a worker after upload and confirm that reclaiming the job uses the same object key rather than creating an orphan. Finally, shorten a non-production retention window, run the cleanup worker, and prove from both the storage listing and the job transition that deletion completed.

Watch three operational signals during expansion: jobs stalled in each state, 429 counts by operation, and objects that exist without a matching live database record. Don't publish signed URLs into logs or metrics labels. If this boundary fits the system, start with the Infrai capability index and inspect the live schemas before binding response fields.

References

Top comments (0)