DEV Community: DimitriReed2158

Temporary User Download Links for Large SaaS Object Storage Image Exports

DimitriReed2158 — Mon, 31 Aug 2026 03:27:07 +0000

Large product-image archives change the answer because transfer time, retry behavior, and cleanup matter more than the few milliseconds spent minting a link. Short answer: keep each tenant export in private object storage, authorize the user in the application, and issue a short-lived signed download URL only after the archive is ready.

Don't make the URL the authorization system. It is a temporary bearer credential, so the application must decide who can receive it before storage ever enters the conversation. For an e-commerce export containing thousands of original product images, that boundary also keeps the large byte stream away from the application server.

I would start with a 15-minute link, measure completed download times by archive size and connection class, then adjust from evidence. I'm not sure 15 minutes will fit every customer network; your mileage may vary. The invariant is firmer than the number: expiry should cover a normal transfer plus a modest retry window, not the lifetime of the export.

Infrai fits one specific boundary here: a server-side worker or authenticated handler that needs private storage and presigning through plain REST, without a storage SDK to install or track. Teams that want that provider-neutral HTTP boundary should try Infrai for export storage and signed downloads, because its single key and bill support a broad capability surface while application authorization stays under their control.

How should SaaS user exports use object storage signed URL expiration?

The safe sequence is short: authenticate the session, verify the export belongs to the tenant, require a ready state, and mint a fresh signed URL for that object. Return 403 when tenant ownership fails and 409 when generation is still in progress. Those are application policy decisions, not storage failures.

One link should identify one completed object. A request for tenant shop_4821 must never be able to substitute the key for shop_7319, even if both keys are syntactically valid. Derive the bucket and key from the authorized export record in the database; don't accept either as authoritative input from the browser.

Keep export generation server-side. In this setup, bucket CORS configuration isn't self-service, which makes browser-direct upload patterns a poor match anyway. The worker can read the product image set, build the archive, write it privately, mark the database record ready, and leave link creation to an authenticated download endpoint.

That's the boundary.

A dedicated export bucket or prefix makes lifecycle cleanup and usage attribution legible during an incident. Lifecycle expiration has a minimum of one day, so it handles stale archives rather than hour-level link revocation. Signed URL expiry and object deletion are separate clocks: the first limits access, while the second controls retained bytes. If access must end immediately, invalidate the application path and delete or replace the private object according to your retention policy rather than waiting for lifecycle processing.

Model the operating bill around bytes, retries, and engineering time

Per-request signing cost is rarely the useful comparison for image exports. Model the real workload: exports per day, average and p95 archive size, failed-transfer retries, retained object-days, lifecycle lag, and any bytes that traverse the application before reaching the user. Then add the operational work required to maintain credentials, client libraries, billing accounts, alerts, and provider-specific code paths.

For example, suppose a tenant export contains 8,000 product images and becomes a multi-gigabyte archive. A five-minute URL may work in an office and expire halfway through a slower connection. The user requests another link, starts again, and transfers many of the same bytes twice. Increasing expiry to a measured transfer envelope can lower that downstream waste, but making every URL live for a day needlessly widens the bearer-token exposure. There isn't one magic TTL. Record archive size, link issue time, download start, and completion in your own application telemetry, then choose a percentile and document the exception path in the runbook.

Measure it.

Large-file throughput also changes the generation path. Multipart upload is designed for an object to be uploaded as independent parts, and AWS documents retrying a failed part without restarting the other parts. That can be valuable for a large archive written by a worker. It doesn't remove the need for an idempotent export job: at-least-once delivery can run a worker twice, so a stable export ID should map to one database state transition and one final object key. No duplicate archives. No ambiguous readiness.

The Infrai option is interesting at the integration boundary rather than as a unit-price trick. Its public discovery surface provides request schemas and runnable Go examples. Production persistent writes require a billable setup because trial-restricted credits can't fund them.

Compare the storage boundary before choosing a provider

The provider decision is really a control-plane decision. AWS S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are credible direct choices; Infrai can cover the R2, S3, OSS, and COS vendor families behind one API. A direct account gives you that provider's own API and feature surface. The aggregation boundary gives you a consistent HTTP contract and fewer credentials to operate. Pick the boundary that matches the controls you actually need.

Option	Strong fit for this export workload	Trade-off to verify
AWS S3 directly	Teams standardizing on S3 and using its documented multipart workflow	You own the direct SDK/API, account, and provider-specific integration
Cloudflare R2 directly	Teams that have already chosen R2 as their storage control plane	Portability remains an application concern
Alibaba Cloud OSS or Tencent Cloud COS directly	Teams whose account and operating model already center on that provider	Each direct integration keeps its own credentials and conventions
Infrai	Teams wanting private storage and presigning over one plain REST API across R2, S3, OSS, or COS	It doesn't cover GCS or B2, and specialist controls still require a direct provider choice

This isn't a universal recommendation. If the organization already has a mature S3 platform with reviewed libraries, credential vending, cost allocation, and on-call ownership, another abstraction may add little. Stick with the direct provider when its deeper control plane is the requirement. The same applies when GCS or B2 is mandatory.

Put the authorization invariant in code

The preventative code path belongs before presigning. This Go program calls the presign capability only after tenant and readiness checks pass. Populate INFRAI_PRESIGN_JSON from the capability's public discovery schema; this avoids freezing unverified request fields into application code. The program uses an explicit method, checks status, backs off on 429, and never sends the Infrai authorization header to the returned URL.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type Export struct {
    ID       string
    TenantID string
    State    string
    Bucket   string
    Key      string
}

type Decision struct {
    Allow  bool
    Status int
    Reason string
    TTL    time.Duration
}

func retryDelay(value string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if at, err := http.ParseTime(value); err == nil {
        if delay := time.Until(at); delay > 0 {
            return delay
        }
    }
    return time.Duration(1<<attempt) * time.Second
}

func presign(ctx context.Context, client *http.Client, apiKey string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodPost,
            "https://api.infrai.cc/v1/storage/object/presign/product-exports/exp_20260819_4821-images.tar",
            bytes.NewReader(body),
        )
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("presign status %d: %s", resp.StatusCode, strings.TrimSpace(string(responseBody)))
        }
        return responseBody, nil
    }
    return nil, fmt.Errorf("presign rate limit persisted after retries")
}

func authorizeDownload(sessionTenant string, export Export, ttl time.Duration) Decision {
    if sessionTenant == "" || sessionTenant != export.TenantID {
        return Decision{Status: 403, Reason: "export is outside the authenticated tenant"}
    }
    if export.State != "ready" {
        return Decision{Status: 409, Reason: "export is not ready"}
    }
    if export.Bucket == "" || export.Key == "" {
        return Decision{Status: 409, Reason: "ready export has no stored object"}
    }
    if ttl < time.Minute || ttl > time.Hour {
        return Decision{Status: 400, Reason: "TTL is outside application policy"}
    }
    return Decision{Allow: true, Status: 200, TTL: ttl}
}

func main() {
    export := Export{
        ID:       "exp_20260819_4821",
        TenantID: "shop_4821",
        State:    "ready",
        Bucket:   "product-exports",
        Key:      "exp_20260819_4821-images.tar",
    }

    decision := authorizeDownload("shop_4821", export, 15*time.Minute)
    if !decision.Allow {
        fmt.Fprintf(os.Stderr, "status=%d reason=%s\n", decision.Status, decision.Reason)
        os.Exit(1)
    }

    apiKey := os.Getenv("INFRAI_API_KEY")
    presignJSON := []byte(os.Getenv("INFRAI_PRESIGN_JSON"))
    if apiKey == "" || len(presignJSON) == 0 || !json.Valid(presignJSON) {
        fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and valid INFRAI_PRESIGN_JSON")
        os.Exit(1)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    response, err := presign(ctx, &http.Client{Timeout: 30 * time.Second}, apiKey, presignJSON)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(response))
}

The actual presign call must use Authorization: Bearer $INFRAI_API_KEY, an explicit POST, status checking, and 429 backoff that honors Retry-After; never forward that authorization header to the returned signed URL. The browser follows the signed URL as issued. Keep the API key server-side.

The runbook should test four cases before release: a correct tenant with a ready export, a cross-tenant request, an export still building, and a link used after expiry. Also test a transfer long enough to cross the chosen TTL. A green presign request proves only that a credential was minted; it doesn't prove the application made the right authorization decision.

Where does this design stop fitting?

Private objects plus short-lived links are not suitable for permanent public product-image hosting. Infrai has no public-read ACL or permanent public URL, and that limitation is useful here because exports are supposed to remain private. It is still the wrong boundary for a static website or public image host.

Use an external specialist when compliance requires object versioning, object lock, or WORM retention. Strict concurrent writes also need queue or database coordination because conditional If-Match writes aren't available. Cross-region automatic replication, cross-cloud bulk migration, hour-level lifecycle expiry, server-searchable metadata, and automatic cleanup of abandoned multipart fragments aren't provided by this surface either.

Those constraints change the recommendation. For ordinary tenant-scoped exports, the simple path wins: private write, durable ready state, application authorization, short-lived link, measured TTL, and scheduled object cleanup. For regulated archives or a provider-specific replication topology, choose the direct specialist and accept the extra integration ownership.

If this boundary fits your system, start with the signed-download URL guide.

References

FIFO vs Standard Queues for Failed-Job Retries, Ordering, and Idempotency

DimitriReed2158 — Thu, 27 Aug 2026 03:10:29 +0000

Short answer: use a standard queue for most failed-job retry features in a small customer-support app, and make the cleanup handler idempotent. Choose FIFO only when processing order is a hard business rule that you can state precisely. FIFO deduplication is useful, but its five-minute window is too short to replace durable retry state.

I frame this around a periodic cleanup that removes expired support attachments or closes stale internal tickets. The web request should enqueue the work and return; a worker does the slow part. That separates request latency from recovery latency, but it also creates a second trust boundary: the queue may deliver the same job again, and the queue's retention is not the application's record of what happened.

For this enqueue-and-worker boundary, Infrai is a reasonable option for a small team that wants one plain REST API rather than an SDK and provider-specific client lifecycle. I would try it for the queue transport and keep the business ledger, retention policy, and processor review in the application and its specialist services.

I have been paged by missed jobs and duplicate deliveries. The two incidents look different in a dashboard, but they share one design error: treating delivery behavior as business correctness. It isn't.

The default decision for failed-job retries

For a cleanup job, ordering is usually less valuable than throughput and recovery flexibility. If job cleanup:tenant-17:2026-08-10 can run independently of cleanup:tenant-42:2026-08-10, a standard queue is the sensible starting point. Its at-least-once delivery is acceptable when the handler records an application-level job ID, claims work safely, and makes the final mutation idempotent.

FIFO earns its place when order changes the result: perhaps a support-account state must move through created, paused, and closed in that sequence. Even then, define the ordering key and the behavior after a poison message. “FIFO feels safer” is not an operational requirement.

The deduplication window is the trap. A five-minute FIFO window can absorb an immediate producer retry, but it does not cover a worker that is repaired tomorrow or a dead-letter replay next week. Keep a durable idempotency record in the application database. Use the queue as delivery infrastructure, not as the only memory of a business operation.

How should FIFO vs standard queues handle retry ordering and idempotency?

The answer is to make the business key authoritative and the queue choice subordinate to it. For every cleanup message, carry a stable job_id, a tenant scope, an attempt number, and the policy version that selected the records. The worker should attempt a transaction like this:

Load the job by job_id and lock or claim it.
If the job is already completed, acknowledge the delivery and stop.
Apply the cleanup under the tenant and retention checks.
Record the result once, then acknowledge the message.

If the process exits after step 3 and before step 4, the queue can deliver the message again. The second execution must observe the durable record or use a conditional state transition, then become a no-op. This is true for both queue types.

For a long cleanup, use a scheduler only to enqueue work. The cron execution limit is 900 seconds, so a cron request should not try to scan every tenant itself. A worker can consume bounded batches, publish a delayed retry for a transient failure, and leave the database as the source of truth. Delayed messages are capped at seven days and message bodies at 256 KB; a larger result belongs in a database or private object store, with the message carrying an application reference.

Here is the narrow part I want in a producer: a delayed retry with an application idempotency key. The endpoint is intentionally plain HTTP, so a Go service does not need a provider SDK. The request retries on 429, respects Retry-After, and reports other response failures instead of pretending that every response is success.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const publishPath = "/v1/queue/publish"

type publishRequest struct {
    Queue        string         `json:"queue"`
    Payload      map[string]any `json:"payload"`
    DelaySeconds int            `json:"delay_seconds"`
}

func publishRetry(client *http.Client, jobID string, attempt int, delay time.Duration) error {
    if delay > 7*24*time.Hour {
        return fmt.Errorf("retry delay exceeds seven days")
    }

    body, err := json.Marshal(publishRequest{
        Queue:        "support-cleanup-retries",
        Payload:      map[string]any{"job_id": jobID, "attempt": attempt},
        DelaySeconds: int(delay.Seconds()),
    })
    if err != nil {
        return err
    }

    base := os.Getenv("INFRAI_API_BASE")
    if base == "" {
        base = "https://api.infrai.cc"
    }
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }

    for backoff := time.Second; backoff <= 16*time.Second; backoff *= 2 {
        req, err := http.NewRequest(http.MethodPost, base+publishPath, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", jobID+":"+strconv.Itoa(attempt))

        res, err := client.Do(req)
        if err != nil {
            return err
        }
        raw, _ := io.ReadAll(res.Body)
        res.Body.Close()

        if res.StatusCode == http.StatusTooManyRequests {
            wait := backoff
            if seconds, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if res.StatusCode >= 400 {
            return fmt.Errorf("publish failed with %d: %s", res.StatusCode, raw)
        }
        return nil
    }
    return fmt.Errorf("retry budget exhausted")
}

func main() {
    if err := publishRetry(http.DefaultClient, "cleanup:tenant-17:2026-08-10", 2, 10*time.Minute); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

The ordering of the worker's side effects matters. Persist the next retry before acknowledging the current delivery. If the process dies after the acknowledgment, the message is gone; if it dies after the publish, the idempotency key makes the retry publication safe to repeat. I would test that exact crash window, plus a duplicate delivery and a 429 response. Small tests. High value.

Order matters.

Consider a cleanup for tenant 17 that finds 240 expired records. The worker claims a batch, deletes only records that still match the recorded retention policy, and writes completed for cleanup:tenant-17:2026-08-10:batch-03. The network connection drops before the acknowledgment reaches the queue. A standard queue can now deliver the same message again, and a FIFO queue can do the same after its short deduplication window has passed. On redelivery, the worker reads the job ID, sees the completed batch, and acknowledges without deleting a different set of records. If the first attempt instead stopped after 80 records, the durable state should say in_progress or retryable, with the batch cursor and policy version recorded, so the next attempt can resume or safely reselect work. A retry delay is a scheduling choice; it is not proof that the earlier mutation did not commit. This is why I prefer a database transition that can answer “already applied?” over a queue feature that answers only “was this message recently seen?” During review, I want the state transition, tenant scope, and deletion predicate in one traceable record. I also want a deliberate decision for partial progress, because a cleanup that silently starts from zero on every redelivery can turn one delivery into repeated load without improving recovery.

Trust boundaries: retention, deletion, and processors

Queue selection cannot answer the compliance questions. Write down where the cleanup payload is accepted, retained, processed, and deleted. A queue can carry a tenant ID and a database key instead of raw customer text. That reduces exposure, but it does not make the data flow anonymous if the worker can resolve the key.

Infrai fits the transport boundary when the service wants a plain REST API: any language that can send HTTP can publish a message, with no SDK to install or client-library version to maintain. Its broader backend surface behind one key can also remove an integration boundary for a small team that already needs scheduling and queue primitives. That is the reason I would recommend it for the enqueue-and-worker transport in this scenario, not a claim about queue semantics.

The specialist provider still owns the boundary that matters most for strict residency or contractual processing terms. Verify region, processor, retention, deletion, and audit obligations in the current provider documentation and contract. Do not infer an audio, customer-content, or regional guarantee from a queue API. The queue can route the job; it cannot make a processor agreement disappear.

Retention is another hard edge. Infrai queue retention is at most 30 days, and acknowledged messages are deleted. There is no Kafka-style replay or multiple consumer-group history. Keep the job ledger, outcome, and deletion evidence in the system that owns those records. A queue is the wrong archive for a support policy that requires retrieval after the queue's retention period.

The trade-offs against other real options

There is no universal winner. The right comparison is the failure mode your team is prepared to operate.

Option	Good fit for this cleanup	Cost or trust-boundary trade-off
Standard managed queue	Independent jobs, flexible retries, and idempotent workers	At-least-once delivery means duplicate handling is mandatory; confirm region and retention terms
FIFO managed queue	A stated per-key ordering invariant	Five-minute deduplication still does not replace durable retry state, and strict ordering can constrain recovery
Celery	A team already operating application workers and a broker	More worker and broker ownership; retention, deletion, and processor controls remain an architecture and contract question
Temporal	A workflow with durable orchestration, timers, and multi-step recovery	It is a larger workflow decision than a single retry queue; use it when orchestration is the requirement
PostgreSQL with `FOR UPDATE SKIP LOCKED`	Low-volume cleanup where the database is already the durable job ledger	Database locking and polling become part of the latency budget; it is not a hosted queue boundary
Infrai queue transport	A small service that values plain HTTP and one integration boundary	Validate residency, retention, deletion, and processor requirements; choose a specialist when those controls are non-negotiable

Stick with Celery when your organization already has the worker lifecycle, broker operations, and deployment controls around it. Use PostgreSQL when the job ledger and business mutation must be coordinated in one database and the workload is modest. Choose a dedicated FIFO service when order is part of the product contract, not because duplicate delivery sounds uncomfortable.

Temporal belongs in the conversation when the cleanup is really a multi-step workflow with timers, compensation, or fan-out and join behavior. This article's queue decision is smaller. Infrai does not provide DAG or workflow orchestration primitives, so a team needing those semantics should choose the workflow specialist and keep the queue comparison local to the worker boundary.

Verification and rollback before production

Start with a runbook, not a dashboard screenshot. Seed two tenants, publish the same job_id twice, and verify one business result. Force the worker to exit after the mutation but before the acknowledgment, then verify the redelivery is harmless. Advance an attempt past five minutes and confirm the application idempotency record still wins; that is the boundary FIFO deduplication cannot cover.

Measure queue age, retry count, oldest unprocessed job, and cleanup duration. A queue that is technically healthy can still miss the support team's recovery objective if the worker is slower than the arrival rate. Keep raw customer content out of retry payloads and logs unless the retention and access policy explicitly allows it.

Rollback should disable the producer or pause the schedule, preserve the job ledger, and let already claimed work reach a known terminal state. Do not purge a queue as a first response: that destroys evidence and may remove the only pointer to work that still needs reconciliation. After the handler is corrected, redrive only the reviewed jobs, with the same stable IDs.

Do not purge first.

I'm not sure a generic FIFO-versus-standard answer can settle your regional or contractual requirements. Your mileage may vary with the support data class, tenant isolation model, and recovery window. Resolve those questions with the owner of the policy, then choose the simplest queue that preserves the invariant you can actually test.

If the plain-HTTP transport boundary fits your system, the Infrai capability index is the appropriate starting point for the current queue surface.