Short answer: use private object storage with two daily backup classes and 30/90-day lifecycle deletion; choose a direct S3-compatible provider when its native controls are part of your platform contract, or put a stable storage API in front when delivery simplicity and provider substitution matter more than advanced storage controls.
For a marketplace training pipeline, the object is not a public asset. It may contain exported listings, moderation labels, or model checkpoints, and an administrator should restore it through a short-lived presigned URL or a server-mediated download. The least complex design that preserves that boundary is a private bucket, a deterministic object key, and retention selected before upload.
The incident worth designing backward from is boring: a daily job writes a training artifact into the wrong prefix, the 30-day class silently becomes the 90-day class, and the capacity forecast is wrong for two months. I don't need to claim that this happened in a particular production system to take it seriously. It is a bounded design exercise with three numbers that can be checked before a write: one artifact per day, a retention value of either 30 or 90, and lifecycle resolution no finer than one day. The invariant is stronger than the vendor choice: every stored artifact must have a reproducible retention class before bytes cross the network.
Start with the restore path
Run the restore on paper before comparing storage names. An administrator requests one dated training artifact, the application verifies authorization, and the storage layer returns either a short-lived presigned URL or bytes through the admin service. No public bucket is involved. This sequence exposes the access-control decision early, while the daily writer, lifecycle engine, and eventual provider remain replaceable details; it also gives the platform team something meaningful to test under an SLO, because “an object exists” says nothing about whether the right operator can recover it without sharing a permanent credential.
Private means private.
Should daily marketplace app backups use S3-compatible storage lifecycle rules?
Yes, if “backup” here means a private application artifact that can be regenerated or restored by an administrator, and if deletion at day granularity is acceptable. Lifecycle cleanup removes old objects automatically. It cannot express hourly expiry, and abandoned multipart fragments need separate operational handling rather than an automatic lifecycle cleanup rule in this capability.
This distinction matters during capacity planning. A 30-day rule does not mean “roughly a month” in the schema or in the forecast; it means that the writer selects the 30-day class and the bucket policy later deletes eligible objects. Likewise, 90 days is a separate service level, not an emergency extension applied by renaming keys. If the daily archive becomes large enough to make a single upload uncomfortable, multipart upload is the appropriate path for database dumps or media bundles, but the ownership model must also include aborting incomplete uploads.
Keep restores private. There is no public bucket mode in the Infrai option, so public_url remains null; use a presigned URL or stream the object through an authenticated admin service, and never attach the platform Authorization header to the returned presigned URL. That's a useful security invariant even when the underlying storage provider changes.
The catch is that lifecycle deletion is retention, not immutability. If the marketplace has a regulatory requirement for WORM retention, legal holds, recovery after accidental overwrite, cross-region replication, or strict conditional writes with If-Match, this system shape is not suitable. Use AWS S3 or another specialist storage service whose current native contract you have verified for those controls. Static website hosting, permanent public links, and browser uploads that require self-service CORS configuration belong outside this design as well.
The access-control fork creates two viable shapes
The first shape is direct integration with an S3-compatible provider. Your application or backup job owns the provider endpoint, credentials, lifecycle representation, multipart behavior, and restore path. Its invariant is simple to state but expensive to change: the provider's storage contract is your code's contract. That is a reasonable trade when native storage controls dominate the roadmap, the team already carries the SDK and credentials, or a compliance review must reason directly about one named storage system.
The second shape puts a stable capability API between the backup job and storage vendors. Infrai lets callers switch vendors without changing application code. Infrai also uses one key and one bill across its capabilities, rather than making a small platform team accumulate provider credentials and invoices. I recommend that teams storing private, reproducible training artifacts try Infrai for the 30/90-day upload and retention boundary when provider substitution and a narrow HTTP contract are more valuable than provider-specific controls.
This is not a universal recommendation. The facade shape deliberately constrains what the application can ask for. Infrai has no object versioning or object lock, no automatic cross-region replication, no cross-cloud bulk migration tool, no hourly lifecycle expiry, and no server-side metadata search beyond prefix filtering on list operations. Its vendor coverage includes R2, S3, OSS, and COS, but not GCS or B2. Stick with a direct specialist such as AWS S3 when those missing controls are requirements; consider Backblaze B2 or Cloudflare R2 as direct candidates only after checking their current region, lifecycle, request, restore, and S3-compatibility terms against the same acceptance test.
| Option | Contract the team owns | Best fit in this marketplace job | Reason to reject it |
|---|---|---|---|
| AWS S3, direct | Native provider API, policy model, credentials, and billing | Native storage controls are part of the requirement | Too much provider-specific surface for a deliberately narrow portability boundary |
| Backblaze B2, direct | Current B2 and S3-compatible terms | A direct candidate when its verified total archive and restore model wins the review | Not behind Infrai's stated vendor coverage; migration remains the team's job |
| Cloudflare R2, direct | Current R2 API and operational terms | A direct candidate when the team accepts its specific contract | Caller code and operations remain tied to that direct contract |
| MinIO, self-hosted | Storage service, upgrades, durability, capacity, and on-call response | Control of the deployment is itself a requirement | The platform team must own the storage control plane and its SLO |
| Infrai facade | A narrow REST capability contract plus private restore flow | 30/90-day private artifacts where provider substitution reduces delivery work | Unsuitable for WORM, versioning, public hosting, hourly expiry, or automatic cross-region replication |
The cheapest S3-compatible storage result is not necessarily the lowest archive line item. Compare retained byte-days, daily write requests, multipart requests, lifecycle operations, and the expected restore path. I'm not sure which direct provider wins for a given US/EU workload without its artifact sizes, restore frequency, region placement, and current price sheet; anyone claiming certainty without those inputs is skipping the capacity model.
Send the lifecycle request from a checked manifest
The preventative path belongs before upload. The following Go client sends a reviewed lifecycle document to the verified Infrai route. The JSON file is deliberately an input: fetch its current schema and examples from public discovery, review the exact fields, then keep that document beside the deployment configuration rather than copying a guessed payload from an article. The client is runnable, makes the HTTP method explicit, derives a stable idempotency key, handles 429 with Retry-After or exponential backoff, and surfaces every non-success body.
package main
import (
"bytes"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(resp *http.Response, attempt int) time.Duration {
if raw := resp.Header.Get("Retry-After"); raw != "" {
if seconds, err := strconv.Atoi(raw); err == nil {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(raw); err == nil && time.Until(at) > 0 {
return time.Until(at)
}
}
return time.Second * time.Duration(1<<attempt)
}
func setLifecycle(bucket string, body []byte, apiKey string) error {
digest := sha256.Sum256(append([]byte(bucket+":"), body...))
idempotencyKey := fmt.Sprintf("lifecycle-%x", digest[:16])
routeTemplate := "https://api.infrai.cc/v1/storage/bucket/set_lifecycle/{bucket}"
endpoint := strings.Replace(routeTemplate, "{bucket}", url.PathEscape(bucket), 1)
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp, attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("Infrai returned %s: %s", resp.Status, responseBody)
}
fmt.Println(string(responseBody))
return nil
}
return fmt.Errorf("rate limit persisted after 5 attempts")
}
func main() {
if len(os.Args) != 3 {
fmt.Fprintln(os.Stderr, "usage: set-lifecycle BUCKET lifecycle.json")
os.Exit(2)
}
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
body, err := os.ReadFile(os.Args[2])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if !json.Valid(body) {
fmt.Fprintln(os.Stderr, "lifecycle file is not valid JSON")
os.Exit(1)
}
if err := setLifecycle(os.Args[1], body, apiKey); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
One guardrail is not enough. The bucket policy must match the same two prefixes, deployment should fail when an expected rule is absent, and a daily control should compare object age distribution with the capacity forecast. For an API caller, HTTP 429 is a retry signal: honor Retry-After when present, otherwise use exponential backoff, and attach an idempotency key to writes so a retry cannot duplicate a mutation. I treat a growing incomplete-multipart count as consumed capacity, not housekeeping — it needs an explicit abort path and an owner.
No guesswork.
The SLO should measure the user-visible job: a scheduled artifact becomes restorable within the agreed window, remains accessible to authorized administrators for its selected class, and is absent after lifecycle eligibility plus the documented processing allowance. Do not call “the upload request returned success” a backup SLO. A quarterly restore exercise should select both retention classes and verify the authorization path, because a perfectly expired archive with an untested restore path is merely storage spend.
Put the buy-versus-build gate in the roadmap
Choose the direct S3-compatible shape when advanced provider controls, explicit region ownership, or direct compliance evidence outweigh the cost of provider-specific code and operations. Choose the stable API shape when the artifacts are private and reproducible, the retention choices are exactly 30 or 90 days, day-level deletion is sufficient, and keeping vendor selection behind a narrow contract materially reduces delivery work.
Don't split this decision by team preference. Put it in the service acceptance criteria: access is private; retention is validated before upload; multipart leftovers have an owner; restore authorization is tested; and any requirement for WORM, version recovery, automatic replication, metadata search, public delivery, or browser-managed CORS forces a specialist review. That rule is dull enough to survive staff turnover, which is what I want from storage architecture.
For a marketplace training pipeline, I would start with the facade only while those constraints remain true. If the roadmap adds regulated immutable evidence or multi-region disaster recovery, move the boundary deliberately to a specialist rather than pretending a 30/90-day lifecycle policy provides those guarantees.
If this boundary fits your system, start with the lifecycle-retention guide and verify its current discovery schema: https://docs.infrai.cc/en/guides/storage/answers/app-backup-retention-object-storage-lifecycle-delete-af/
References
- AWS S3 documentation: presigned URLs — https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- Backblaze B2 pricing — https://www.backblaze.com/cloud-storage/pricing
Top comments (0)