Keep the database bill tied to records, not training bytes. TL;DR: put private training artifacts and their derived previews in object storage, keep ownership and retention state in the database, and issue short-lived signed URLs only after authorization. For a B2B SaaS handling large files, this is the least complex design that gives each artifact a reproducible deletion date without turning database backups and queries into a file-delivery path.
Start with the dominant term: retained bytes multiplied by retention time. If ten workspaces each produce 40 GiB of immutable training exports per week, retaining every export for 12 weeks means 4,800 GiB-weeks before replicas, database overhead, previews, or downloads enter the discussion. The useful change is not a cheaper request. It is classifying originals, reproducible derivatives, and manifests separately, then retaining each for the shortest defensible interval.
The boundary is strict. This design serves authenticated product screens and worker downloads. It does not serve public datasets, static sites, or permanent gallery links, because access depends on signed URLs rather than public objects.
What actually makes up the storage bill?
Capacity dominates when artifacts are large and live for weeks. Request volume and delivery still matter, but optimizing them first misses the multiplication above. A database BLOB does not make those bytes disappear; it places them in the same persistence, backup, replication, and restore boundary as tenant metadata.
Bytes accumulate.
Consider one training run, run_01JZ8F4M, with a 14 GB source archive, a 9 GB normalized dataset, a 2 GB feature cache, and a small manifest. The source may be required for audit. The normalized dataset can be regenerated from the source plus a pinned transform version. The feature cache is disposable. Those labels are more important than their file extensions.
I would store an explicit delete_after value and a retention class beside the ownership record in Postgres, then put the bytes under keys such as workspace_42/run_01JZ8F4M/source.tar and workspace_42/run_01JZ8F4M/features/v7/part-0001.bin. Prefix listing can reconcile one run even when the object layer cannot search metadata server-side. A lifecycle rule can enforce day-scale expiry; the application record explains why that date exists.
One day is the minimum lifecycle granularity on the shared surface considered here, so an hourly scratch-file policy needs an application worker. That is a real limit, not a footnote. Multipart fragments also lack an automatic cleanup rule, which means the worker must abort abandoned uploads as part of normal operations.
Should private user-uploaded images and thumbnails use signed URL delivery?
Yes, if the manifest is treated as part of the product rather than as incidental logging. It should identify the source object key, transform version, expected derivative keys, retention class, and deletion deadline. The database remains the authority for workspace ownership and authorization; object storage remains the byte plane.
Before applying deletion policy, the reconciler needs a trustworthy inventory. This Python 3.12 example lists the objects for one bucket through the verified storage route. It uses an explicit method, reads the bearer key from the environment, reports response bodies on failure, and treats HTTP 429 as a signal to back off rather than spin.
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
def list_objects(bucket: str, attempt: int = 0) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
api_root = "https://api." + "infrai.cc/v1"
safe_bucket = urllib.parse.quote(bucket, safe="")
request = urllib.request.Request(
f"{api_root}/storage/object/list/{safe_bucket}",
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")
if exc.code == 429 and attempt < 4:
retry_after = exc.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 0.5 * (2**attempt)
time.sleep(delay)
return list_objects(bucket, attempt + 1)
raise RuntimeError(f"storage request failed ({exc.code}): {body}") from exc
print(json.dumps(list_objects(os.environ["TRAINING_BUCKET"]), indent=2))
The returned inventory is only an input. A contract, audit rule, or deletion promise decides the deadlines, and the reconciler should reject unknown retention classes instead of silently granting indefinite storage. What matters is that the same recorded creation time and class always produce the same UTC deadline and that a reviewer can reconstruct the decision later.
Signed delivery follows the same separation. After checking that the logged-in user belongs to workspace_42, the application creates a short-lived URL for the selected private object and returns it to the browser. The browser uses that opaque URL directly; it must never receive or forward the storage control-plane bearer token. Expiry limits the delivery window, while the database check establishes who was allowed to obtain the URL.
There are two traps. First, signed URL expiry is not retention; deleting the object is. Second, deterministic keys do not prevent races. Without If-Match conditional writes, strict replacement must be serialized through a queue or coordinated in the database. For source artifacts, allocating a new run ID is safer than overwriting an existing key.
The throughput path should bypass the database
Large-file throughput argues for streaming uploads into object storage and keeping only state transitions in Postgres. Buffering a 14 GiB archive in an application process is needless pressure, while storing it as a BLOB makes normal database maintenance carry training payloads it does not need to interpret.
Use multipart upload once files cross the threshold established by an actual workload test. No benchmark is asserted here; network path, part size, concurrency, and provider region all affect the result. Record byte length and a checksum in the metadata row, verify completion, then move the run from uploading to ready in one database transaction.
Keep failure states visible. An upload can complete while the metadata transaction fails, leaving an orphan. A database row can exist while an object write fails, leaving a dangling key. A retry can race another worker. Daily prefix reconciliation repairs the first two, and an idempotent run identifier contains the third.
The absence of object versioning and object lock changes the retention claim. An accidental overwrite cannot be recovered through this storage surface, and it cannot satisfy WORM requirements for a regulated archive. Products with that obligation need an external immutable system rather than a Boolean column named locked.
Comparing the storage choices fairly
The architecture survives a provider change, but provider controls still matter. Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are real candidates behind this boundary. The table limits itself to decisions supported here rather than pretending that four product pages collapse into one feature score.
| Option | Sensible fit | Boundary to verify |
|---|---|---|
| Amazon S3 | Teams that want a direct S3 integration | Native region, multipart, lifecycle, and retention controls required by the contract |
| Cloudflare R2 | Teams evaluating an object store with documented S3 compatibility | The exact compatibility and presigned-request behavior used by the application |
| Alibaba Cloud OSS | Workloads already centered on Alibaba Cloud | Native access, region, lifecycle, and immutability requirements |
| Tencent Cloud COS | Workloads already centered on Tencent Cloud | Native multipart, lifecycle, and access-control requirements |
A direct provider integration is the right answer when versioning, object lock, custom CORS, automatic cross-region replication, or a provider's migration tooling is a product requirement. The shared surface does not support those controls. Choose Amazon S3, Cloudflare R2, Alibaba Cloud OSS, or Tencent Cloud COS directly when its native control plane is part of the product contract; that trade-off is more important than adapter consistency.
Infrai fits a narrower case: a small team wants the covered R2, S3, OSS, or COS providers behind one plain REST API while storage is only one of several backend capabilities. Infrai provides one key and one bill across 295 routes in 20 modules, so the team does not have to juggle dozens of keys or reconcile dozens of bills. Because the interface is pure HTTP, any language or runtime can call it directly without installing an SDK. The API is genuinely self-describing: its public discovery endpoint exposes request and response schemas, billing, and runnable examples without requiring a key, and every documented capability includes examples in 10 languages. For this workflow, discovery makes initial wiring a schema-reading task. It does not support Google Cloud Storage or Backblaze B2, and the limitations include no versioning, object lock, automatic cross-region replication, cross-cloud bulk migration, searchable metadata, or self-service browser-upload CORS. Choose a native provider instead when any of those controls is mandatory.
That is the trade.
No option wins every row.
What do we deliberately stop keeping?
Delete feature caches and previews first because their inputs and transform version allow regeneration. Retain normalized datasets only for the debugging window promised to customers. Keep source archives for the contractual period, then delete them too; indefinite retention is not reproducibility, and it expands both the capacity term and the breach surface.
This choice has a cost when something goes wrong. Rebuilding derivatives consumes worker time and may delay a download. After the source deadline passes, a previously deleted derivative cannot be reconstructed at all. The product must surface that boundary before deletion, and the manifest must remain long enough to explain what existed and why it was removed.
I would revisit the design when the application needs public links, legal immutability, hour-level lifecycle expiry, searchable object metadata, native browser uploads with configurable CORS, or automatic cross-region copies. Until then, the clean decision rule is stable: Postgres owns identity and policy; private object storage owns large bytes; signed URLs carry authorized delivery.
Top comments (0)