The short answer is to treat thumbnail metadata as part of the upload contract, not as an afterthought attached to the image later. Set the metadata on the thumbnail upload, then validate it with a metadata update or a head-style read before the object is exposed to downstream jobs. For a multi-tenant shop, the bucket and key namespace still matter more than the label itself: a correctly written tenant_id does not repair a key that can collide across tenants.
The practical sequence is four checks: derive a tenant-scoped key, send metadata in the object write, validate the stored object, and test the expiry policy against the deletion deadline. The last check is easy to skip because metadata looks tiny. Retention is not tiny when every generated size remains stored.
What is the storage bill actually retaining?
For a thumbnail pipeline, the retained unit is an object plus its bytes, while metadata is the index-like context that tells a worker which tenant, source asset, dimensions, and expiry policy it is looking at. If one source image produces four thumbnail sizes, the pipeline retains four object records and four sets of metadata. That is the useful quantity to count before changing an API call; counting source uploads alone hides the storage that the resize job creates.
The dominant retention term is therefore the number of thumbnail objects multiplied by their stored bytes and their time in storage. Metadata persistence does not remove that term. It prevents the same objects from becoming ungovernable blobs.
Four outputs. Four contracts.
For an e-commerce tenant, I would keep the key shaped around an immutable tenant boundary, for example tenants/<tenant-id>/products/<asset-id>/thumb-320.webp. The actual convention can differ, but the boundary must be present before an upload is retried. A shared products/<asset-id> prefix is a collision waiting for the right two tenants to choose the same ID.
Then make the deletion deadline explicit. A one-day lifecycle is the shortest supported lifecycle here, so it cannot represent an hourly deadline. If the business rule says “delete by tomorrow,” a one-day policy may fit after checking the provider's lifecycle semantics; if it says “delete in six hours,” an application job or another storage system has to enforce that deadline. The deliberate trade-off is clear: retain less and a late support investigation may lose the thumbnail; retain longer and tenant data remains present longer than the workflow needs.
Why does metadata disappear after a successful upload?
Usually, “upload succeeded” proves only that the object body was accepted. It doesn't prove that the metadata parameters were attached to that request, nor that a later resize step preserved them when it wrote a new object.
There are two writes to audit:
- The thumbnail object write at
/v1/storage/object/put/{bucket}/{key}. Its request must include the metadata settings alongside the body and the tenant-scoped key. - The repair or validation write at
/v1/storage/object/set_metadata/{bucket}/{key}. Use it to confirm the API format and update metadata when a thumbnail already exists.
Keep the check close to the write. Read back the object metadata, compare the tenant identifier and retention fields with the values computed by the resize job, and fail the job if the comparison is wrong. Do not infer success from an HTTP 2xx alone if the response body is not being inspected; a malformed metadata parameter is a data-quality error, not a cosmetic warning.
The common trap is a technically valid retry. The first request stores the body. A second request stores a resized body but omits metadata. Nothing looks broken in an image viewer, while indexing and deletion logic quietly lose their inputs. I initially treat the body and metadata as one record; in a pipeline with separate resize stages, they are two independently replaceable pieces unless the code makes the contract explicit.
Here is the smallest useful shape for a metadata update. The key and metadata values come from the tenant-scoped job record, while the bearer key comes from the environment. Keep the request ID or idempotency key stable for a retry in the production client, and handle a 429 with exponential backoff rather than immediately repeating the write.
import os
import requests
def set_thumbnail_metadata(bucket, key, tenant_id, expires_at):
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
url = f"{base_url}/v1/storage/object/set_metadata/{bucket}/{key}"
payload = {
"metadata": {
"tenant_id": tenant_id,
"expires_at": expires_at,
}
}
response = requests.post(
url,
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
json=payload,
timeout=15,
)
if response.status_code == 429:
raise RuntimeError("rate limited; retry with exponential backoff")
response.raise_for_status()
return response.json()
This is a validation and repair call, not a substitute for putting metadata on the original thumbnail write. The unified REST shape is useful here because Infrai gives backend services one key and one bill, and its public discovery surface describes capabilities without requiring a key. That convenience does not add version recovery, metadata search, or conditional writes; the earlier boundary still applies.
How should tenant isolation shape the fix?
Tenant isolation is a three-layer decision: namespace, authorization, and query behavior. Metadata helps with the third layer, but it is not an authorization boundary. A list operation that filters only by prefix still needs a server-side prefix derived from the authenticated tenant, and a worker must refuse keys outside that prefix.
Object storage options have different answers here:
| Option | Tenant-isolation strengths | Important boundary |
|---|---|---|
| Amazon S3 | IAM policies, bucket policies, object tags/metadata, versioning, and Object Lock support a layered design | The policy and key-prefix model still needs careful testing; metadata is not a substitute for authorization |
| Cloudflare R2 | S3-compatible object operations and bucket-level separation fit teams already using the S3 API shape | It is not a drop-in replacement for every S3 feature or governance workflow; check the feature matrix before moving retention controls |
| Azure Blob Storage | Containers, Azure RBAC, Blob Index Tags, versioning, and immutability policies cover richer governance cases | The control plane and authorization model are Azure-specific, so a migration adds operational vocabulary |
| A unified REST storage surface | One key and one bill across backend services can reduce credential and billing-dashboard sprawl; the storage routes keep the upload and metadata update calls in one interface | No public/public-read ACL, no versioning or object lock, no If-Match conditional write, and metadata is not server-queryable; strict immutability and concurrent writers need external coordination |
The last row can be a sensible fit for a backend that values one interface across services and can enforce tenant boundaries in its own application layer. It is a poor fit for a public image host, an immutable financial archive, or a design that expects metadata queries from storage itself. The deciding question is not which product has the shortest upload call. It is which system owns the isolation proof.
What should the retention test prove?
I would test the thumbnail path with two tenants, two asset IDs, and at least two resize outputs. That small matrix catches more than a single happy-path upload:
- Tenant A cannot read or overwrite Tenant B's key, even when the asset IDs match.
- Each output has metadata that identifies its own tenant, source asset, dimensions, and deletion deadline.
- A retry of the resize stage does not leave a new object without metadata.
- The validation path can update metadata through the object metadata setting endpoint and reports a malformed request instead of silently accepting it.
- The lifecycle rule matches the deadline's granularity. A one-day minimum is not an hourly scheduler.
There is another operational edge. Without If-Match, two writers cannot express strict conditional replacement through the storage call alone. Put the resize job behind a queue or coordinate the version decision in a database. Without object versioning or object lock, an accidental overwrite is not recoverable from the storage layer; retain the source or a separate audit copy when the business requirement demands that safety.
Metadata also cannot serve as a search index in this setup: listing supports prefix filtering, not server-side metadata queries. If an operator needs “all thumbnails for tenant A expiring today,” maintain that index in a database or derive it from a tenant-scoped key and application records. This is an architectural boundary, not a reason to overload every upload with more labels.
The decision rule
Fix the upload contract first. Generate the tenant-scoped key, include metadata in the thumbnail write, validate the stored values, and use the metadata-setting operation for an explicit repair path. Then measure retention by generated object count, stored bytes, and deadline compliance rather than by source-image count.
Choose a storage service with richer native governance when you need public delivery, version recovery, WORM retention, conditional writes, cross-region replication, or metadata indexing. Choose the unified REST approach when one credential and one billing surface across backend services are valuable and your application is prepared to own isolation, concurrency, and indexing. That is a real boundary, not a footnote.
For deletion-sensitive documents and thumbnails, the cleanest system is the one that can explain three things for every object: which tenant owns it, why it still exists, and what event will delete it. Metadata makes that explanation inspectable. The key design and retention controller make it enforceable.
Top comments (0)