Thumbnail Metadata Not Persisting in Object Storage After Upload: A Reliable Fix

#objectstorage #metadata #thumbnails

Thumbnail metadata that disappears after an upload is usually an upload-contract problem, not an indexing problem. The resized bytes and the metadata are separate parts of the request; if the metadata is omitted, placed in the wrong parameter, or sent to a later call that fails silently, the object can be perfectly readable while its descriptive fields are gone.

For a developer-tool service serving generated reports to authenticated customers, I would make the thumbnail write explicit, verify the stored object, and treat metadata as application state rather than decoration. The practical sequence is: build the thumbnail, include its metadata in the object upload request, inspect the result, then use the object metadata setting operation to correct or validate it. Do not make public access the fallback for a delivery problem. It weakens the access boundary and does not repair persistence.

What is the bill actually made of?

The storage bill is rarely explained by the metadata itself. The dominant term is normally the number and size of retained thumbnail objects, plus request volume and any retention policy. Metadata changes the usefulness of those bytes: a report_id, rendition, content type, or generation marker can determine whether a worker can find the right thumbnail without opening the object. It does not turn object storage into a query database.

Bytes are not metadata.

That distinction matters for retention. Keep the metadata that supports authentication, routing, and cleanup. Stop keeping transient resize inputs and duplicate thumbnails once the report lifecycle no longer needs them. The trade-off is uncomfortable but clear: deleting those objects reduces retained data, while an accidental overwrite or an overly aggressive lifecycle rule can remove the only convenient reconstruction input. With no object versioning or object lock, an overwrite is not a recoverable historical version; keep authoritative report state in a database and retain only what the product actually promises to restore.

For this workload, I would record a small, stable metadata set such as the rendition name and source report identifier, and put searchable fields in the database. The storage list operation is prefix-oriented, not a server-side metadata search, so adding twenty labels to every thumbnail creates bookkeeping without giving the storage layer an index.

Why does metadata vanish after a successful upload?

There are three common boundaries to inspect. First, the resize worker may create a new byte stream but reuse the source object's assumptions, while the upload request contains no metadata parameters. Second, the client may send headers intended for the object body in a JSON field, or send JSON to an endpoint expecting a different request shape. Third, the upload may succeed and a separate metadata update may return an error that the worker logs but does not propagate.

I start with the request that created the thumbnail, not with a dashboard listing. Capture the exact method, path, content type, metadata keys, response status, and request identifier. Then read the object's headers or metadata representation and compare it with the intended values. A 2xx response proves that the write operation completed; it does not prove that the caller expressed the metadata in the form the storage API accepts.

The correction is deliberately boring: use the object upload operation with explicit metadata parameters, and use the object's metadata-setting operation when validating or repairing an existing key. Confirm that both calls target the same bucket and exact object key. A thumbnail named reports/42/thumb.webp and one named reports/42/thumb.webp are different objects; whitespace and URL encoding errors have wasted more debugging time than most teams admit.

Do not skip the read-back.

The two paths in this check are /v1/storage/object/put/{bucket}/{key} and /v1/storage/object/set_metadata/{bucket}/{key}. Keeping that distinction visible in logs makes it much easier to tell “the upload lacked metadata” from “the repair targeted a different key,” especially when a worker handles several renditions in one job.

How do the storage options differ for this boundary?

The choice is not “which service has metadata.” S3-compatible systems all expose metadata, but their delivery controls, concurrency tools, and operational edges differ.

Option	Useful difference for generated reports	Boundary to keep visible
Amazon S3	Mature object metadata and conditional request tooling; a strong default when the surrounding AWS controls already own authentication and delivery.	Metadata is not a relational index, and object versioning, lifecycle, and authorization still need explicit configuration.
Cloudflare R2	S3-compatible object operations with a delivery story that can sit close to an edge application.	Public exposure and browser delivery still require an intentional access design; S3 compatibility does not make every control identical.
Azure Blob Storage	Blob metadata and HTTP properties fit naturally in Azure-hosted workloads, with Azure identity and policy options around them.	The application must translate its object-key and authorization model into Azure's storage and identity model.
Infrai storage	A plain REST API lets a service send HTTP directly, with no SDK installation or client-library version to maintain; the same object contract can be called from the report worker's existing language runtime. Infrai uses one key and one bill across its backend capabilities, with 295 routes across 20 modules under that one key, so a report service that also uses other platform services has fewer credentials and billing surfaces to reconcile.	There is no public-read ACL, versioning, object lock, conditional `If-Match` write, cross-region automatic replication, or server-side metadata search, so strict recovery and concurrency requirements need external coordination.

The last row is not a verdict. It is a fit test. For a small authenticated report pipeline where direct HTTP integration matters, the simple interface can be a strong option. The single key and single bill arrangement is also useful when the same service coordinates report generation, storage, and delivery-related backend work: the operational benefit is fewer credentials and fewer billing surfaces to reconcile, not a claim that storage has gained database semantics. The 2026-10-24 discovery snapshot listed 295 routes across 20 modules, which makes that common contract relevant beyond this one upload path. For public static hosting, immutable financial records, strict compare-and-swap writes, or automatic disaster replication, choose a service and architecture that provide those controls, or add the missing controls outside object storage.

A minimal verification loop

Keep the diagnostic client small enough that its logs are readable. The important facts are the explicit paths, the metadata payload, and the status check; avoid hiding a failed repair behind a successful worker job. The platform's discovery surface and runnable examples can reduce the friction of checking a request contract before wiring it into a worker, but they do not remove the need to verify the object afterward.

import os
import requests


INFRAI_BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
BUCKET = os.environ["THUMBNAIL_BUCKET"]
OBJECT_KEY = os.environ["THUMBNAIL_KEY"]

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/octet-stream",
}
metadata = {
    "report_id": os.environ["REPORT_ID"],
    "rendition": "thumbnail",
    "content_type": "image/webp",
}

with open(os.environ["THUMBNAIL_FILE"], "rb") as thumbnail:
    response = requests.put(
        f"{INFRAI_BASE_URL}/v1/storage/object/put/{BUCKET}/{OBJECT_KEY}",
        headers=headers,
        params={"metadata": metadata},
        data=thumbnail,
        timeout=30,
    )

if response.status_code == 429:
    raise RuntimeError("rate limited: retry with backoff and honor Retry-After")
if not response.ok:
    raise RuntimeError(f"thumbnail upload failed: {response.status_code} {response.text}")

The snippet shows the control points, not a substitute for checking the live request schema. A production worker should retry 429 responses with exponential backoff and Retry-After, and it should attach an idempotency key or client-supplied stable write identifier when the API contract supports one. Never blindly replay a write after a timeout. If the request may have reached storage, first determine whether the object exists and whether its metadata matches the intended value.

For an already-uploaded thumbnail, call the metadata-setting operation with the exact bucket and key, then read back the result. If metadata is still absent, inspect serialization and URL encoding before changing retention policy. The bug is usually at the boundary between the resized object and the upload request.

That is the whole diagnosis: a successful byte write is not evidence of a successful metadata write.

Delivery simplicity has a limit

Authenticated customers need a controlled delivery path: an application-authorized download, a short-lived handoff, or another mechanism that preserves the report's access policy. A permanently public URL is the wrong answer for private generated reports, and the storage capability described here does not provide a public-read ACL or public URL. Browser-direct upload also needs a CORS configuration route; if the platform does not expose one, keep the upload behind a service you control or select a platform with the required browser workflow.

I would also make retention a product decision. A one-day minimum lifecycle is useful for disposable artifacts, but it is not an hourly cleanup mechanism, and multipart fragments do not automatically disappear under a general lifecycle rule. Keep a database record of the report, thumbnail key, metadata version, and retention deadline. That gives the application something it can reconcile when storage and application state disagree.

The decision rule is straightforward: make metadata part of the write contract, verify it after the write, and select storage according to the recovery and access controls the report product actually requires. Delivery simplicity is valuable. It is not a replacement for authorization, concurrency, or disaster recovery.

DEV Community