DEV Community

SolomonFletcher5872
SolomonFletcher5872

Posted on

Node.js Store AI-Generated Images in Object Storage: Retention Example

Node.js AI Image Lifecycles: Retention Tests for Private Buckets

The operational constraint is deletion, not image upload. Short answer: store AI-generated images in private object storage, keep retention in an application record, and mint a presigned download URL only after authorization. That gives a developer tool something testable: one artifact, one policy version, one deletion decision.

The experiment starts with a deliberately boring question: can a retry, a browser download, and a retention worker all refer to the same image without creating a second copy or extending its life by accident? A direct upload-and-return-URL flow is easy to demo and hard to audit. Before copying either pattern, measure orphaned objects, deletion latency, duplicate generation IDs, and denied-download decisions. Those measurements matter more than the storage client.

Retention governance starts with a replayable record

I start with a fixed replay, not a bucket. It contains generated images for three teams, two retention policies, one repeated generation ID, one interrupted cleanup job, and one download after expiry. The simple implementation writes bytes, stores a URL, and relies on a scheduled prefix cleanup. The candidate writes an exact key, records a state transition, and asks for deletion of that exact key only after the record is eligible. This turns a vague storage choice into a governance test: the same input should produce the same retention decision even after the storage adapter changes.

That fixture makes the decision concrete. Can a retry, a browser download, and a retention worker all refer to the same image without creating a second copy or extending its life by accident? Before copying either pattern, measure orphaned objects, deletion latency, duplicate generation IDs, and denied-download decisions.

The numbers decide.

The second implementation is more work up front. It is also the only one that can answer “why does this object still exist?” without reconstructing intent from logs. A retry after an ambiguous database response should look up gen_884 and reuse its key. It must not call the image model again merely because persistence is uncertain. That is prompt-cost aware, and it keeps inference evaluation separate from storage recovery.

Run this in a notebook with deterministic clocks, then move the same assertions into CI. I'm not sure your review window should be one hour or seven days; usage data and policy requirements should settle that, not a copied constant.

Can the artifact survive a storage-provider migration?

Store metadata before treating the bytes as the whole artifact. A useful record has a generation ID, object key, content type, creation time, retention deadline, policy version, and lifecycle state. The URL is deliberately absent. It is a temporary capability, not durable metadata.

This distinction fixes a common notebook-to-prod mistake: the notebook keeps the returned URL and calls that persistence. When the URL expires, the notebook has no principled way to mint another one; when the policy changes, there is no record explaining which deadline applied. The application record is the source of truth, while the bucket holds the payload. A migration should move bytes and preserve those records without changing the policy engine, which is a much better acceptance test than checking whether a new SDK can upload a PNG.

Here is the policy kernel I would put under an eval harness. It has no provider SDK, so boundary dates and retry behavior can be tested with fixed timestamps before an integration test needs credentials.

from dataclasses import dataclass
from datetime import datetime, timezone


@dataclass(frozen=True)
class ImageArtifact:
    generation_id: str
    object_key: str
    content_type: str
    retain_until: datetime
    policy_version: str
    state: str = "retained"

    def eligible_for_deletion(self, now: datetime) -> bool:
        if now.tzinfo is None:
            raise ValueError("now must include a timezone")
        return self.state == "retained" and now >= self.retain_until


artifact = ImageArtifact(
    generation_id="gen_884",
    object_key="training/team-17/run-884/image.png",
    content_type="image/png",
    retain_until=datetime(2026, 8, 20, tzinfo=timezone.utc),
    policy_version="training-review-v2",
)

assert not artifact.eligible_for_deletion(
    datetime(2026, 8, 19, 23, 59, 59, tzinfo=timezone.utc)
)
assert artifact.eligible_for_deletion(
    datetime(2026, 8, 20, tzinfo=timezone.utc)
)
Enter fullscreen mode Exit fullscreen mode

Three details are doing real work here. The object key is derived from an application ID, not supplied by a browser. The policy version travels with the record, so a later audit can explain the decision. And the deadline comparison is explicit at the exact boundary. Tiny details. Expensive to repair later.

A deletion replay exposes the dangerous failure

Use states such as retained, eligible, deletion_requested, deleted, and deletion_recorded. A worker claims an eligible row, requests deletion for its exact object key, and records the outcome. If the process stops between those steps, the next run needs an idempotent decision, not a broad prefix delete. The eval should assert that repeated cleanup never changes the retention deadline and never touches another generation.

Start with the awkward replay.

An object write can succeed while the database response times out with DB_TIMEOUT. The caller retries. If the retry invents a second key, one logical training image now has two blobs and possibly two retention deadlines; if cleanup sees only the row, the orphan remains invisible. Later, a viewer can request a URL while a worker is claiming the old row. Those are separate transitions, so the trace must show which one won, which key it addressed, and whether the next retry is safe. This is why I test the policy with a fake clock and a fake adapter before I test a real client: the scenario is a state-machine problem first, and an object-storage problem second.

The useful scorecard is small:

Test Expected invariant Useful evidence
Repeated generation ID One logical artifact maps to one key ID-to-key lookup
Cleanup at the deadline The boundary is consistent UTC timestamp and policy version
Interrupted cleanup A retry is safe Job ID, attempt, outcome
Download after expiry No new capability is issued Authorization decision
Cross-team request The caller cannot read another team Tenant and actor record

How can Node.js serve private object storage with presigned download URLs?

The request path should be boring and ordered. Load the application record by generation ID. Check the caller's tenant and authorization. Check that the artifact is not deleted and that the requested operation is allowed. Only then ask the storage adapter to create a short-lived GET URL for the stored key.

A presigned download URL is a bearer capability. Whoever receives it can use it until it expires, so putting it in a durable database column turns a temporary access decision into a long-lived leak. A human preview, an evaluation worker, and a mobile download may need different expiry policies. Keep that choice in the application policy and record the issuance event.

Direct browser downloads also involve CORS. If the browser fetches the signed URL from a different origin, the storage response must have headers that match the application's cross-origin rules. MDN's CORS guide explains the browser checks. CORS decides whether browser code can read a response; it does not grant storage authorization or make a private bucket public.

The catch is that this pattern is not suitable for a permanent public gallery, anonymous hotlinking, or an archive whose records must be undeletable. Use a public delivery model for openly licensed images, or choose an archive with the required immutability controls for records that cannot be deleted. Those are different capabilities, not reasons to stretch a private review design beyond its job.

What should the storage adapter and observability prove?

The adapter should expose concepts the application can test: write bytes at an exact key, delete an exact key, and create a temporary GET capability. It should not make the retention policy depend on a bucket listing, a provider URL format, or a browser-generated filename. A fake adapter can then verify the state machine, while an integration test verifies headers, content type, and the real object-storage contract.

Test the failure classifications separately. MODEL_RETRY, DB_TIMEOUT, OBJECT_WRITE_RETRY, and DOWNLOAD_DENIED lead to different recovery actions. A generic “request failed” branch hides whether the system might regenerate an image or issue a URL after the deadline. I want those classifications in traces and in the test fixtures, because an eval harness that cannot distinguish them is measuring noise.

The application should observe orphan count, duplicate generation IDs, deletion latency, URL issuance, denied requests, and policy versions in use. Keep timestamps in UTC. Record the actor or job, exact key, attempt number, and outcome. Your mileage may vary on retention windows and URL expiry, but the observability fields are stable enough to define before launch.

The decision rule is simple: choose a private bucket and presigned downloads when controlled, temporary review is the job; choose a public delivery or immutable archive when permanence is the job. In both cases, test the lifecycle against the promise made to users. An image that can be generated is not necessarily an image that can be governed.

References

Top comments (0)