Short answer: treat every campaign upload as a lease, keep the original and derivative records separate, and make an explicit, idempotent delete the final state transition after the lease expires. For a logistics team generating responsive thumbnails, that policy protects bandwidth without turning a late-arriving dispatch review into a missing-file incident.
A thumbnail service is easy to demo and surprisingly easy to leave untidy. A driver uploads a phone video at a depot; an inference worker creates a small preview; a campaign dashboard reads that preview from a CDN. The campaign ends, but three independent objects can remain: the source video, the thumbnail, and a queue or database row pointing at each one. Retention is therefore a data-flow decision, not a cron expression.
What should a campaign asset retention path delete, and when?
Start with a state machine that names ownership. received means the upload is complete, derived means every requested thumbnail exists, published means a campaign can read it, and expired means no new consumer may fetch it. Only an operator or scheduled policy can move an expired asset to deleted. A failed derivative stays received or derived; it does not silently qualify for cleanup.
For temporary campaign media, I use two clocks. The source lease is long enough for audit and reprocessing, while the derivative lease is shorter because thumbnails are cheap to regenerate. The exact interval belongs in configuration and in the campaign record, not in code comments. Your mileage may vary when legal hold, carrier disputes, or a customer export extends the source lease.
The delete operation must be explicit and repeatable. A worker claims an expired record, sends a delete request for the object key, records the provider's success response, and then marks the record deleted. If the worker is interrupted after the object is gone but before the database commit, the next attempt should treat an already-absent object as success. That is the useful definition of idempotency here.
Here is a small Python policy object and worker boundary. The storage client is deliberately generic, so the same contract can sit in front of an S3-compatible store, a self-hosted gateway, or another HTTP service.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Protocol
class ObjectStore(Protocol):
def delete(self, key: str) -> bool:
...
@dataclass
class Asset:
key: str
kind: str
uploaded_at: datetime
hold_until: datetime | None = None
state: str = 'published'
def eligible(asset: Asset, now: datetime, lease: timedelta) -> bool:
if asset.state != 'published':
return False
if asset.hold_until and now < asset.hold_until:
return False
return now >= asset.uploaded_at + lease
def expire(asset: Asset, store: ObjectStore, now: datetime, lease: timedelta) -> bool:
if not eligible(asset, now, lease):
return False
removed = store.delete(asset.key)
if removed:
asset.state = 'deleted'
return removed
now = datetime.now(timezone.utc)
thumbnail = Asset('campaign/42/thumb-320.webp', 'thumbnail', now - timedelta(days=8))
expire(thumbnail, store, now, timedelta(days=7))
The important line is not the date arithmetic. It is the hold check and the state transition after a confirmed delete. In production, persist a deletion attempt ID and an audit timestamp with the asset row; that gives support staff a way to explain why a preview disappeared without retaining the bytes forever.
How do quality, bandwidth, and format choices change the policy?
Responsive thumbnails are a quality-versus-bandwidth bargain. A 320-pixel preview loads quickly on a handheld scanner, while a larger preview preserves text on a pallet label. Keep width, height, codec, and quality in the derivative key so a later recipe can coexist with an earlier one. Do not overwrite a derivative that an in-flight page may still be reading.
The format is part of the contract. Browsers negotiate supported media types, and the practical options differ for still images, animated images, and video containers; MDN's media formats guide is a useful compatibility reference. Store the declared MIME type and measured dimensions beside each object, then test representative Android, iOS, and desktop clients before shortening a lease.
I once assumed that deleting the database row would be enough. It wasn't. A CDN cache and a retrying thumbnail job can both outlive that row, so the cleanup flow also needs a tombstone or generation token. Consider a dispatcher that retries a timed-out upload while a cleanup worker sees the campaign as expired: without a generation token, the retry can recreate a thumbnail after the purge and make the dashboard appear to resurrect a deleted asset. The token is stored with the campaign version, copied into each derivative request, checked immediately before the write, and recorded in the audit event. That extra handshake takes more database work, but it closes the race between late producers, cache freshness, and an operator who reasonably believes the campaign is gone. Workers check the token before writing a derivative; readers stop accepting a generation marked expired. Three words: delete the bytes too.
Delete deliberately.
Testing the expiry boundary before production
An eval-driven test harness should generate a matrix of asset kinds, clock offsets, holds, and retry outcomes. Assert that a held source never reaches delete, that a thumbnail at exactly its lease boundary does, and that a second delete leaves the final state unchanged. Include a property test for idempotency: applying the same expiry event twice produces one logical deletion.
Test the notebook-to-prod handoff with real metadata fixtures, not only a mocked filename. A fixture should include a Unicode-safe key, an unexpected MIME declaration, a zero-byte upload, and a video whose duration exceeds the campaign limit. I am not sure which edge will dominate your traffic; capture that uncertainty in dashboards showing derivative generation latency, bytes retained by state, delete age, and the count of assets blocked by holds.
Keep logs free of signed URLs and customer payloads. Record an opaque asset ID, policy version, state before and after, and the reason code (lease_expired, legal_hold, or manual_request). Those fields make a failed cleanup reviewable while preserving the privacy boundary around campaign media.
The operational choice is a policy, not a vendor feature
A managed object store may provide lifecycle rules, while a self-hosted service may require your worker to issue the delete call. Either way, the application owns the decision about campaign status, legal holds, and derivative dependencies. Choose the simpler storage contract when your team cannot operate reconciliation jobs; choose a richer pipeline when you need per-customer leases, audit exports, or multiple thumbnail recipes.
The catch is that aggressive expiry is unsuitable when dispatch investigations routinely arrive after a campaign closes. In that case, keep originals under a documented hold and expire only derivatives, or retain a low-resolution evidence copy under a separate policy. Stick with a longer lease when regeneration depends on a model or source that you may not be able to reproduce.
Before shipping, walk through one upload, one retry, one hold, one CDN miss, and one duplicate delete on paper. Then run the same sequence in staging with clocks you control. A clean retention design makes deletion boring: every object has an owner, every exception has a reason, and bandwidth savings never decide whether an investigator can see what happened.
Top comments (0)