Short answer: Serve a watermarked derivative from a separate object, never rewrite the museum's master image. Keep the master immutable, make derivative creation idempotent, and let the portal publish only a validated, access-controlled thumbnail. That gives visitors responsive images while preserving the archival file byte for byte.
I learned to treat this as a scheduling problem after a thumbnail worker was retried during an upload burst. The queue delivered one job twice, and both attempts tried to write the same public object. Nothing was lost, but the incident made the invariant obvious: a retry may repeat computation, not change the preservation record. The first alert showed only queue depth, so the on-call had to trace the upload ID through the manifest, temporary object, and CDN log before deciding whether another render was safe. That took longer than the render itself. We added a duplicate-claim counter and a manifest lookup to the runbook after that review.
It failed loudly.
That distinction matters in a collection portal. A curator may replace a display caption, but an imaging master usually has a chain of custody, checksums, and retention rules. A web-friendly copy belongs in a different namespace with a different lifecycle.
What should a museum portal preserve when it publishes watermarked images?
Preserve the original bytes and their identity first. On ingest, record a content hash, media type, dimensions, capture identifier, rights statement, and preservation storage key. Treat that row as append-only. A thumbnail request can refer to the row, but it cannot mutate it.
The public derivative needs its own key. I use a tuple such as (master_hash, transform_version, watermark_version, size), encoded as a stable path. If the watermark artwork or resize policy changes, the version changes and a new object is produced. Old derivatives can be retired according to policy without rewriting the master.
There is a useful operational split:
| Record | Purpose | Mutable fields |
|---|---|---|
| Master manifest | Chain of custody and preservation | Status metadata only, by approval |
| Derivative manifest | Public thumbnail and watermark identity | Build state, access policy, expiry |
| Delivery log | Evidence of who received what | Append-only event data |
Do not infer a rights decision from a file extension. Browsers support multiple image formats and codecs, and format choice affects compatibility and payload size; the MDN media formats guide is a good baseline for testing actual browser targets. The rights statement and portal policy still decide whether a derivative may be delivered.
Build the derivative path as a replayable job
The upload handler should commit the master manifest before enqueueing derivative work. A worker then claims a job, reads the immutable source, renders into a temporary buffer, applies the watermark, validates the output, and atomically publishes the derivative. A lost acknowledgement causes another claim; it must not create a second logical derivative.
Here is the shape of that path in Go. The storage methods are deliberately generic so the same fencing rules can sit above object storage, a filesystem, or a managed image service.
package thumbnails
import (
"context"
"fmt"
"time"
)
type Job struct {
ID string
MasterKey string
MasterHash string
TransformVersion string
WatermarkVersion string
Size int
Fence int64
}
type Store interface {
Claim(ctx context.Context, id string, lease time.Duration) (Job, error)
ReadMaster(ctx context.Context, key string) ([]byte, error)
PublishDerivative(ctx context.Context, key string, data []byte, fence int64) error
Complete(ctx context.Context, id string, fence int64, key string) error
Release(ctx context.Context, id string, fence int64, reason string) error
}
type Renderer interface {
Thumbnail(ctx context.Context, source []byte, size int) ([]byte, error)
Watermark(ctx context.Context, image []byte, version string) ([]byte, error)
}
func Run(ctx context.Context, store Store, renderer Renderer, id string) error {
job, err := store.Claim(ctx, id, 60*time.Second)
if err != nil {
return err
}
source, err := store.ReadMaster(ctx, job.MasterKey)
if err != nil {
_ = store.Release(ctx, job.ID, job.Fence, "master read failed")
return err
}
thumb, err := renderer.Thumbnail(ctx, source, job.Size)
if err != nil {
_ = store.Release(ctx, job.ID, job.Fence, "thumbnail render failed")
return err
}
marked, err := renderer.Watermark(ctx, thumb, job.WatermarkVersion)
if err != nil {
_ = store.Release(ctx, job.ID, job.Fence, "watermark render failed")
return err
}
key := fmt.Sprintf("derivatives/%s/%s/%d", job.MasterHash, job.WatermarkVersion, job.Size)
if err := store.PublishDerivative(ctx, key, marked, job.Fence); err != nil {
return err
}
return store.Complete(ctx, job.ID, job.Fence, key)
}
The fence prevents an expired worker from publishing after a replacement worker has claimed the job. PublishDerivative and Complete must reject stale fences. The derivative key is deterministic, so an accepted retry overwrites the same derived object or observes it already present; it never invents a new public URL.
That's the whole point.
One concrete failure mode is a process dying after publication but before completion. The next worker should check the deterministic key and validate its metadata before doing expensive work. Another is a process dying before publication; the next worker renders normally. Those cases look identical in queue metrics, so retain job state, fence, and object version in the manifest.
Keep raw visitor identifiers out of object keys and metric labels. Use a short request ID in logs, and put the full audit event in an access-controlled store. Watermarking is an access signal, not encryption; a downloaded derivative can still be copied.
How do responsive thumbnails, watermark policy, and access checks fit together?
Resolve authorization before expensive rendering when possible. The portal can return a signed URL only after checking collection policy, embargo dates, and the derivative manifest. The URL should point to the derivative namespace, never the master key. Cache headers must match the policy: a public, stable derivative can be cached longer than an embargo-sensitive one.
Responsive delivery is a bounded set of sizes, not arbitrary user input. Predeclare widths such as 320, 640, and 1280 pixels, then include the selected size and format in the derivative identity. This keeps storage growth predictable and makes cache invalidation a version change. Your mileage may vary on the exact widths; measure the portal's layouts and the connection profiles of its visitors.
Keep it boring.
Validate dimensions, decoded pixel limits, color profile handling, and output media type before publication. A malformed upload should be quarantined with an operator-visible reason. Never replace the master with a normalized copy just because a browser decoder rejected the source.
The trade-off is visible. A derivative pipeline costs storage and a little delay on first view. In return, it gives the archive an immutable source and a clear place to enforce presentation policy. If a portal requires pixel-perfect, unwatermarked research downloads, this path is not suitable for that endpoint; keep a separately authorized original-download workflow. Stick with on-demand rendering only when traffic is sparse and cold-start latency is acceptable.
Compare implementation choices by failure behavior
A self-hosted image worker offers control over codecs, fonts, and where bytes are processed, but your team owns patching and capacity. A managed transformation service can shorten operations work, yet its retention, region, and watermark primitives become part of the compliance review. A CDN image layer is excellent at delivery and cache fan-out, but it should consume an approved derivative, not reach into preservation storage.
I would compare options with the same test corpus: a large TIFF master, a PNG with transparency, an unusual color profile, a rights-restricted object, and a duplicate upload. Record output dimensions, media type, watermark placement, cache behavior, and audit events. Do not rank tools on a single throughput number; a fast renderer that cannot prove which master produced a public image fails the real requirement.
Run a canary with a small collection subset. Watch queue age, duplicate claims, derivative validation rejects, source-read latency, and unauthorized URL attempts. Keep the previous transform version available so rollback means stopping new admissions and serving the last known-good derivative set. Reconciliation comes first; replaying every queued item can hide whether an object was already published.
The selection rule is simple: choose the implementation that preserves the master, makes retries harmless, and leaves an audit trail a curator can understand. Rendering speed is useful. Provenance is the gate.
Top comments (0)