Short answer: put byte validation and one review-sized derivative before publication, then defer the full compression matrix until approval. Model the workflow as explicit state transitions, and make telemetry answer queue-age and failure questions without turning every upload ID into a permanent label.
That boundary matters for a developer tool that accepts screenshots and captions. A rejected item should not trigger six encodes, while an approved item should not make the first reader wait for a cold codec path. I count storage and cardinality together: a log line is bytes retained, and a label is a future index with a potentially unbounded value set.
What must remain true across the pipeline?
The architecture record starts with invariants, not a vendor choice. Original bytes are immutable and private. A moderation decision covers the image and its caption as one revision. Only an approved state can reach a public serving key. Retries are idempotent. Every generated derivative names the source hash, transformation policy, and output format so a policy change can replay from one source.
There are three placement choices. Their failure boundaries are different.
| Placement | Useful property | Failure boundary |
|---|---|---|
| All derivatives at upload | Predictable reads after approval | CPU and storage are spent on items that may be rejected |
| All derivatives on demand | Small upload critical path | First approved read can encounter a timeout, stampede, or codec failure |
| Review derivative at upload; final derivatives after approval | Bounded review latency and regenerable outputs | Requires two queues and explicit state transitions |
The third row is the decision rule here. It keeps the quarantine path small while preserving a replay point for future formats. It also makes the public read path simpler: a missing derivative is a build state, not permission to inspect a pending object.
Keep that boundary boring.
How can an API moderate user uploaded images and captions?
The API acknowledges an upload only after it has a durable object address and an idempotency key. The worker checks the bytes, not the filename, before decoding. Browser-supplied MIME metadata is a hint; the server must reject malformed headers, implausible dimensions, and unsupported formats before an image library spends significant memory.
The critical request can stay intentionally plain:
curl -X POST https://example.test/v1/image/moderate \
-H 'Idempotency-Key: intake-12f9' \
-F 'asset=@screen.png;type=image/png' \
-F 'caption=Toolbar regression on compact view'
The response returns a moderation identifier and pending_review. A queue message contains that identifier, the immutable object key, the caption revision, and a policy version. It does not carry image bytes or an unbounded caption history. The worker writes one bounded review rendition, records its content hash, and uses a compare-and-set transition to move the item into in_review.
Here is the part that is easy to miss. A retry can happen after the derivative is written but before the state transaction commits. If the output key is derived from the source hash and policy version, the second attempt observes the same object instead of creating a second copy. The transition remains the authority. Files are not the state machine. That distinction has saved more than one queue design from duplicate work, because object stores and queues do not share a transaction boundary, and a worker crash between those two writes is ordinary behavior rather than an exotic edge case that testing can wish away.
A reviewer approval enqueues the compression job. Rejection marks the moderation revision terminal and keeps the original under the configured retention rule, never under the public serving prefix. The caption is versioned with the decision so an edit cannot silently inherit an older image verdict.
Which telemetry proves the boundary is working?
I would retain 100% of state changes, moderation outcomes, and terminal failures. Those records answer audit questions. Successful encode timings can be aggregated into one-minute histograms; a p95 and p99 are usually more useful than one event for every asset. Keep a small tail of slow successes so a regression is still inspectable.
Labels must have bounded sets: stage, result, policy_version, and perhaps a coarse format. Do not put the upload ID, object key, caption text, or exception message in a metric label. Put request identifiers in trace context and structured logs with a retention limit. The W3C Trace Context specification defines the propagation fields; it does not require retaining every span forever.
This is a deliberate loss of detail. It is also reversible: sampled traces can point to a request, while counters continue to show whether pending_review is growing. A queue-age alarm is actionable. A dashboard with millions of unique asset labels is merely expensive.
The useful measurements are queue age, review latency, derivative failure rate, retained bytes by state, and the ratio of regenerated to first-pass outputs. Alert on age and failed transitions, not raw event count. A twelve-event intake sample should not be mistaken for a throughput benchmark; it is a test fixture for checking that each transition emits the expected bounded fields.
When is on-demand compression the right exception?
On-demand generation fits a large variant matrix, sparse approval traffic, or an experiment with a new quality policy. It needs a single-flight guard keyed by source hash and transformation policy. Ten simultaneous browser requests must not become ten encodes. Cache the completed object, and return a non-public build state while the derivative is pending.
Upload-time work remains appropriate for the review rendition because reviewers need a stable, bounded artifact. Run decoders in a constrained worker pool and test decompression bombs, huge dimensions, truncated files, and captions containing control characters. The image format guide on MDN is a useful reference for browser formats, but it is not a substitute for server-side byte validation.
I rejected the option of producing every size before moderation. It makes reads look tidy, yet couples rejected content to the entire derivative inventory and makes a codec-policy change an expensive sweep. That option has a valid use case: a closed catalog where every accepted asset is guaranteed to receive the same variants immediately. An open upload queue has a different risk profile.
This approach is not a universal fit. A tiny internal tool with five trusted uploaders may reasonably process synchronously and keep no review queue. Conversely, a regulated archive may require a separate evidence store and legal hold controls that this state machine does not provide. Those are boundary conditions, not reasons to hide the trade-off.
The final record is compact: immutable source, one quarantine rendition, two idempotent queues, and bounded telemetry. No product name is needed to make that decision. The important question is where an error becomes visible, who can see the bytes at that point, and whether the next retry repeats work.
Top comments (0)