The classified ad photos are not safe merely because an upload returned 201; the lifecycle validation boundary must keep each photo private until moderation has approved it. The moderation queue is red. A listing page is serving a thumbnail that the review service has never seen, and the on-call view shows only photo_id=8f31 and a growing retry counter. That page is the end of the story, not the beginning.
Short answer: put every classified ad photo through a quarantine state machine, validate bytes and declared media format before a public key exists, and publish derivatives only after a moderation decision with an auditable expiry path.
This boundary is less glamorous than choosing an image API, but it protects the moderation-coverage SLO. A safe upload is an untrusted object with a known next transition, not a file that happened to return HTTP 201.
Why does a classified ad photo need a lifecycle boundary?
The failure I look for first is an ordering bug: the web request writes listing/123/cover.jpg, a thumbnail worker notices it, and a moderation worker receives a later event. Under load, the thumbnail wins the race. The buyer sees a polished derivative while the source is still awaiting review. Nothing crashed; the system violated its policy. In a busy marketplace, that race can also trigger a cache fill, a search-index update, and a seller notification before anyone has a chance to retract the listing, so the cleanup work grows with every downstream consumer rather than with the one mistaken write.
That race is expensive.
Treat the object as a state machine instead:
received -> quarantined -> inspected -> moderation_pending -> approved | rejected -> expired
The public namespace must contain only approved derivatives. A private quarantine key can be scanned, re-read, and deleted without creating a cache entry. Store an immutable digest, byte count, media-type decision, scanner version, and moderation decision beside the object. Those fields let an incident responder answer “what was published?” without trusting a mutable filename.
The state transition should be conditional. A worker may move inspected to moderation_pending only if the object generation and digest still match the record it inspected. That compare-and-set check catches a replaced upload that kept the same photo_id. It also gives you a clean metric: count rejected transitions, not just failed HTTP requests.
What should upload validation measure before moderation?
Start with the bytes, then compare them with the declared type. The browser's Content-Type is a hint; magic bytes and a decoder are evidence. MDN's media-format guidance is a useful reminder that a format label covers containers, codecs, and browser support separately. A file can be a valid container and still be unsuitable for the derivative pipeline.
The first pass should be cheap and deterministic: enforce a byte ceiling, reject truncated streams, parse dimensions, and normalize orientation into a canonical derivative. The second pass can be asynchronous and expensive: malware scanning, perceptual-hash extraction, and policy classification. Never make the expensive pass a reason to keep a public object around.
Here is the shape of a small Go gate. It reads at most one byte beyond the configured limit, uses http.DetectContentType only as an input to policy, and records the result rather than silently rewriting the source.
package intake
import (
"fmt"
"io"
"net/http"
)
type Decision struct {
Allowed bool
Mime string
Bytes int64
Reason string
}
func Validate(r io.Reader, maxBytes int64) (Decision, error) {
if maxBytes < 1 {
return Decision{}, fmt.Errorf("maxBytes must be positive")
}
limited := io.LimitReader(r, maxBytes+1)
data, err := io.ReadAll(limited)
if err != nil {
return Decision{}, err
}
if int64(len(data)) > maxBytes {
return Decision{Reason: "byte_limit"}, nil
}
if len(data) == 0 {
return Decision{Reason: "empty"}, nil
}
mime := http.DetectContentType(data)
switch mime {
case "image/jpeg", "image/png", "image/webp":
return Decision{Allowed: true, Mime: mime, Bytes: int64(len(data))}, nil
default:
return Decision{Mime: mime, Bytes: int64(len(data)), Reason: "media_type_policy"}, nil
}
}
The allow-list is a product decision, not a claim that these are the only safe formats. If animated content, HEIC, or short videos matter to your marketplace, add a decoder and an explicit moderation policy for each. Your mileage may vary across browsers and image libraries; record the decoder version so a later reprocessing job does not produce an unexplained change.
From alert to action: instrument the boundary, not the symptom
The useful alert fires before the listing page is wrong. I want a ratio of published_without_approved_decision to all publication attempts, with a target of zero, plus a queue-age SLO for items waiting on moderation. A second alert watches the percentage of quarantined objects whose lifecycle timer has expired without a terminal decision. That is a storage leak and a coverage risk at the same time.
When the page alert fires, the runbook should walk backward: fetch the decision record by digest, verify the object generation, inspect the transition event, and compare timestamps for quarantined, inspected, and approved. If the digest differs, quarantine the derivative and mark the listing for re-review. If the timestamps are missing, the instrumentation is the incident, not an excuse to increase retries.
Keep counters separate for policy rejects, decoder errors, scanner timeouts, and queue delays. Aggregating them into “upload failures” hides the trade-off the platform team must make: a stricter decoder may increase rejects while improving moderation coverage. Sample payloads by digest, never by raw image URL, and redact listing metadata from logs unless the responder needs it.
False positives have a cost. If a transient scanner timeout is treated as a rejection, sellers retry uploads and the queue doubles; if it is treated as approval, an unreviewed photo can ship. The boundary should therefore support a bounded needs_review state with a deadline, not an infinite retry loop.
Buy or build the validation and moderation path?
The decision is about on-call load and coverage, not a glossy feature checklist. A managed component can reduce the number of services your team patches, while a self-hosted pipeline gives you control over retention, model updates, and regional data handling. Neither removes the need for the lifecycle contract.
| Choice | Strength | Cost or limitation | Fit for a classified-ad team |
|---|---|---|---|
| Managed media inspection | Fast rollout and vendor-maintained parsers | Data-transfer boundaries and opaque scoring can complicate audits | Good when policy is standard and review volume is predictable |
| Self-hosted decoder plus scanner | Full control of bytes, versions, and retention | You own patching, capacity planning, and 24/7 alerts | Good when listings contain regulated or region-bound media |
| Hybrid quarantine with external moderation | Local custody for uploads, specialized review downstream | Two contracts and more event reconciliation | Good when moderation coverage is the primary axis |
Start with a capacity model: peak uploads per second multiplied by worst-case inspection time, then add headroom for reprocessing. Set an SLO for the age of the oldest moderation_pending item and a separate latency objective for the user-facing upload acknowledgement. A synchronous request should acknowledge quarantine, not pretend that moderation finished.
This architecture is not suitable when the product requires instant, unreviewed public posting or when your team cannot operate a durable queue. In those cases, reduce the supported media surface and use a simpler synchronous gate; stick with a mature hosted workflow when the compliance boundary and staffing model outweigh custom control. I am not sure a single “accuracy” number can settle that choice, because false negatives and false positives affect different people and different budgets.
Make expiry and reprocessing boring
Every quarantined object needs a retention deadline and a deletion job that is idempotent. Every approved derivative needs a pointer to the source digest and policy version. When a policy changes, enqueue reprocessing by digest; do not overwrite the old decision in place. The listing should remain approved only while the current policy permits it.
Test the transitions with adversarial fixtures: a valid JPEG with a false extension, a truncated PNG, a file one byte over the limit, duplicate uploads with different names, and an event delivered twice. Inject queue delays and worker restarts. The invariant is simple: no public derivative without a current approval record, and no approval record without a matching inspected digest.
That invariant is the useful boundary. It survives a provider swap, a new codec, and a moderation policy rewrite because the contract is about evidence and state, not a particular API.
References
- https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats
- https://www.rfc-editor.org/rfc/rfc9110
- https://sre.google/sre-book/service-level-objectives/
Top comments (0)