Short answer: validate each upload before background cleanup, retain the source as a separately identified asset, and compress only the final delivery derivative. Run those steps during onboarding when a photo must be publishable before a menu goes live; use on-demand generation only when delayed first access is acceptable and derivative demand is genuinely sparse.
That rule treats image work as a lifecycle, not a chain of convenient functions. A food-delivery onboarding flow has a sharper correctness requirement than a casual media gallery: the database must never claim that a menu photo is ready while storage contains only an unchecked source or a partly transformed derivative. The processing engine may retry, but its externally visible effect must occur once. Every accepted transition therefore needs an idempotency key, an immutable input identity, and an audit record that explains what changed.
No shortcuts.
Decision record and invariants
The decision is to process the required delivery derivative before publication, while keeping the original upload and every generated result under distinct identifiers. The menu item points to a derivative only after four gates have succeeded: source validation, safety-policy validation, background cleanup, and final compression. “Four” describes the logical gates in this architecture, not a claim that every organization has the same policy count.
The first invariant is referential: a source identifier never means “whatever bytes currently occupy this key.” It names one accepted upload. The second is causal: a derivative records the source identity and the transformation policy revision that produced it. The third is temporal: only the final derivative can enter the publishable state. The fourth is accounting-like: retries may repeat computation, but a compare-and-swap transition prevents them from publishing multiple competing results for the same source and policy revision.
This is an exactly-once mindset applied to effects, not a promise of exactly-once execution. Workers can be interrupted after writing an object but before acknowledging a job. Another worker can then run the same operation. A deterministic derivative key, coupled with a conditional state transition, makes that repetition harmless; the audit trail can record both attempts while the menu exposes one committed result. Don't make the queue responsible for a guarantee that belongs in application state.
Failure boundaries should be explicit. Invalid media stops before cleanup. A policy rejection remains attached to the source record rather than being disguised as a generic processing failure. A cleanup rejection cannot advance to compression. A compression attempt writes to a provisional derivative identity, and publication occurs only after the completed derivative has been associated with its source and policy revision. The source remains available for a future policy revision, subject to the merchant's retention and deletion rules.
The compliance boundary matters here. Retention periods, consent language, deletion deadlines, and the definition of an unacceptable image depend on jurisdiction and organizational policy; this architecture can enforce a supplied rule and preserve evidence, but it cannot invent the rule. I'm not sure which retention window applies to your merchants, and the answer has to come from counsel and the data owner before rollout—not from a worker default.
What should merchant menu photo background cleanup, lifecycle validation, and compression guarantee?
It should guarantee that a customer-visible reference resolves only to an approved final derivative, and that an operator can trace that derivative back to one immutable source, one policy revision, and one sequence of state transitions. That is stronger and more useful than saying that three image operations returned success.
Start by defining the visible result. A menu search index may store tags and an image reference, but the image reference is eligible for indexing only after the same publication transition that makes it eligible for delivery. Otherwise search can surface a dish whose photo is still provisional. Keep tagging as a downstream consumer of the committed asset event; don't let a tagger infer readiness from the mere existence of an object. A representative test corpus should include each source category the onboarding flow actually accepts, the target dimensions the application actually serves, and examples that policy owners have labeled unacceptable. The expected output is not “looks better.” It is a testable tuple: accepted or rejected source, expected terminal lifecycle state, expected derivative identity, and whether a publish event exists. Visual review remains necessary for cleanup quality, while lifecycle assertions can be deterministic. The media format decision also belongs at the delivery boundary. Format support and codec characteristics differ, so source acceptance and delivery encoding shouldn't be collapsed into one assumption. Record the detected source format, choose the delivery format from client and product requirements, and verify representative outputs using the media-format guidance in the reference below. Your mileage may vary across the actual client population; observed client requirements, rather than a fashionable default, should resolve that uncertainty.
Upload-time versus on-demand processing
The primary choice is where to place the publishability barrier. Both strategies can preserve the same state machine, but they assign latency and operational risk to different actors.
| Decision axis | Process during upload | Generate on demand |
|---|---|---|
| Publication rule | Onboarding waits for a committed derivative | Menu can exist before a derivative is requested |
| First-view behavior | Reads resolve an already committed asset | First access may initiate or wait for work |
| Sparse derivative demand | May create outputs that are never read | Avoids work until a derivative is requested |
| Retry ownership | Onboarding workflow owns retry state | Read path and background work need a shared contract |
| Audit point | Approval and asset commit form one boundary | Approval and generation occur at different times |
| Best fit | Every published dish requires a ready photo | Photos are optional or most derivative variants are rarely requested |
For food-delivery merchant onboarding, upload-time processing is the cleaner default because image readiness is part of the publication decision. The catch is that it is not suitable when onboarding must accept metadata immediately even though media approval is intentionally asynchronous, or when the system supports a large set of optional derivative variants with sparse demand. In that case, keep lifecycle validation before publication but generate the optional variants on demand. The important distinction is between delaying a nonessential derivative and delaying evidence that the source is allowed to be shown.
On-demand generation is also valid when the product explicitly tolerates a placeholder and the first request is not a contractual latency boundary. It still needs idempotency. Two readers can request the same absent derivative at nearly the same time, so both must converge on the same derivative identity and one committed state rather than racing to mutate a menu record.
The critical path in Go
The code below models the transaction boundary rather than any particular image library. Transformer implementations perform validation, cleanup, and compression; Repository owns conditional state changes and the audit trail. The service derives its idempotency key from stable identities, so a redelivered command targets the same logical result.
package media
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
)
type State string
const (
Uploaded State = "uploaded"
Validated State = "validated"
Cleaned State = "cleaned"
Compressed State = "compressed"
Publishable State = "publishable"
)
type Asset struct {
SourceID string
PolicyRevision string
State State
SourceBytes []byte
CleanBytes []byte
DeliveryBytes []byte
}
type Transformer interface {
Validate(context.Context, []byte, string) error
RemoveBackground(context.Context, []byte) ([]byte, error)
Compress(context.Context, []byte) ([]byte, error)
}
type Repository interface {
Load(context.Context, string) (Asset, error)
Transition(context.Context, string, State, State, Asset) error
AppendAudit(context.Context, string, State, string) error
}
type Service struct {
Transform Transformer
Repo Repository
}
func (s Service) Prepare(ctx context.Context, sourceID, policyRevision string) error {
key := derivativeKey(sourceID, policyRevision)
asset, err := s.Repo.Load(ctx, sourceID)
if err != nil {
return err
}
if asset.PolicyRevision != policyRevision {
return errors.New("policy revision mismatch")
}
if asset.State == Uploaded {
if err := s.Transform.Validate(ctx, asset.SourceBytes, policyRevision); err != nil {
return err
}
if err := s.Repo.Transition(ctx, key, Uploaded, Validated, asset); err != nil {
return err
}
asset.State = Validated
_ = s.Repo.AppendAudit(ctx, key, Validated, policyRevision)
}
if asset.State == Validated {
asset.CleanBytes, err = s.Transform.RemoveBackground(ctx, asset.SourceBytes)
if err != nil {
return err
}
if err := s.Repo.Transition(ctx, key, Validated, Cleaned, asset); err != nil {
return err
}
asset.State = Cleaned
_ = s.Repo.AppendAudit(ctx, key, Cleaned, policyRevision)
}
if asset.State == Cleaned {
asset.DeliveryBytes, err = s.Transform.Compress(ctx, asset.CleanBytes)
if err != nil {
return err
}
if err := s.Repo.Transition(ctx, key, Cleaned, Compressed, asset); err != nil {
return err
}
asset.State = Compressed
_ = s.Repo.AppendAudit(ctx, key, Compressed, policyRevision)
}
if asset.State == Compressed {
if err := s.Repo.Transition(ctx, key, Compressed, Publishable, asset); err != nil {
return err
}
_ = s.Repo.AppendAudit(ctx, key, Publishable, policyRevision)
}
return nil
}
func derivativeKey(sourceID, policyRevision string) string {
sum := sha256.Sum256([]byte(sourceID + "\x00" + policyRevision))
return hex.EncodeToString(sum[:])
}
There is deliberate friction in this interface. Transition receives both the expected and next state; an implementation must reject a stale writer rather than silently overwrite current state. In production, audit persistence and state mutation should share the same transactional boundary or use an outbox whose insertion shares that boundary. The sample leaves those storage mechanics behind an interface because choosing a database without supplied requirements would pretend to settle a decision that the architecture does not settle.
One detail deserves scrutiny: ignoring the return from AppendAudit is acceptable only if the repository has already made the audit event durable as part of Transition, and AppendAudit merely forwards that durable record. If audit insertion can fail independently, change the interface so the transition cannot commit without it. Auditability is an invariant, not optional telemetry.
Deployment follows the same discipline. Introduce new transformation policy revisions without reusing old derivative identities, canary them against the representative corpus, and compare terminal states plus reviewed outputs before allowing publication. Operational counters should distinguish validation rejection, cleanup rejection, compression rejection, stale transition, and successful publication; a single “image failed” counter cannot tell an operator whether to correct merchant input, policy, or capacity.
Rejected option, and when to choose it
The rejected design is a read-path pipeline that discovers an original photo, removes its background, compresses the result, and updates the menu reference during the customer request. It combines an unbounded transformation with a read, makes concurrent readers participants in lifecycle coordination, and weakens the moment at which the onboarding system can honestly say “published.” A cache reduces repeated work but does not create the missing state transition or audit evidence.
Stick with on-demand processing when derivatives are optional, variant demand is sparse, and the product permits a placeholder or delayed response. Even then, validate the source and settle its retention state before publication, use a deterministic key per source and policy revision, and commit the generated derivative conditionally. This is not a universal upload-time mandate; it is a rule that required evidence belongs before publication, while optional representation can wait.
The final acceptance test is compact: given the same source identity and policy revision twice, the workflow may record two attempts but exposes one publishable derivative; given a new policy revision, it creates a distinct derivative and preserves lineage; given a rejected source, it emits no publish event; and given a deletion instruction under the applicable policy, it can identify the source and every derivative derived from it. Those properties make background cleanup and compression subordinate to lifecycle correctness—which is where they belong.
Top comments (0)