Duplicate Image Processing Costs: 4 Checks That Stop Reprocessing the Same Assets

Short answer: stop duplicate event-photo processing with a stable asset identity, a versioned transformation key, an atomic claim before work begins, and a final check that the expected output still exists. A queue-level duplicate check alone is too early and too fragile.

Check	Pick this when	What it catches	Main limitation
Source key plus version	Upload keys are stable and immutable	Repeated events for one stored object	Misses byte-identical copies under new keys
Byte digest plus recipe	Exact duplicate files are common	Renames, copied uploads, and event retries	A re-encoded image gets a new digest
Perceptual fingerprint	Near-duplicate photos matter	Resizes and visually similar copies	False matches need a review policy
Atomic processing ledger	Workers can overlap	Concurrent claims for the same work	Needs lifecycle and retention rules

The practical default is the second and fourth rows together. Compute identity once at ingestion, combine it with the transformation recipe, then let exactly one worker claim that key. Keep the source-key check as a cheap first filter. Add perceptual matching only if the business really wants to treat similar photos as the same asset.

Start there.

How should you debug image processing cost spikes from duplicate event photo assets?

Start with evidence at the boundary of each stage: upload accepted, job published, claim attempted, transform started, output written, and delivery requested. Give every log line assetId, contentDigest, recipeVersion, jobId, and attempt. Those fields turn a cost graph into a traceable count. They also separate three failures that look identical on a bill: one upload emitted several jobs, one job was delivered several times, or several workers passed a non-atomic skip check.

Build three ratios from counters rather than sampling logs: jobs published per accepted asset, transform starts per unique transformation key, and output writes per transform start. A jump in the first points upstream. A jump in the second points at deduplication or concurrency. A normal start ratio with unexpectedly high bytes read and written points toward storage behavior, such as repeatedly downloading the source before discovering that the output already exists.

No guesswork.

Keep the diagram in words: upload -> identify -> claim -> check output -> transform -> publish. Put a counter on every arrow and a timer around the transform. Then compare one narrow window before and during the spike, grouped by recipe version and event source. A deployment that changed a recipe version can legitimately invalidate old results; a producer retry that changes only jobId cannot.

One warning: don't use the queue message ID as the asset identity. Retries often represent the same intent with different delivery metadata, while a deliberate new crop of the same source is different work. Identity belongs to the image and the requested transformation, not to transport.

This is where the missing skip check usually becomes visible. If two workers log claim=won for one transformation key, the claim isn't atomic. If the ledger says complete but the derived object is absent, the system needs to repair the result instead of skipping forever. If a worker downloads 18 MB and only then learns that another worker finished, the check is in the wrong place.

Tiny counters. Big clue.

Pick source identity when uploads are immutable

A source key plus an object version is the least complex choice when one key can never be overwritten and copied files should remain distinct business records. It avoids reading the full object merely to decide whether work has already happened. For a property manager importing photographer folders, that rule may be correct: two listings can own separate records even when the bytes happen to match.

The catch is overwrite semantics. A bare path such as events/launch/front-door.jpg isn't enough if later uploads can replace its bytes. Include the immutable object version or another ingestion-generated identity. Don't choose this method when mobile clients routinely upload the same file under new names and the desired behavior is to process it once across the entire account. Use a byte digest in that case.

Pick content identity for exact duplicate assets

A cryptographic byte digest survives renames and storage moves, so it is a strong key for exact duplicates. Pair it with a canonical recipe version. The version matters because background removal model settings, output format, dimensions, or masking policy can change the correct result even when the source bytes don't.

Formats complicate the word "same." JPEG, PNG, WebP, AVIF, and other image formats use different encoding and compression capabilities; MDN's image format guide is a useful reference when defining accepted inputs and outputs. Two files that look alike can have different bytes, and therefore different byte digests. That's expected. If visual equivalence is the requirement, byte identity is the wrong abstraction.

Perceptual fingerprints can group re-encodes, resizes, and near-duplicates, but they introduce thresholds. A threshold loose enough to catch a recompressed ballroom photo may also group two deliberate burst shots. I'm not sure there is a universal safe threshold because the acceptable collision depends on the property workflow and the cost of suppressing a legitimate output. Measure candidate pairs on a labeled sample, and keep perceptual matching advisory until false-match handling is explicit.

For most pipelines, exact deduplication is the clean first move — deterministic, explainable, and easy to audit.

Make the skip check atomic and output-aware

A read-then-write check races. Worker A reads "not processed." Worker B reads the same thing. Both start an expensive transform. The fix is an atomic insert or compare-and-set on a transformation key, not a faster preliminary read.

Here is a vendor-neutral TypeScript shape. The ledger owns concurrency; object storage confirms that a supposedly completed result is still usable. The interfaces deliberately hide the database, queue, image engine, and storage provider.

type Job = {
  assetId: string;
  sourceKey: string;
  sourceDigest: string;
  recipeVersion: string;
  attempt: number;
};

type Claim =
  | { state: "acquired"; token: string }
  | { state: "running" }
  | { state: "complete"; outputKey: string };

interface ProcessingLedger {
  claim(key: string): Promise<Claim>;
  complete(key: string, token: string, outputKey: string): Promise<void>;
  release(key: string, token: string): Promise<void>;
}

interface ObjectStore {
  exists(key: string): Promise<boolean>;
  read(key: string): Promise<Uint8Array>;
  write(key: string, bytes: Uint8Array): Promise<void>;
}

interface BackgroundRemover {
  remove(bytes: Uint8Array): Promise<Uint8Array>;
}

const transformationKey = (job: Job): string =>
  `${job.sourceDigest}:${job.recipeVersion}`;

async function processPhoto(
  job: Job,
  ledger: ProcessingLedger,
  store: ObjectStore,
  remover: BackgroundRemover,
): Promise<"created" | "skipped" | "deferred"> {
  const key = transformationKey(job);
  const claim = await ledger.claim(key);

  if (claim.state === "running") return "deferred";

  if (claim.state === "complete") {
    if (await store.exists(claim.outputKey)) return "skipped";

    // Reclaiming completed work whose output was removed is a ledger policy.
    return "deferred";
  }

  const outputKey = `derived/${key}.png`;

  try {
    const source = await store.read(job.sourceKey);
    const output = await remover.remove(source);
    await store.write(outputKey, output);
    await ledger.complete(key, claim.token, outputKey);
    return "created";
  } catch (error) {
    await ledger.release(key, claim.token);
    throw error;
  }
}

The database implementation must enforce uniqueness on the transformation key. Application-level promises aren't enough across processes. The acquired claim also needs a lease or equivalent recovery policy so an interrupted worker doesn't reserve work forever. That policy should distinguish a currently running claim from an abandoned one, record the attempt, and emit a metric whenever ownership is recovered.

Notice the ordering: claim first, fetch second. This avoids paying storage-read and cache-fill costs for work another worker already owns. After a completed claim, check the named output before skipping. Lifecycle policies, manual cleanup, or retention changes can remove derived data while leaving the ledger record; an output-aware check prevents that stale record from silently suppressing regeneration.

Use structured outcomes as metrics: created, skipped, and deferred. Alert on transform starts per unique key and on recovered claims, not raw queue depth alone. Queue depth can rise during a healthy import burst. Duplicate starts show actual wasted processing.

Know where deduplication stops helping

This design is not suitable when every delivery must produce a fresh, independently auditable artifact, even for identical bytes and recipes. In that workflow, preserve separate job records and deduplicate only the immutable source download or cache entry. Stick with source identity when legal or tenancy boundaries prohibit cross-record content matching.

A byte digest also won't merge visually identical re-encodes. Choose perceptual matching only with a labeled evaluation set and a decision about false matches. Cache retention remains a business choice: retaining every derived image reduces future processing but grows storage, while aggressive expiry does the reverse. Track stored derived bytes, cache-hit ratio, regeneration count, and transform starts together. One chart cannot pick that trade-off for you.

Fix the counters first. Then make the claim atomic.

References

• MDN, "Image file type and format guide": https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types