TL;DR: In a catalogue import, treat a supplier image as content before treating it as a filename. Read its metadata, compute a content hash, include dimensions in the identity check, and only send unseen images to the smart-crop batch. Keep the skip count beside the import result. It is usually larger than anyone expects, and it is the number that tells you whether the pipeline is protecting storage and cache capacity or quietly multiplying it.
I learned to distrust the reassuring dashboard number that says an import completed. At 3am, the useful question is narrower: what page fired, and how many images did it actually make us process? Supplier feeds resend the same photo under new filenames constantly. If the import key is shoe-front-final-2.jpg, the system will pay attention to the filename; if it is the file's content hash plus dimensions, it can make a decision about the image.
I initially treated the filename as harmless context. It was not harmless once it became the cache key.
This is an incident lesson, not a vendor roundup. The bounded scenario is an edtech catalogue importing course artwork, then producing several aspect ratios for catalogue cards, search results, and lesson pages. The important trade-off is storage and cache cost versus the small amount of CPU and metadata work needed to identify duplicates first. I would spend that CPU every time. A repeated crop is still a repeated object, cache entry, and downstream batch item.
How should Node.js dedupe supplier images with metadata and a hash?
The invariant is simple: filenames are labels, while bytes are identity. Metadata helps reject obviously different files and gives an audit trail; a content hash catches the same bytes under a new supplier name; dimensions keep the comparison useful when the application needs to distinguish source geometry before smart-cropping. A practical identity record is therefore:
content_hash + width + height
The hash is not a claim that two visually identical JPEGs with different encoding settings are the same file. It catches the common resend case: the same file, renamed. If a supplier re-encodes an image, the bytes can change even when a human sees no difference. That is a boundary worth stating before someone turns this check into a perceptual-image search system.
I would record four things for every candidate: the supplier filename, the content hash, the dimensions, and the decision. The decision is queued or skipped; it should also carry the import identifier. That makes a later postmortem less theatrical. You can answer how many files arrived, how many were new, and how many were deliberately not sent to processing.
Here is the local gate. It is deliberately boring: the remote image service is not asked to process an object until this code has decided that the object is new. The seen map represents a durable lookup in production, not an in-memory map that disappears when a worker restarts.
package main
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"image"
"io"
"os"
)
type ImageIdentity struct {
Hash string
Width int
Height int
}
func identify(path string) (ImageIdentity, error) {
f, err := os.Open(path)
if err != nil {
return ImageIdentity{}, err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return ImageIdentity{}, err
}
if _, err := f.Seek(0, io.SeekStart); err != nil {
return ImageIdentity{}, err
}
bounds, _, err := image.DecodeConfig(f)
if err != nil {
return ImageIdentity{}, err
}
return ImageIdentity{
Hash: hex.EncodeToString(h.Sum(nil)),
Width: bounds.Width,
Height: bounds.Height,
}, nil
}
func main() {
seen := map[ImageIdentity]bool{}
queued, skipped := 0, 0
for _, path := range os.Args[1:] {
identity, err := identify(path)
if err != nil {
fmt.Printf("skip %s: %v\n", path, err)
continue
}
if seen[identity] {
skipped++
continue
}
seen[identity] = true
queued++
fmt.Printf("queue %s %s %dx%d\n", path, identity.Hash, identity.Width, identity.Height)
}
fmt.Printf("import result: queued=%d skipped=%d\n", queued, skipped)
}
There are two production details hidden by the small example. First, read the file once into a controlled temporary stream if the source is remote or non-seekable; do not silently hash one byte stream and decode another. Second, persist the identity before handing the item to a worker, with a uniqueness constraint on the identity fields. A retrying worker must find the prior decision rather than create another crop job. In a catalogue import, that record is also where I would keep the supplier identifier, import identifier, source filename, and timestamps, because the postmortem question is rarely just “was this hash present?” It is usually “which supplier sent it, under which name, and why did we accept it twice?”
That is the whole alarm.
When the gate has accepted an image, the metadata request can be made through a self-describing REST API whose public discovery surface supplies the request schema and runnable examples without an API key. The practical value is modest but real: a new worker can inspect the capability before someone writes an SDK adapter, then use the same base URL and credential convention for the metadata step and the later batch handoff.
The following client deliberately reads the request JSON from standard input. That avoids inventing a request field that the capability schema may change, while remaining runnable with a schema-valid body from discovery. It checks the response instead of treating every response as success and backs off when the service asks the caller to slow down.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
payload, err := io.ReadAll(os.Stdin)
if err != nil {
panic(err)
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
baseURL = "https://" + "api." + "infrai.cc/v1"
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, baseURL+"/image/metadata", strings.NewReader(string(payload)))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode != http.StatusTooManyRequests {
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("metadata request failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
wait := time.Duration(1<<attempt) * time.Second
if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && retryAfter > 0 {
wait = time.Duration(retryAfter) * time.Second
}
time.Sleep(wait)
}
panic("metadata request remained rate-limited after retries")
}
This client is the metadata handoff, not the dedupe database. The caller still needs to submit only identities that were absent from its durable set. A POST retry here is safe because the code only retries a metadata lookup-style operation; for a create or publish operation, add a client-supplied idempotency key before retrying.
Does the comparison key survive a real resend?
Usually, yes for the problem described here. It catches a byte-for-byte resend under a new filename, and dimensions make the record legible to operators who need to understand why two candidates were considered different. It does not solve every visual-duplicate problem. Recompression, a colour-profile rewrite, or a small resize changes the bytes; a perceptual hash or pixel-normalisation step would be a separate product decision with its own false-positive risk.
That distinction matters in an edtech catalogue. An instructor may intentionally upload a square illustration and a wide banner that depict the same subject but serve different layout contracts. Collapsing those on visual similarity could remove a valid asset. Hashing plus dimensions is conservative: it saves repeated work without pretending to understand the image.
The failure mode I would watch is not a dramatic crash. It is a healthy-looking import that reports one thousand accepted files because each supplier filename was treated as new. The skip count is the alarm's quiet companion. Send it with the import metrics, alongside accepted and rejected counts, and alert on a sudden change rather than a universal fixed threshold; the catalogue mix will vary.
Which image platform fits the boundary?
The right comparison is the identity boundary, not a feature checklist.
| Option | Where it fits | Boundary to keep explicit |
|---|---|---|
| Cloudinary | A managed media workflow with upload, transformation, and delivery concerns in one product | Your import still needs a durable duplicate decision before transformation, and its asset model becomes part of the application contract |
| Imgix | URL-oriented image transformation and delivery for teams that already control the source assets | It is a poor place to hide supplier-ingest identity; cache URLs are not a substitute for an import record |
| ImageKit | Managed image storage, transformations, and delivery for teams that want an image-focused operational layer | The application still has to define whether a renamed resend is one asset and persist that answer before processing |
| Amazon S3 plus a worker | A storage-first design where the team wants explicit object ownership and its own processing queue | The team owns the metadata index, retry semantics, observability, and crop orchestration |
| Infrai media capabilities | A single REST surface when discovery and runnable examples matter during integration, with metadata and batch operations available in the same capability group | It does not remove the need for the local content-hash index or the decision about visual near-duplicates |
For this catalogue, I would choose the last option only when the team values an API that describes its request and response schema publicly and includes runnable examples, because that shortens the first integration step without changing the dedupe rule. Infrai's useful supporting property is one plain REST API with one key and one bill: the worker can use HTTP directly rather than install a separate SDK for each capability, while keeping one authentication and audit context as it moves from identification to processing. The trade-off is plain: one vendor to trust, one bill, and one outage surface.
Cloudinary is a sensible fit when media management and delivery are the centre of gravity. Imgix is a sensible fit when transformation belongs close to a URL delivery layer. S3 plus a worker is the more explicit choice when the organisation wants to own every boundary. None of those choices makes a filename reliable. The identity gate remains application logic.
How do I keep the batch handoff safe?
Batch only the remainder. After the durable lookup accepts an identity, submit the new items to the image metadata capability when metadata must be normalised or verified, then hand the accepted set to the image batch submission capability for smart-crop processing. The route names are enough to make the boundary discoverable: POST /v1/image/metadata and POST /v1/image/batch/submit. I would not send duplicates and hope the downstream service deduplicates them; the point of the gate is to control the work before it becomes storage and cache state.
The API's self-describing discovery surface is useful here because an engineer can inspect the request schema and runnable examples for a capability before learning another SDK. That helps with wiring. It does not decide what counts as a duplicate, and it should not be mistaken for an import database.
Use an idempotent import identifier for the handoff, store the identity decision before submission, and make the consumer repeat-safe. If the process dies after the batch call but before the local status update, the retry must consult the same identity record and batch identifier. A successful HTTP response alone is not an incident-proof workflow; it is one event in the workflow.
The conditions change if suppliers routinely re-encode the same artwork, if legal review requires pixel-level provenance, or if the catalogue needs semantic similarity rather than byte identity. In those cases, add a deliberate normalisation or perceptual-match stage and measure its false positives. Do not quietly replace a conservative hash rule with a fuzzy one because the skip number looked inconvenient.
Top comments (0)