Building a Content-Addressed Image Cache for a Markdown Publisher

Why this matters

A Markdown publisher often handles the same image more than once. An author reruns a preview, fixes a paragraph, rebuilds a diagram, or restarts an interrupted publication. Re-uploading every unchanged asset wastes network calls and creates unnecessary storage objects. Reusing a URL based only on a filename is worse: the file may have changed while its name stayed the same.

The useful question is narrower than “how do I cache images?”:

How can a publisher reuse an uploaded image only when the processed bytes match, while still recovering when its local cache record is missing?

I traced and exercised the image path in a TypeScript Markdown publisher. Its answer is a layered cache: normalize the image, put a shortened content hash in the object key, check persistent publication state, then ask storage before uploading.

What I built or tested

The pipeline accepts a local article image and returns a delivery URL. For raster input it rotates according to metadata, caps width at 1600 pixels without enlargement, and writes WebP. GIF and SVG input follow a pass-through path that preserves the format.

After processing, the pipeline combines four pieces of identity:

• storage provider;
• article slug;
• normalized source basename;
• a 16-character SHA-256 prefix derived from the processed bytes.

I tested four calls with generated images and local test doubles:

Call	Input and state	Observed result
1	First blue PNG	Uploaded one object and saved one record
2	Same blue PNG	Reused the saved URL; no upload
3	Orange PNG at the same source path	Produced a different URL and uploaded a second object
4	Same orange PNG, fresh database, existing storage	Recovered the storage URL; no third upload

This experiment makes the cache boundary concrete. It is sensitive to processed content, but it is scoped by article and basename rather than being one global content store.

Setup

The experiment used Node.js 22, TypeScript, Sharp, an in-memory database double, and a storage double that counted exists and upload calls. It generated two 40×20 PNG fixtures in a temporary directory and made no network requests.

The production-facing function has a small dependency-injected surface:

await processAndUploadImage({
  sourcePath,
  articleSlug,
  cacheDirectory,
  storage,
  database,
});

Both collaborators are interfaces. The database can find and save an image record by provider and object key. The storage provider can check an object, derive its public URL, or upload a local processed file. That separation lets the cache branches run under deterministic test doubles.

Step-by-step walkthrough

1. Normalize before assigning identity

For PNG, JPEG, and WebP input, the publisher first writes a normalized WebP. It reads the optimized bytes, hashes those bytes, and places the shortened digest in the output filename:

const optimized = await readFile(temporaryPath);
const hash = createHash("sha256")
  .update(optimized)
  .digest("hex")
  .slice(0, 16);
const outputPath = path.join(outputDirectory, `${baseName}-${hash}.webp`);

Hashing after normalization matters. Two source files are considered equivalent only when this pipeline produces the same bytes for them. The cache does not promise semantic image equivalence.

GIF and SVG skip raster conversion. Their original bytes are hashed, copied, and kept under their original extension. That distinction should be visible in tests because an encoder upgrade can affect raster keys without affecting pass-through assets.

2. Build an article-scoped object key

The object key is constructed from the sanitized article slug and processed filename:

const objectKey = `blog/${articleSlug}/${path.basename(optimized.outputPath)}`;

A resulting key has the shape:

blog/cache-example/diagram-<processed-hash>.webp

Keeping the article slug and basename makes storage easy to inspect and prevents unrelated articles from sharing an opaque global namespace. It also means this is not pure global content-addressing: identical bytes under different article slugs or basenames can produce different keys.

3. Use the database as the fast path

The pipeline asks persistent publication state for the provider and object key. It reuses the stored URL only when the stored hash also matches the newly processed hash:

const saved = await database.findImage(storage.name, objectKey);
if (saved?.hash === optimized.hash) return saved.url;

This avoids a storage request on the normal repeat path. In the experiment, the second call returned the first URL without another exists or upload operation.

4. Let storage repair a missing record

A local database can be deleted, replaced, or unavailable to another runner while the uploaded object still exists. Treating a database miss as permission to upload would create redundant writes.

The fallback checks storage first. If the object exists, the pipeline derives its public URL; otherwise it uploads. Either way, it saves the resulting provider, key, hash, and URL back to the database.

The diagram shows the two cache layers: persistent state is the fast path, while object storage is the recovery authority before a new upload.

5. Deduplicate concurrent references separately

Persistent caching does not solve every duplicate. One Markdown document can reference the same source path multiple times while a first upload is still pending. The publication operation therefore keeps a map from validated source path to its in-flight Promise<string>.

That is a separate layer with a separate scope:

• the Promise map deduplicates repeated references during one process;
• the database reuses known URLs across runs;
• the storage existence check recovers when the database does not know the object.

What went wrong

The phrase “content-addressed cache” can hide three constraints in this implementation.

First, a database cache hit happens after image processing. The pipeline must produce the optimized bytes and hash before it can know the object key. A repeat run saves a storage request and upload, but it still pays the file read and Sharp conversion cost.

Second, the digest is shortened to 16 hexadecimal characters. That keeps filenames manageable, but it is not the full SHA-256 digest. The design should not be described as collision-proof.

Third, the key includes the article slug and source basename. That is useful operational scoping, but it means identical processed bytes are not necessarily shared across articles or differently named source files.

The existing focused image test exposed another limitation in verification: it checks conversion, maximum width, non-empty output, and hashed naming, but it does not directly regression-test the database-hit or storage-recovery branches. I used a local experiment to cover those behaviors; a permanent unit test would be the stronger long-term guard.

Fix or mitigation

The immediate mitigation is to describe and test the cache according to what it actually guarantees: reuse of an article-scoped processed object, not global deduplication and not zero-cost reruns.

A reusable implementation checklist is:

1. Normalize bytes before computing the identity used for upload reuse.
2. Include an explicit scope in the object key when operational isolation matters.
3. Match both the stored key and processed hash before returning a cached URL.
4. On a database miss, check storage before uploading.
5. Persist every recovered or uploaded URL before declaring asset preparation successful.
6. Deduplicate in-flight work separately from persistent reuse.
7. Add tests for same-content reuse, changed-content invalidation, and storage recovery.

If CPU cost becomes significant, add a source-level fingerprint and processing-profile version before Sharp runs. That would be a new cache layer, not a replacement for hashing the final uploaded bytes. The version must include settings that affect output, such as dimensions, quality, format, and encoder behavior; otherwise a “fast” hit can return an asset generated under stale rules.

Trade-offs

This design favors inspectable storage layout and safe recovery over maximum deduplication.

• Article and basename scoping can store identical processed bytes more than once.
• Processing before lookup keeps the uploaded-byte identity honest but consumes CPU on repeat runs.
• A database makes normal hits fast, while also becoming state that must be migrated and backed up.
• The storage fallback tolerates missing database records, but it adds a remote existence request on every database miss.
• Short hashes produce practical filenames, while requiring an explicit acceptance of collision risk.

A global content store could deduplicate more aggressively by keying only on a full digest. It would also make ownership, cleanup, privacy boundaries, and article deletion harder to reason about. The right scope depends on whether storage efficiency or per-article isolation is the stronger requirement.

How I verified it

The four-run experiment observed exactly two uploads and two stored objects. The same-content rerun returned the same URL. Changing the source bytes changed the URL. Replacing the database while keeping the storage double recovered the existing URL and saved a new local record without incrementing the upload count.

I also ran the focused image test. It passed and confirmed that an 1800-pixel PNG becomes a non-empty, 1600-pixel-wide WebP whose filename contains a 16-character hexadecimal digest.

Before publication, the TypeScript check passed and the full Vitest suite passed all 12 tests across seven test files. The article validator also rendered the Mermaid diagram and completed a publisher dry run without a network write.

The experiment used generated fixtures and .invalid delivery URLs. It did not claim storage latency, compression savings, or cross-machine performance, and it did not contact a publishing platform.

Conclusion

A reliable publisher needs more than “upload if the filename is new.” It needs a precise identity for the bytes it will deliver and a recovery path for disagreement between local state and remote storage.

Normalize, hash, check persistent state, check storage, then upload. Keep in-flight deduplication as its own layer, and be honest about scope: this implementation caches processed assets inside an article-oriented namespace and saves network work, not all processing work.

That narrower guarantee is still valuable. It makes reruns predictable, changed images visible, and missing local records recoverable without turning every restart into another upload.

AI assistance disclosure

AI assisted with outlining and drafting. Every technical claim was checked against the repository or the recorded local experiment, and the article contains no private paths, credentials, or production data.