Store each original and each derived preview as separate private objects, resize asynchronously in a bounded Node.js worker, and authorize the final object with a short-lived signed URL. For healthtech reports, the deciding constraint is large-file throughput: the API must never become the place where a multi-page upload, image decoding, transformation, and customer download all compete for memory.
TL;DR: treat a preview as a durable derivative, not a request-time effect. A database row ties the authenticated customer and report to opaque object keys. A job carries keys and an immutable recipe version. The worker streams the source into a constrained Sharp transform, writes the result under a new key, and records success only after storage confirms the write. The delivery API checks authorization before signing that exact derivative. Watch queue age, processing duration, bytes, failures, and process memory. Those signals tell you whether the pipeline is keeping up.
The picture in words is short: customer -> application authorization -> signed preview URL -> private object storage. Off the request path: upload -> private original -> bounded queue -> resize worker -> private preview. Keep those two paths separate. It makes both security and capacity easier to reason about.
Replace the synchronous mental model
The tempting design is one route that fetches an original, calls resize(), buffers the result, and sends it to the browser. It looks efficient because there are few moving parts. Under concurrent large reports, though, the route owns network reads, decoded pixels, encoding work, and response delivery at the same time. A small HTTP timeout can also trigger another attempt while the first transform is still consuming resources.
The before model is "a thumbnail is a response." The after model is "a preview is a versioned object produced by a job." That shift creates a clean contract. The web process authenticates. The queue absorbs bursts. Workers enforce concurrency. Object storage handles delivery after the application grants access.
Do not put customer email addresses, patient identifiers, report titles, or other sensitive data in object keys. Use opaque identifiers, and keep the ownership relationship in application data. A useful logical record contains customerId, reportId, sourceKey, previewKey, recipeVersion, and status. An object key is a locator, not proof of permission.
A signed URL is also not an ownership check. It is a time-limited capability for one object and operation. Authenticate the caller, load the report, verify that the caller may read it, then issue the URL. Log the authorization decision and object identifier, but avoid logging the full URL because its query string contains credentials.
A bounded TypeScript worker
Here is the core pattern. The interfaces deliberately hide the storage and queue implementation. The important pieces are portable: stream input, restrict accepted formats, cap output dimensions, write to a new key, and make retries converge on the same result. All code is TypeScript.
import sharp from "sharp";
import { pipeline } from "node:stream/promises";
import { PassThrough, Readable } from "node:stream";
type PreviewJob = {
reportId: string;
sourceKey: string;
previewKey: string;
recipeVersion: 3;
};
type ObjectStore = {
read(key: string): Promise<Readable>;
write(
key: string,
body: Readable,
options: { contentType: "image/webp"; ifAbsent: boolean },
): Promise<void>;
};
type JobMetrics = {
durationMs(value: number, labels: { outcome: "ok" | "error" }): void;
inputBytes(value: number): void;
outputBytes(value: number): void;
};
export async function createPreview(
job: PreviewJob,
store: ObjectStore,
metrics: JobMetrics,
): Promise<void> {
const startedAt = performance.now();
let outcome: "ok" | "error" = "error";
try {
const source = await store.read(job.sourceKey);
let inputBytes = 0;
let outputBytes = 0;
source.on("data", (chunk: Buffer) => {
inputBytes += chunk.length;
});
const transform = sharp({
failOn: "warning",
limitInputPixels: 40_000_000,
})
.rotate()
.resize({
width: 1200,
height: 1200,
fit: "inside",
withoutEnlargement: true,
})
.webp({ quality: 78 });
const countedOutput = new PassThrough();
countedOutput.on("data", (chunk: Buffer) => {
outputBytes += chunk.length;
});
const upload = store.write(job.previewKey, countedOutput, {
contentType: "image/webp",
ifAbsent: true,
});
await Promise.all([pipeline(source, transform, countedOutput), upload]);
metrics.inputBytes(inputBytes);
metrics.outputBytes(outputBytes);
outcome = "ok";
} finally {
metrics.durationMs(performance.now() - startedAt, { outcome });
}
}
limitInputPixels is a decode guard, not a file-size limit. Compressed input can expand dramatically when decoded, so validate the upload separately and isolate transformation work from the web process. Sharp documents that its operations use libvips and that most operations can run with a small memory footprint, but your service still needs explicit concurrency and resource limits. A stream reduces whole-file buffering; it does not make memory usage zero.
The deterministic previewKey should include the source object's immutable identity plus recipeVersion. That makes a retry target the same object. ifAbsent expresses the desired idempotency contract: if another attempt already produced that key, the worker should verify or accept the completed derivative according to the storage adapter's documented conditional-write behavior. Never overwrite an old recipe in place. Readers with a previously signed link should continue receiving the bytes that link named.
Backpressure first.
Keep worker concurrency boring and explicit. Start with a fixed per-process limit derived from the worker's CPU and memory allocation. Increase it only while observing resident memory, event-loop delay, transform latency, and queue age under representative files. No invented magic number survives different input dimensions and codecs.
Observe the queue, not just the HTTP route
A 200 from the upload endpoint says almost nothing about preview readiness. The useful service-level story begins after enqueueing. Instrument the transition from accepted to available, then split that time into queue delay and processing duration. Avoid high-cardinality metric labels such as customerId, reportId, or object key. Put those identifiers in structured logs and traces, subject to the system's data-handling policy. Metrics answer "is the fleet unhealthy?" Logs answer "what happened to this job?" A trace can connect the upload request, enqueue operation, worker span, storage write, and later authorization check without turning every identifier into a time series. The alert should follow customer impact. Queue depth alone is ambiguous because a large but fresh burst may drain normally. Oldest-job age crossing the preview-readiness objective is sharper. Pair it with a low-throughput or high-failure condition so the page says what kind of response is likely needed. Retries need classification, too. A transient storage timeout can be retried with capped exponential backoff and jitter. An unsupported input or pixel-limit rejection will not improve on attempt five; record a terminal state immediately. This distinction prevents poison jobs from consuming the capacity needed by valid customer reports.
I would put five signals on the first dashboard:
- Oldest ready-job age, which exposes a backlog even when workers are still completing jobs.
- Processing duration by outcome and recipe version.
- Input and output byte distributions, plus decoded-pixel rejection counts.
- Attempts and terminal failures, with a bounded dead-letter path for investigation.
- Worker resident memory and event-loop delay alongside active-job count.
Measure both paths.
How should object storage handle image thumbnail resizing and signed delivery?
Keep the bucket private and make the application the policy decision point. A download flow should accept a report identifier, derive the preview key from trusted data, verify the authenticated customer's access, and ask the storage adapter for a signed GET URL with a short expiry. The browser then downloads directly from object storage. The application does not proxy the bytes.
type Report = {
id: string;
customerId: string;
previewKey: string;
previewStatus: "pending" | "available" | "failed";
};
type PreviewDependencies = {
findReport(id: string): Promise<Report | null>;
signRead(key: string, expiresInSeconds: number): Promise<string>;
};
export async function authorizePreview(
reportId: string,
authenticatedCustomerId: string,
deps: PreviewDependencies,
): Promise<{ status: 200; url: string } | { status: 404 | 409 }> {
const report = await deps.findReport(reportId);
// One response for missing and unauthorized records avoids disclosing existence.
if (!report || report.customerId !== authenticatedCustomerId) {
return { status: 404 };
}
if (report.previewStatus !== "available") {
return { status: 409 };
}
return {
status: 200,
url: await deps.signRead(report.previewKey, 60),
};
}
Sixty seconds here is an example policy input, not a universal recommendation. Set expiry from the time a client realistically needs to start the transfer, the sensitivity of the object, and the consequences of the URL being shared. Expiry does not revoke a download already completed, and a bearer URL can be used by anyone who obtains it while it remains valid. Keep it out of analytics payloads, referrer-bearing pages, support screenshots, and routine logs.
For large derivatives, test expiry behavior with slow clients and the storage implementation's documented semantics. Also test HTTP range requests if your viewer relies on them; RFC 9110 defines byte ranges, but support and signature requirements belong to the chosen storage adapter. This is exactly the kind of boundary that deserves an integration test rather than an assumption.
What about on-demand resizing and cache hits?
This pattern has limits. It is a poor fit when callers can request arbitrary dimensions, previews are rarely opened, originals change constantly, or the team cannot operate a queue and worker fleet. In those cases, choose an on-demand transformer behind strict dimension limits and a cache, or skip derivatives when the client can safely render the original. The trade-off is direct: pre-generation spends storage and write work for predictable reads; on-demand resizing spends compute on cold reads and needs protection against repeated or adversarial variants.
On-demand transformation moves work closer to reads, so a cold request pays authorization, fetch, decode, encode, and write costs. In a report portal with a small, known preview set, asynchronous generation makes readiness explicit and keeps customer download latency independent of resize time. That is a throughput choice, not a universal rule.
Pre-generation also makes failure visible before the customer opens the report. The UI can represent pending, available, and failed states without holding an HTTP connection open. A retry updates job state; it does not ask a person to refresh until a hidden transform happens to succeed.
Cache semantics still matter. Immutable, versioned preview keys allow long-lived caches without serving a newly changed recipe under an old name. The signed URL may change while the underlying object identity stays stable. Decide whether a shared CDN is permitted for the data classification, and verify its cache-key and authorization behavior. A private origin does not automatically make every downstream cache private.
Do lifecycle rules remove the need for cleanup jobs?
No. Lifecycle rules are useful for age-based transitions and expiration, but application state knows facts that storage does not: a report was deleted, a recipe was superseded, or a legal hold applies. Use both layers with clear ownership. The application should enqueue deletion or retention changes when business state changes. A periodic reconciler should find orphaned originals, derivatives, and database rows. Storage lifecycle rules provide a final age-based boundary for eligible objects.
AWS documents that lifecycle configuration can transition or expire objects and that rules can filter by prefix, tag, size, or combinations. Treat those capabilities as one implementation example, not as the domain model. Your neutral storage interface should expose the retention behavior the application needs, and deployment tests should verify how the selected implementation handles versioned objects, incomplete multipart uploads, and deletion markers.
There is one more operational trap: deleting the database row first destroys the map needed to find object keys. Mark the report for deletion, enqueue object cleanup, confirm the result, and only then remove or tombstone the mapping according to retention policy. Keep audit events free of report contents and signed credentials.
The final decision rule is compact. Precompute a bounded set of health-report previews when predictable read latency and controlled large-file throughput matter. Stream bytes, cap concurrency, version recipes, authorize before signing, and operate the system from queue age rather than HTTP success alone. Every arrow in the diagram now has one job.
References
- https://sharp.pixelplumbing.com/api-constructor
- https://sharp.pixelplumbing.com/api-resize
- https://nodejs.org/api/stream.html
- https://www.rfc-editor.org/rfc/rfc9110.html#name-range-requests
- https://owasp.org/www-community/vulnerabilities/Unrestricted_File_Upload
- https://opentelemetry.io/docs/concepts/signals/
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
Top comments (0)