Here's a failure I've seen more than once on upload-heavy products: the endpoint returns 200, the user closes the tab, and twenty minutes later there's no order in the system because a worker choked on a vendor's proprietary container format. Nobody told the user. Nobody told the worker's author either, because the retry policy quietly redelivered the file until it hit a dead-letter queue nobody monitored.
The standard advice — "put a queue in front of it" — doesn't prevent any of this. The queue was never the hard part.
After building ingestion for dental CAD files, construction models, bulk carbon data, and business documents, I've settled on a different frame: a file pipeline is four stages with different latency budgets and different failure semantics. Throughput is decided by how early you reject bad input and how much derived work leaves the request path — not by which broker you pick.
Four stages, not one queue
The four stages are gate, ingest, extract, derive.
- Gate is synchronous and cheap: auth, quota, size, extension, magic-byte sniffing. It runs while the user is still watching and its budget is milliseconds.
- Ingest moves bytes into object storage and writes the metadata record. It should touch your API process as little as possible.
- Extract is asynchronous, per-format parsing: opening the container, pulling metadata, validating structure. This is where the real engineering lives.
- Derive produces the artifacts your product actually reads — thumbnails, previews, normalized records, optimized model formats. Also async, and unlike extract, usually cacheable and re-runnable.
The reason to name the stages is that the boundaries between them are where you retry, cache, and shed load. A parse failure shouldn't re-run the upload. A thumbnail failure shouldn't re-run the parse. When everything is one "process file" job, every retry repeats all the work and every failure is opaque.
On the Carbon Management Solution project — bulk emissions data, where a user uploads a single file containing any amount of data that then gets validated and processed — we ran this on serverless AWS with Lambda, SNS, SQS, and DynamoDB, structured so that a certain service is responsible for a certain stage of the process. Stage isolation was the design, not an accident of the stack.
Honest caveat: service-per-stage multiplies your operational surface. Four services means four sets of alarms, four deploy pipelines, four things to reason about at 2 a.m. At low volume, one worker pool consuming from a couple of queues is the right call. Draw the stage boundaries in code first; split the infrastructure when a stage actually needs independent scaling.
Validation is a two-tier problem
The all-async reflex — "accept everything, validate in the worker" — is how you get the silent-failure story from the intro. But the opposite reflex, deep validation in the request handler, is how upload endpoints time out. Validation is two tiers, and the split is not negotiable.
What has to fail while the user is still watching
Anything a user can fix by picking a different file must fail synchronously: wrong extension, oversized file, empty file, exhausted quota, a MIME type that doesn't match the bytes. If the user learns about these hours later from an email, your gate failed at its one job.
The gate checks are cheap because they read a few bytes, not the structure:
import { fileTypeFromBuffer } from "file-type";
const ALLOWED = new Set(["model/stl", "application/zip", "application/pdf"]);
const MAX_BYTES = 500 * 1024 * 1024;
export async function gate(head: Buffer, declaredSize: number, user: User) {
if (declaredSize === 0 || declaredSize > MAX_BYTES) {
throw new UploadError("size_rejected");
}
if (await quotaExceeded(user, declaredSize)) {
throw new UploadError("quota_exceeded");
}
// Sniff real content type from magic bytes — never trust the filename
const sniffed = await fileTypeFromBuffer(head);
if (!sniffed || !ALLOWED.has(sniffed.mime)) {
throw new UploadError("type_rejected");
}
}
The line to look at is the sniffing: content type comes from the bytes, not the filename. Renamed .exe files and mislabeled archives are a when, not an if.
On BEGO, a German dental CAD/CAM company, we went further and put scenario detection in the order wizard itself: the upload step auto-detects whether the user dropped 3Shape output, exocad output, standalone STLs, or a mixed set, and routes the flow accordingly. That's still gate-tier work — it reads signatures and file lists, not full structure — and it means the user is corrected before the order exists, not after.
What can only fail inside a worker
Structural correctness of a real-world format is unknowable until you parse it. Whether a proprietary archive decrypts, whether the XML inside references parts that exist, whether an STL mesh is watertight — none of that fits a request budget, and none of it should.
So the contract with the user changes shape: the synchronous response means "your file is accepted for processing," never "your file is valid." The pipeline then needs a way to deliver bad news — a per-file status the UI polls or subscribes to, flipping from processing to failed with a reason a human can act on. On BEGO, the order platform tracks per-file status through queue processing for exactly this reason.
The failure mode to design against: sync checks can be spoofed and deep checks can't be rushed. Any team that tries to collapse the two tiers into one ends up with either a slow gate or a lying one.
Metadata extraction is format work disguised as infrastructure
Generic pipeline tutorials treat "extract metadata" as one box in the diagram. In practice it's the majority of the engineering, because it's per-format parsing, and formats are hostile.
On BEGO the extract stage deep-parses laboratory formats — .3ox, .dentalProject, .constructionInfo, .modelInfo— plus encrypted .bego/.begostl archives with decryption and retry logic, pulling out patient and order metadata, materials, colors, and tooth mapping. None of that generalizes. Every one of those parsers is bespoke work against a vendor format that ships no public spec and changes without notice.
On MemoMeister, a document-management SaaS we built from scratch for the German market, extraction went a step further into classification: workers analyzed the keywords, data formats, and other characteristics of an upload to determine the document type and route it to the right user or group. The file's type came from its content, not its name.
Two lessons from shipping that kind of extraction. First, content-derived classification is probabilistic — you will guess wrong, so the design must include a manual-correction path, not just a confidence score. Second, budget for vendor formats changing silently underneath you; a parser that worked for a year is not a parser that works.
This is why "which queue should I use" is usually the wrong first question. The broker is a commodity. The .dentalProject parser is not.
Get bytes out of your API process
One flat infrastructure rule: file bytes should not flow through your API. The client uploads directly to object storage; the API's job is to issue the upload authorization, write the metadata record, and enqueue the job.
BEGO uploads via pre-signed URLs to cloud storage. MemoMeister stored documents on Amazon S3, with workers picking files up after a successful upload. The pattern is the same either way: the API hands out a scoped, expiring write permission and steps out of the data path.
The shape of it, with a commit step:
// 1. API issues the upload — after gate checks, including quota
const key = uploads/${user.id}/${randomUUID()};
const url = await getSignedUrl(
s3,
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
ContentLength: declaredSize, // cap what the URL can write
}),
{ expiresIn: 300 },
);
await db.file.create({ key, userId: user.id, status: "pending_upload" });
// 2. Client PUTs bytes straight to storage
// 3. Client confirms; API verifies the object exists, then enqueues
export async function commit(key: string) {
const head = await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));
await db.file.update({ key }, { status: "uploaded", etag: head.ETag });
await queue.send({ key, etag: head.ETag });
}
The part to look at is step 3. Without an explicit commit, you have no moment at which the upload is known-complete, and your extract stage starts firing on half-written objects.
Be honest about what this pattern costs: validation now happens after the write. A client can obtain a URL and never call commit, so you need a sweeper for orphaned objects. Quota has to be enforced before the URL is issued, on the declared size, and re-checked at commit. Direct-to-storage upload is still the right default — it just isn't free.
The queue decisions that actually change throughput
Now the section everyone expects, argued honestly: broker choice matters far less than the policies around it.
BEGO runs its workflow processing on Redis-backed queues under NestJS and Kubernetes. MemoMeister ran RabbitMQ, with some workers calling third-party services like Google Vision and others monitoring database changes to update document state. Carbon Management Solution used SNS and SQS. Three stacks, one set of decisions that mattered:
Queue per stage, not one queue for everything. Extraction backing up shouldn't delay thumbnail generation for files already parsed. Separate queues give you separate backpressure and separate scaling knobs — and keep CPU-bound parsing workers apart from IO-bound workers waiting on third-party APIs, which want completely different concurrency settings.
Visibility timeout longer than your worst file. On SQS-style brokers, a message picked up but not deleted becomes visible again after the timeout. Set it shorter than your slowest legitimate parse and you get two workers processing the same giant file, doubling load exactly when the system is already struggling.
A dead-letter queue you actually watch. Bounded retries, then park the message. An unmonitored DLQ is the silent-failure story again, one hop removed.
One warning on MemoMeister's DB-watching workers: they worked, but they couple the pipeline to your schema — every migration becomes a pipeline change. I'd reach for explicit events first and treat DB-watching as a legacy-integration tool.
Idempotency, retries, and poison files
Standard queues are at-least-once delivery: the broker guarantees a message arrives, not that it arrives once. Operationally that means every worker you write will eventually run twice for the same file. Plan for it or debug it.
The fix is idempotency keyed on the stored object — bucket, key, and ETag — checked before the expensive work:
async function handle(msg: { key: string; etag: string }) {
const claimed = await db.extraction.tryClaim(msg.key, msg.etag); // unique index
if (!claimed) return; // duplicate delivery or a lost race — done
const meta = await parse(msg.key); // the expensive part
await db.extraction.complete(msg.key, msg.etag, meta);
}
Keying on the ETag rather than just the key means a re-uploaded file with new bytes is new work, while a redelivered message for the same bytes is a no-op.
Then there are poison files — the input that kills the worker itself. The classic is the model that OOMs the process: the worker dies before it can nack or delete, the visibility timeout expires, the message comes back, and the next worker dies too. Your fleet is now a crash loop with one file as the ignition. Defenses are boring and essential: a max-receive count routing to the DLQ, memory limits that fail the job instead of the pod, and a size ceiling in the gate so the 9 GB file never enters the pipeline at all.
Derived artifacts are the product
Here's the second assumption worth breaking: the pipeline doesn't exist to store files. It exists to produce cheap-to-read outputs. If the artifact your product serves is still expensive to consume, you optimized the wrong half.
BEGO generates STL thumbnails at extract time so the order UI never touches a mesh. And the clearest evidence I have is Vitus, a Danish constructability platform handling .ifc, .nwd, .rvt, .3dm, and other CAD formats for construction companies like Munck, Femern A/S, VINCI, and COWI. Their viewer struggled not because ingestion was slow, but because the format being loaded was heavy.
The fix was a bundle of read-side changes — migrating models to SVF2, parallel loading, web workers off the main thread, list virtualization, IndexedDB caching. Together they cut model size by 5x and loading time by roughly 2.5x, with multi-model loading 5x faster. To be precise about what those numbers are: client-side model-loading gains from the whole bundle of changes, not a server-side pipeline benchmark, and no single change gets the credit. But that's exactly the point — the wins came from changing the data format and the loading strategy. Worker count had nothing to do with it.
The tax on derived artifacts: they need versioning, and a format migration is a backfill, not a deploy. When Vitus moved to SVF2, every existing model needed re-derivation. Stamp every artifact with the version of the deriver that produced it from day one, so "re-derive everything older than v3" is a query and a queue-fill instead of a forensic project.
Constraints that beat the ideal design
Everything above assumes you get to choose where processing runs. Sometimes you don't, and residency, encryption, and hardware outrank performance.
MemoMeister was restricted to third-party services that stored and processed data within Germany. That single compliance line forced a self-hosted OnlyOffice deployment instead of a managed document service — an architectural decision made by a regulation, not a benchmark.
Visbion is the sharper example: their Image Cube compresses and encrypts DICOM 3.0 medical images on dedicated routing hardware inside mobile scanning trailers, serving NHS Breast Screening Services as the UK's largest installed base of dedicated imaging routing hardware. The heaviest processing happens at the edge, on a device in a trailer, before any cloud pipeline sees a byte — because bandwidth from a mobile trailer and encryption requirements for medical imaging say so. Our work there was consulting on fleet-deployment automation, compression and encryption refinement, and a modular redesign — tuning that constraint-driven architecture, not replacing it with a textbook one.
The trade is real: edge and self-hosted processing give up elasticity and easy observability for compliance and bandwidth. Check these constraints before drawing the architecture, because they don't negotiate afterwards.
Where to spend the effort, in order
If I'm starting an ingestion system today, the order is: a cheap synchronous gate that rejects everything a user can fix on the spot; bytes going straight to object storage with an explicit commit; idempotent workers with a DLQ someone watches; and only then worker scaling and broker tuning. The per-format parsers will take longer than all of the infrastructure combined — budget accordingly.
Reach for the full four-stage split when files are big, formats are hostile, and derived artifacts are what users actually consume. Skip the ceremony when you're thumbnailing avatars — a gate check and one background job is a complete architecture at that size.
And check the read side last, every time. If the artifact coming out of the pipeline is still expensive to consume, no amount of queue work will save you — you've optimized the half of the system your users never touch.
Top comments (0)