Designing a File Conversion Pipeline: Lessons from Building One

In the middle of the night, a batch job that had run cleanly for three weeks decided to eat 40 PDFs and spit out nothing. No error in the logs, no stack trace, just... silence. That's the night I stopped treating file conversion as a "quick utility" and started treating it as actual system design.

If you've ever built something that converts files, whether it's images, documents, or data formats, you know the pitch sounds simple: take file A, turn it into file B. The reality is closer to building a small, opinionated compiler that has to deal with garbage input, half-broken encodings, and users who will absolutely upload a 200MB file named final_final_v3(1).PDF and expect it to just work.

Here's what I actually learned building a conversion pipeline, past the "just use a library" stage.

The naive version breaks fast

The first version of almost every conversion tool looks like this:

async function convertFile(inputPath, outputFormat) {
  const buffer = await fs.readFile(inputPath);
  const result = await converter.convert(buffer, outputFormat);
  return result;
}

This works in a demo. It works for your first ten test files. Then someone uploads a corrupted PDF, or a 300-page scanned document, or a file that's technically a .docx but was renamed from .zip by a confused user, and your single function has to somehow handle all of that gracefully. It won't, because it wasn't designed to.

The real lesson: conversion isn't one operation. It's a pipeline of distinct stages, and each stage fails in its own specific way.

Breaking it into stages actually matters

Once I stopped writing one big function and started thinking in stages, debugging got dramatically easier. A reasonable pipeline looks something like:
Validate → Parse → Transform → Render → Verify → Deliver

Each stage has a narrow job and a narrow failure mode.

Validation catches the obvious stuff before you waste compute: wrong MIME type, file too large, password-protected when you don't support that, zero-byte upload. This sounds trivial until you realize a huge share of "conversion failed" support tickets are actually validation failures that slipped through.

function validateInput(file) {
  if (file.size === 0) throw new ConversionError('EMPTY_FILE');
  if (file.size > MAX_SIZE) throw new ConversionError('FILE_TOO_LARGE');
  if (!isSupportedMimeType(file.mimeType)) {
    throw new ConversionError('UNSUPPORTED_FORMAT');
  }
}

Parsing is where most of the real complexity lives, especially with PDFs. A PDF isn't really "one format"; it's a container that can hold vector graphics, embedded fonts, scanned raster images, form fields, and metadata, all mixed. Parsing a PDF for text extraction is a completely different problem than parsing it to preserve table layout, which is a different problem again from parsing it for image extraction. Treating "parse the PDF" as a single step is where many tools quietly cut corners, which is exactly why formatting breaks or tables collapse into a wall of text after conversion.

Transform is the actual conversion logic, mapping the parsed structure to the target format's model. This is where you decide what to do when the source format supports something the target doesn't. Word docs support tracked changes; plain text doesn't. You have to make an explicit decision (flatten, strip, warn) rather than let the library decide silently.

Verify is the stage almost everyone skips, and it's the one that would've saved me from that 2 a.m. incident. A conversion that completes without throwing an error isn't the same as a conversion that succeeded. Did the output file actually open? Does it have the expected page count, roughly the expected size, and non-zero content? A cheap sanity check here catches a huge class of silent failures.

async function verifyOutput(outputPath, expectedMinSize) {
  const stats = await fs.stat(outputPath);
  if (stats.size < expectedMinSize) {
    throw new ConversionError('OUTPUT_TOO_SMALL', { size: stats.size });
  }
}

Queue it, don't block it

The other thing that changes everything: conversion is almost never instant, and it shouldn't block a request-response cycle. Once you're handling anything beyond tiny files, you want a job queue, not a synchronous endpoint.

app.post('/convert', async (req, res) => {
  const jobId = await queue.add('convert', {
    fileId: req.body.fileId,
    targetFormat: req.body.targetFormat,
  });
  res.json({ jobId, status: 'queued' });
});

This buys you retries, backpressure when load spikes, and the ability to actually see where a job is stuck instead of a request just timing out. It also means one huge file doesn't take down conversions for everyone else hitting your service at the same time, which is a real failure mode if you're running this on a single worker process.

Idempotency saves you from yourself

If a worker crashes mid-conversion and your queue retries the job, does running it twice cause a problem? If you're writing partial output to a shared path, the answer is often "yes, and it's ugly." Designing each job to be safely retryable, writing to a temp path first and only moving to the final location on success, is a small habit that prevents a lot of 3 a.m. pages.

Format edge cases are the real work

Here's the part nobody tells you upfront: the "conversion logic" itself is maybe 30% of the work. The other 70% is edge cases. Fonts that don't embed properly and fall back to something ugly. Hyperlinks that survive a PDF-to-Word conversion structurally but lose their href. Tables that render fine visually in the source but have no real row/column structure underneath, so any converter has to guess. Scanned PDFs that need OCR before there's even text to convert.

None of this shows up until you throw real-world files at the system, which is exactly why "it works on my test files" and "it works in production" are such different claims.

This is also, honestly, the reason I keep a browser-based tool bookmarked for quick one-off conversions rather than running everything through my own pipeline every time. When I just need a PDF turned into a clean Word file or a JPG without spinning up a job, I use PDF Conveter instead of reinventing that particular wheel for a single file. It's a decent reminder, too: building your own pipeline makes sense when conversion is core to your product, but for a one-off task, borrowing a tool that's already handled these edge cases is the more sensible engineering call.

What I'd tell my past version

Design for failure before you design for the happy path. Log at each stage boundary, not just at the top level, so you can tell whether a job died in parsing versus rendering versus delivery. And build your verify step early, because "no error thrown" is a very low bar for "the user got what they needed."