#2 Why I Chose BullMQ Instead of Processing Files Inside the API

Uttkarsh singh — Fri, 05 Jun 2026 06:20:03 +0000

While building a file conversion and sharing app, my first implementation handled file conversion directly inside the API request.

It worked.

But as file sizes increased, requests stayed open for hundreds of milliseconds while the server was busy processing files.

The problem wasn't the conversion itself.

The problem was that users were waiting for work that didn't need to happen synchronously.

So I moved file conversion to BullMQ workers backed by Redis.

The flow became:

Upload → Job added to BullMQ → API responds immediately → Worker processes file → Status updated → User receives result

Why BullMQ worked better here:

Faster API responses
Request handling separated from heavy file processing
Better scalability through worker concurrency
Retries and failure handling built into the queue
More predictable system behavior under load

Results from load testing

API p95 latency reduced by 212x (800 ms → 3.8 ms)
Throughput increased by 4.8x (1.3 → 6.2 requests/sec)
Tested with 20 concurrent requests

Tradeoff:

Additional infrastructure (Redis + workers)
More operational complexity
Eventual consistency instead of immediate completion

If every request needed an instant response, synchronous processing might still be the simpler choice.

This was a good reminder that performance improvements often come from moving work out of the critical request path rather than making the work itself faster.

#1 Why I Chose Polling Over WebSockets for File Processing?

Uttkarsh singh — Sun, 24 May 2026 07:06:59 +0000

While building a file conversion and sharing app, I needed a way to show users the status of their file processing.

My initial thought was WebSockets.

Realtime updates sounded like the obvious choice.

But after thinking about the actual requirement, I realized users didn’t need instant push notifications — they only needed occasional feedback about whether conversion had finished.

So I went with polling every 2 seconds.

The flow became:

Upload → Job queued (BullMQ) → Worker processes file → Frontend polls status → Result ready

Why polling worked better here:

Simpler architecture
No persistent connections to manage
No communication layer between workers and socket server
Easier to debug and reason about

Tradeoff:

More HTTP requests
Users continue polling until processing completes
Not suitable for low-latency or collaborative experiences

If the requirements change to things like chat, live collaboration, or instant notifications, I’d revisit WebSockets.

This was a good reminder that architecture decisions depend more on requirements than technology preferences.

DEV Community: Uttkarsh singh

#2 Why I Chose BullMQ Instead of Processing Files Inside the API

#1 Why I Chose Polling Over WebSockets for File Processing?