From 16 Hours to Minutes: The Hard Part of Multithreading Isn't the Threads

Rami — Tue, 22 Sep 2026 18:19:55 +0000

Multithreading is easy when every piece of work is independent.

Throw the work into a queue, start a bunch of workers, and let them fight it out.

Unfortunately, many real-world systems aren’t that simple.

We recently looked at modernizing a legacy batch-processing system with a fairly straightforward job:

Read a CSV.
Load some state from a database.
Apply a calculation.
Update the state.
Store the result.

The catch?

A file containing a few hundred thousand rows could take up to 16 hours to process.

Our goal was to scale the architecture toward processing around a million operations in minutes rather than hours.

What looked initially like a performance problem quickly became a much more interesting concurrency problem:

How do you massively parallelize a workload while preserving ordering for related operations?

The giant `for` loop

Conceptually, the legacy system looked something like this:

for row in csv:
    state = database.read(row.item)
    result = calculate(state, row.action)
    database.write(result)

Simple. Predictable. Correct.

And painfully sequential.

Every row involved database access, computation, and another database operation before moving to the next row.

Another complication: the system supported an all-or-nothing processing mode, so a database transaction could remain open for essentially the entire job.

For a large import, that could mean hours.

Before redesigning anything, though, we needed to understand where the time was actually going.

Measure first

Suppose 300,000 operations take 16 hours.

Some rough math gives us:

16 × 60 × 60 × 1000
--------------------  ≈ 192 ms / operation
      300,000

Roughly 200 ms per operation.

We then broke an operation into its major components:

DB Read → Calculation → DB Write

The calculation itself was relatively fast. Database reads were more expensive, and writes were particularly costly.

That immediately exposed two separate opportunities:

Do more calculations concurrently, and stop treating the database as part of every individual operation.

The first one naturally leads to multi-threading.

Just add threads™

Imagine that the total workload represents roughly 16 hours of serial work.

Ignoring overhead for a moment, spreading that work across 32 cores gives us:

16 hours / 32 ≈ 30 minutes

Add more cores and, at least theoretically, we start getting into the range of minutes rather than hours.

Great.

Let’s use 90 workers.

Problem solved.

Except now we’ve broken the system.

The ordering problem

Our CSV isn’t actually a collection of completely independent operations.

Consider this simplified input:

Item-A, Action-1
Item-B, Action-1
Item-C, Action-1
Item-A, Action-2
Item-D, Action-1
Item-A, Action-3

Actions belonging to different items can run concurrently.

But actions belonging to the same item cannot.

Action-2 for Item-A must operate on the state produced by Action-1.

And Action-3 must operate on the state produced by Action-2.

So this is perfectly valid:

Item-A: A1 → A2 → A3
Item-B: B1 → B2
Item-C: C1

while all three item chains execute concurrently.

But this isn’t:

Thread 1: Item-A / Action-1
Thread 2: Item-B / Action-1
Thread 3: Item-A / Action-2

Thread 3 could finish before Thread 1.

Now Action-2 has been calculated against the wrong state.

We’ve made the application dramatically faster at producing incorrect results.

Parallel across keys, sequential within a key

That observation became the fundamental rule of the design:

Operations for different keys can execute in parallel. Operations for the same key must execute sequentially.

In our case, the key was an item ID.

Instead of assigning individual rows directly to arbitrary workers, we introduced a router.

Conceptually:

The CSV feeds a router, which distributes operations among workers 0 through N.

The router reads the input sequentially.

When it encounters an item, it assigns that item to a worker.

All future operations for that item go to the same worker.

Each worker consumes its operations through a FIFO queue.

So if the input contains:

Item-A / Action-1
Item-B / Action-1
Item-A / Action-2
Item-C / Action-1
Item-A / Action-3

the queues might become:

Worker 0:
Item-A / Action-1
Item-A / Action-2
Item-A / Action-3
Worker 1:
Item-B / Action-1
Worker 2:
Item-C / Action-1

We’ve preserved ordering for Item-A without sacrificing concurrency between A, B, and C.

This is essentially keyed partitioning.

Hashing gets us most of the way there

An obvious implementation is:

worker = hash(itemId) % workerCount

It has a very useful property:

The same item always produces the same worker.

Therefore:

hash(Item-A) % N → Worker 3
hash(Item-A) % N → Worker 3
hash(Item-A) % N → Worker 3

Ordering becomes much easier because one worker owns the processing sequence for that item.

But now another problem appears.

Distribution matters.

Suppose four workers receive:

Worker	Relative workload (illustrative)
Worker 0	20
Worker 1	7
Worker 2	5
Worker 3	3

The other three workers can finish and go home.

The entire job is still waiting for Worker 0.

At that point, our execution time isn’t determined by average throughput.

The slowest partition determines it.

Load balancing is part of the algorithm

For workloads where the keys are known during routing, another approach is to explicitly assign newly encountered keys across workers.

For example:

Item-A → Worker 0
Item-B → Worker 1
Item-C → Worker 2
Item-D → Worker 3
Item-E → Worker 0
...

Store that assignment:

Item-A → 0
Item-B → 1
Item-C → 2
Item-D → 3

When Item-A appears again, the router looks up its existing assignment and sends it back to Worker 0.

This preserves our important invariant:

same key → same worker

while allowing us to control the distribution more deliberately.

The resulting architecture is roughly:

Stage	Responsibility
Reader	Read the CSV in order.
Router	Send each key to its assigned worker.
FIFO queues	Preserve arrival order for each worker.
Workers	Process their queues concurrently, one operation at a time per worker.

At this point we have ordered parallelism.

But we have also created another problem.

Congratulations, we just DDoS’d our own database

Imagine 90 workers doing this:

READ
CALCULATE
WRITE
READ
CALCULATE
WRITE
READ
CALCULATE
WRITE
...

against the same database.

A million operations multiplied by database reads and writes, now happening concurrently from dozens of threads.

The application might scale.

The database probably won’t appreciate our enthusiasm.

So the next optimization was more important than adding additional threads:

Remove the database from the hot path.

Move the working state into memory

Instead of loading an item repeatedly, we load its initial state once.

DB → State Cache

The first operation works against that state.

State0 + Action1 → State1

The next operation works against the newly mutated in-memory state.

State1 + Action2 → State2

Then:

State2 + Action3 → State3

The database is no longer the source of truth for every intermediate operation.

During the job, the in-memory state is.

So the worker effectively becomes:

state = cache.get_or_load(item)
for action in ordered_actions:
    state = calculate(state, action)
cache[item] = state

If a million-row file represents only 300,000 unique items, we’ve potentially transformed something approaching a million item reads into roughly 300,000 initial-state reads.

More importantly, we’ve eliminated the need to persist every intermediate mutation.

Multi-threading changed the transaction model

This produced an interesting architectural side effect.

Previously, supporting all-or-nothing behavior required keeping a database transaction open while the job ran.

With the new design, intermediate mutations exist only in memory.

Nothing has been committed yet.

That means after every worker drains its queue, we can decide what to do.

For example:

Strict mode

Any operation failed?
        ↓
Discard everything.

Non-strict mode

Some operations failed?
        ↓
Commit successful final states.

The concurrency architecture therefore solved more than a performance problem.

It gave us a cleaner transaction boundary.

Instead of:

BEGIN TRANSACTION
   ...hours of processing...
COMMIT

we can move toward:

PROCESS EVERYTHING IN MEMORY
        ↓
FINAL DECISION
        ↓
SHORT DATABASE TRANSACTION

That’s a very different relationship with the database.

Write once

At the end of processing, we don’t care about every intermediate version of an item.

We care about its final state.

If an item had ten actions:

State0
  ↓ Action1
State1
  ↓ Action2
State2
  ↓
 ...
  ↓ Action10
State10

only State10 needs to be persisted.

So instead of hundreds of thousands or millions of individual writes, workers produce final states that can be written using a bulk operation.

Depending on the database and requirements, that might mean bulk copy into staging tables followed by a set-based MERGE/UPSERT.

Our architecture has now evolved considerably:

Stage	Responsibility
CSV reader	Read operations in file order.
Router	Assign operations by item key.
FIFO queues and workers	Run concurrently across keys, sequentially within each key.
State cache	Hold the evolving in-memory item states.
Bulk commit	Persist final states to the database.

Read once.
Partition by key.
Process concurrently.
Preserve order within each key.
Mutate in memory.
Write final state in bulk.

The important lesson isn’t “use more threads”

It would be easy to summarize this optimization as:

We replaced a single-threaded process with a multi-threaded one.

But that misses most of the interesting engineering.

Adding threads was probably the easiest part.

The actual design work was figuring out what could safely be parallelized.

We couldn’t simply parallelize rows because rows weren’t independent.

Instead, we had to discover the workload’s natural unit of serialization.

In this case:

Constraint	Requirement
Global ordering	❌ Not required
Per-item ordering	✅ Required
Cross-item parallelism	✅ Allowed

Once that distinction became explicit, the architecture became much clearer.

The problem stopped being:

“How do we process a CSV with 90 threads?”

and became:

“How do we partition an ordered stream into independent sequential streams?”

That’s a much better problem to solve.

Threads expose the next bottleneck

There’s another lesson here.

Performance optimization tends to move bottlenecks rather than eliminate them.

Initially:

Single-threaded processing
        ↓
      BOTTLENECK

Add parallelism:

90 workers
   ↓
Database
   ↓
BOTTLENECK

Reduce database round trips:

90 workers
   ↓
In-memory state
   ↓
Bulk database write

Now CPU becomes much more relevant.

And that’s actually what we wanted.

A CPU-heavy calculation engine is something we can scale horizontally or vertically much more predictably than thousands of tiny database transactions.

The goal wasn’t simply to make everything faster.

It was to move the workload toward resources that scale well.

Great. Now where do we get 90 cores?

At this point, the application can actually use a lot of CPU.

So the next question is:

Where should this thing run?

The workload is bursty. We might have many large jobs during peak periods and none at all at other times.

A few obvious AWS options come to mind.

Lambda

Lambda is attractive because it is fully on-demand.

But this workload wants a large amount of CPU inside one execution, and it may run long enough that function-runtime limits become uncomfortable.

So Lambda isn’t a great fit for this particular shape of workload.

Fargate

Fargate gets us closer.

We can package the processor as a container, run it on demand, and avoid managing servers.

The problem is compute size.

If our application benefits from 64, 96, or more CPUs, a single Fargate task becomes limiting. Splitting the work across multiple containers would also force us to solve distributed state and coordination.

That adds complexity we don’t currently need.

EC2

EC2 gives us exactly what the application wants:

one large machine with lots of CPUs and memory.

That means all workers can live inside one process and share the same in-memory state.

No distributed locks.

No remote cache.

No coordination between machines.

From an application perspective, this is the simplest option.

But we don’t want a 96-core server sitting around

A large EC2 instance is great while a job is running.

It’s not great when there are no jobs.

We could use Auto Scaling Groups, but then we’d have to manage things like:

when to scale up,
when to scale down,
which instance size a job needs,
how jobs wait for capacity,
and how to handle different job sizes.

A small file might need 16 CPUs, and a medium one might need 32, etc.

At that point, we’re starting to build our own batch scheduler.

Which is exactly the problem AWS Batch already solves.

AWS Batch (The Solution)

AWS Batch lets us define the compute environments we want and submit containerized jobs with different CPU and memory requirements.

So the flow becomes:

CSV arrives
    ↓
Determine job size
    ↓
Submit Batch job
    ↓
AWS provisions EC2 capacity
    ↓
Run the container
    ↓
Job finishes
    ↓
EC2 capacity goes away

That fits the workload really well.

The application doesn’t need to know how to launch EC2 instances or manage scaling.

It just says I need this much CPU and memory; run me.

And AWS Batch handles the queueing, scheduling, provisioning, execution, and teardown.

The result is exactly what we wanted:

Small job → small compute
Large job → large compute
No job → no compute

The multi-threaded processor stays simple, while the infrastructure scales around the workload.

And what does all that compute cost?

The nice part about this model is that large compute doesn’t necessarily mean high cost.

We’re not paying for a 96-core machine all day. We only pay while the job is running.

For example, if a 96-core instance costs roughly $6/hour:

$6 / hour
÷ 6
──────────
≈ $1 for a 10-minute job

So we can throw a lot of compute at the problem, finish quickly, and then give the machine back.

And when there are no jobs? No EC2 compute; $0 cost.

That’s one of the more interesting consequences of the architecture: using a bigger machine can actually make economic sense when you only keep it for minutes.

Instead of optimizing for the cheapest server, we’re optimizing for the cost of completing a job.

The pattern is much bigger than CSV processing

None of this is particularly specific to transformations, entities, or even CSV files.

The same pattern appears anywhere you have operations that are:

ordered within a key but independent across keys.

For example:

Customer → ordered account operations
Device → ordered telemetry processing
Order → ordered state transitions
Document → ordered transformations
Portfolio → ordered financial events
Entity → ordered event replay

Whenever you encounter a giant sequential loop, it’s worth asking:

Does the entire workload actually need to be sequential?

Often the answer is no.

Only a much smaller unit requires ordering.

Find that unit, partition around it, and suddenly a sequential workload can become massively parallel without sacrificing correctness.

That’s where multi-threading gets interesting.

Not when you create more threads.

When you figure out where you’re allowed to use them.

DEV Community: Rami

From 16 Hours to Minutes: The Hard Part of Multithreading Isn't the Threads

The giant for loop

Measure first

Just add threads™

The ordering problem

Parallel across keys, sequential within a key

Hashing gets us most of the way there

Load balancing is part of the algorithm

Congratulations, we just DDoS’d our own database

Move the working state into memory

Multi-threading changed the transaction model

Write once

The important lesson isn’t “use more threads”

Threads expose the next bottleneck

Great. Now where do we get 90 cores?

Lambda

Fargate

EC2

But we don’t want a 96-core server sitting around

AWS Batch (The Solution)

And what does all that compute cost?

The pattern is much bigger than CSV processing

The giant `for` loop