When more concurrency broke my bulk workflow
I increased concurrency to speed up a high-volume bulk workflow.
At first, it looked like the right move. Smaller runs got faster, throughput improved, and the pipeline seemed healthier.
Then larger runs started failing with PrematureCloseException.
That was the moment I realized the problem was no longer just performance. It had become a system pressure problem.
A few lessons from the debugging journey:
- more parallelism does not always mean more throughput
- chunk size is not just a batch setting — it becomes a stability boundary
- retries only help after the concurrency model is sane
- connection pool behavior matters a lot more under load
- partial-failure handling makes bulk workflows much more trustworthy
What finally helped was not one magic fix. It was a combination of:
- reducing unsafe parallelism
- tuning chunk size more carefully
- adding retry with backoff
- stabilizing connection pool behavior
- treating concurrency as a budget instead of a goal
I wrote the full debugging story here:
https://medium.com/p/758f87e312d5
Curious how others handle this kind of issue in bulk or async workflows.
Top comments (6)
This hits hard. Most people only realize this after things start breaking.
The “concurrency as a budget” point is key. Treating it like a dial to max out is what causes these failures.
One thing I’ve seen help in similar bulk workflows is adding backpressure at the application level instead of relying only on retries or pool tuning.
Basically slowing intake when downstream starts struggling.
Curious, did you try any form of rate limiting or adaptive concurrency control during this?
Really good point.
I didn’t go as far as adaptive concurrency control in this implementation. Most of the improvement came from manual tuning: lowering concurrency, tuning chunk size, and adding retry/backoff.
But I agree — backpressure at the application layer would be a smarter next evolution, so the system can react before downstream instability shows up as actual failures.
Yeah, that makes sense. Getting it stable first is the hard part.
Backpressure would be interesting here, especially something simple like limiting in-flight requests per downstream instead of global concurrency.
I’m planning to try this in a bulk file workflow this weekend to see how early it can prevent those failures instead of reacting after.
Would love to compare notes if you experiment with it too.
That’s a really good idea.
Per-downstream in-flight limits sound much cleaner than treating concurrency as one global number, especially when different downstreams have very different tolerance under load.
I’d be interested to see how your experiment behaves in practice — especially whether it helps catch pressure early enough before retries and failures start stacking up.
Would definitely be happy to compare notes once you’ve tested it.
Makes sense. I’ll keep it simple first.
Planning to test a per-downstream in-flight cap + basic queueing, nothing adaptive yet. Just want to see if it reduces failure spikes early instead of relying on retries.
If it works, next step would be making it dynamic based on error rate or latency.
Will share what I see once I run it on the file workflow this weekend.
This is very close to what I’ve been seeing while load testing a file workflow (FiloraFS-Lite).
When I ramped RPM + concurrency, things looked fine initially — then failure spikes started showing up instead of linear throughput gains.
From the logs:
It felt less like “performance tuning” and more like hitting a pressure limit.
I haven’t implemented backpressure yet, but I’m planning to test a simple per-downstream in-flight cap next (basically treating concurrency as a budget like you mentioned).
Curious — in your case, did latency signals show up before PrematureCloseException started happening?