When more concurrency broke my bulk workflow
I increased concurrency to speed up a high-volume bulk workflow.
At first, it looked like the right move. Smaller runs got faster, throughput improved, and the pipeline seemed healthier.
Then larger runs started failing with PrematureCloseException.
That was the moment I realized the problem was no longer just performance. It had become a system pressure problem.
A few lessons from the debugging journey:
- more parallelism does not always mean more throughput
- chunk size is not just a batch setting — it becomes a stability boundary
- retries only help after the concurrency model is sane
- connection pool behavior matters a lot more under load
- partial-failure handling makes bulk workflows much more trustworthy
What finally helped was not one magic fix. It was a combination of:
- reducing unsafe parallelism
- tuning chunk size more carefully
- adding retry with backoff
- stabilizing connection pool behavior
- treating concurrency as a budget instead of a goal
I wrote the full debugging story here:
https://medium.com/p/758f87e312d5
Curious how others handle this kind of issue in bulk or async workflows.
Top comments (2)
This hits hard. Most people only realize this after things start breaking.
The “concurrency as a budget” point is key. Treating it like a dial to max out is what causes these failures.
One thing I’ve seen help in similar bulk workflows is adding backpressure at the application level instead of relying only on retries or pool tuning.
Basically slowing intake when downstream starts struggling.
Curious, did you try any form of rate limiting or adaptive concurrency control during this?
Really good point.
I didn’t go as far as adaptive concurrency control in this implementation. Most of the improvement came from manual tuning: lowering concurrency, tuning chunk size, and adding retry/backoff.
But I agree — backpressure at the application layer would be a smarter next evolution, so the system can react before downstream instability shows up as actual failures.