Scaling Spring Batch across multiple nodes usually means remote partitioning over a message broker like Kafka or RabbitMQ. That works well, but a broker is another tier to run, secure and monitor. Many teams don't want one just to fan out batch jobs.
Here's the thing: Spring Batch already requires a shared relational database for its job repository. So what if that database did the coordination too?
I wrote about this approach for InfoQ. The core ideas:
Size work to the live cluster. Before splitting a job, the master checks which nodes are actually alive and sizes the partitions to match, instead of using a fixed grid size.
No standing coordinator. Whichever node launches a job becomes the master for that run only. Losing a master affects one job, not the whole cluster, and there's no leader election.
Preassigned partitions, safe claiming. Each partition is assigned to one node upfront. Workers claim their partitions with a guarded compare-and-set UPDATE, so a partition is never run twice by accident or silently lost.
Two-phase failover. Nodes heartbeat to the database. A silent node is first marked unreachable, then removed after a grace period, and its safe-to-rerun partitions are reassigned. A short GC pause doesn't trigger a failover.
Minimal code changes. It plugs into Spring Batch's existing Partitioner and PartitionHandler extension points. Your worker steps stay exactly the same.
When to use it (and when not to)
This fits small to medium clusters where the database is already well managed and operational simplicity matters. If you need very large clusters or sub-second coordination, a broker is still the better choice. The article covers these trade-offs in detail.
📖 Full article on InfoQ: https://www.infoq.com/articles/spring-batch-without-broker/
💻 Open-source implementation (peer-reviewed in the Journal of Open Source Software): https://github.com/jchejarla/spring-batch-db-cluster-partitioning
I'd love to hear how others handle distributed batch processing. Have you run Spring Batch at scale with or without a broker?
Top comments (0)