DEV Community: Janardhan Chejarla

Distributed Spring Batch Coordination, Part 7: Best Practices for Production

Janardhan Chejarla — Sun, 03 Aug 2025 13:27:10 +0000

🚀 Introduction

As you prepare to take your distributed Spring Batch jobs into production using the database-backed coordination framework, it’s critical to establish robust operational practices. This article highlights key recommendations for configuring, monitoring, and managing distributed job executions reliably and efficiently at scale.

⚙️ Configuration Best Practices

✅ Use Static Node IDs in Production

📝 While dynamic UUIDs (e.g., worker-${{random.uuid}}) are useful for local testing, static node IDs (like worker-1, worker-2) are preferred in production.

This ensures:

Clear visibility into node health
Easier debugging and traceability
Consistent partition reassignment logic

📅 Tune Heartbeat and Failure Detection Intervals

Configure the following properties carefully in your YAML:

spring:
  batch:
    heartbeat-interval: 5000
    unreachable-node-threshold: 15000
    node-cleanup-threshold: 30000

heartbeat-interval: Frequency at which nodes update their status.
unreachable-node-threshold: Marks nodes as UNREACHABLE if no update is received.
node-cleanup-threshold: Deletes truly failed nodes after grace period.

Choose these values based on your workload and network reliability.

🔁 Enable Task Reassignment Safely

When defining a ClusterAwarePartitioner, explicitly set:

@Override
public PartitionTransferableProp arePartitionsTransferableWhenNodeFailed() {
    return PartitionTransferableProp.YES;
}

This allows for automatic reassignment of unfinished tasks to active nodes, improving fault recovery.

📝 Note: Set PartitionTransferableProp.YES with caution. Not all tasks are safe to transfer upon failure—especially those involving file I/O, partial state updates, or external system interactions. Ensure your partitioned step is idempotent and can be re-executed without side effects before enabling this.

📡 Observability and Monitoring

🩺 Use Built-in Health Indicators

Spring Boot Actuator exposes two indicators:

/actuator/health → shows batchCluster and batchClusterNode
/actuator/batch-cluster → detailed view of all active nodes and their load

Example snippet:

"batchCluster": {
  "status": "UP",
  "details": {
    "Total Active Nodes": "3",
    "Total Nodes in Cluster": "3"
  }
}

Integrate these with Prometheus, Datadog, or any other monitoring tool.

📊 Track Load Per Node

Use /actuator/batch-cluster to determine:

Which node is handling how many tasks
Status (ACTIVE, UNREACHABLE)
Heartbeat freshness

This can help in rebalancing strategies and horizontal scaling decisions.

🛡️ Fault Tolerance Tips

🚨 Plan for Network Glitches

Configure timeouts with a grace period to avoid false positives from brief network issues.

🧠 Node Self-Recovery

If a node recovers after being deleted (e.g., due to latency), it can re-register and participate again.

📁 Job Design Tips

🔗 Keep Partition Logic Simple and Stateless

Avoid embedding heavy logic or dependencies in your Partitioner implementation. It should rely on basic parameters like row ranges, record offsets, or identifiers.

🧩 Isolate Shared Resources

When writing to shared output (e.g., XML files or databases), ensure:

Thread safety
Separate output files/directories per partition
Avoid overwrites and race conditions

🧭 Final Thoughts

By combining stateless partitioning logic, lightweight DB coordination, and robust monitoring, this framework enables large-scale batch execution with minimal operational overhead.

These best practices help ensure your distributed Spring Batch jobs are resilient, traceable, and ready for production.

⭐️ Support the Project

If you found this article series useful or are using the framework in your projects, please consider giving the repository a ⭐️ on GitHub:

👉 GitHub – spring-batch-db-cluster-partitioning

Your feedback, issues, and contributions are welcome!

Distributed Spring Batch Coordination, Part 6: Developer Control and Partitioning Strategies

Janardhan Chejarla — Sun, 03 Aug 2025 13:26:56 +0000

🔍 Introduction

In any distributed system, failure is not an exception—it’s the norm. Nodes may crash, lose network connectivity, or experience high latency. A resilient Spring Batch framework must gracefully detect and recover from such failures, ensuring consistency and progress without manual intervention.

This part of the series focuses on how failure detection and retry logic work in the database-backed Spring Batch coordination model.

💥 Why Failure Handling Matters

Distributed job execution is susceptible to:

💻 Node crashes or shutdowns
🌐 Network issues causing delayed heartbeats
⏳ Transient processing failures (e.g., DB timeout, file lock)
🔁 Unacknowledged or abandoned partitions

Without an intelligent retry mechanism and failure awareness, these issues can lead to:

Stalled jobs
Incomplete processing
Duplicate processing due to incorrect reassignment

🧠 Node Failure Detection

The framework monitors node health using periodic heartbeats recorded in the BATCH_NODES table. A two-step failure detection process is used:

Mark as UNREACHABLE: If a node fails to update its heartbeat within a configurable time window (spring.batch.cluster.unreachableNodeThreshold), other nodes mark it as UNREACHABLE.
Eviction: If the node remains stale beyond the configured threshold (spring.batch.cluster.nodeCleanupThreshold), it's evicted and deleted from the cluster registry.

This gives temporarily slow nodes time to recover, reducing false positives.

🔄 Reassignment of Tasks

When a node is marked as unreachable, all active partitions associated with that node are checked for reassignment.

The decision is governed by the master step’s partitioner:

return new ClusterAwarePartitioner() {

    @Override
    public List<ExecutionContext> createDistributedPartitions(int availableNodeCount) {
        // Developer-defined logic to split workload
        // (e.g., based on row ranges, file segments, or other domain inputs)
        return generatePartitions(availableNodeCount);
    }

    @Override
    public PartitionTransferableProp arePartitionsTransferableWhenNodeFailed() {
        return PartitionTransferableProp.YES; // or NO, based on task sensitivity
    }

    @Override
    public PartitionStrategy buildPartitionStrategy() {
        return PartitionStrategy.builder()
            .partitioningMode(PartitioningMode.ROUND_ROBIN)
            .build();
    }
};

If this PartitionTransferableProp flag is set to YES, the framework reassigns those partitions to healthy nodes based on the configured strategy (e.g., round-robin, fixed-node).

✅ What’s Next (Part 7 Preview)

In the next part, we’ll cover:

📋 Best practices for running in production
🔐 Securing coordination and metadata tables
🧪 Performance tuning for large-scale workflows
🚀 Integrating with monitoring stacks (Grafana, Prometheus)

👨‍💻 About the Author

Janardhan Reddy Chejarla is a Lead Software Engineer specializing in distributed systems and batch processing frameworks. He is the author of spring-batch-db-cluster-partitioning and a contributor to multiple open-source initiatives.

⭐️ Star the GitHub repo and stay tuned for Part 7 → Production-Grade Best Practices!

Distributed Spring Batch Coordination, Part 5: Monitoring, Observability, and Health Checks

Janardhan Chejarla — Sun, 03 Aug 2025 12:47:02 +0000

🔍 Introduction

As distributed jobs scale across nodes, observability becomes essential. In this part, we explore how the spring-batch-db-cluster-partitioning framework exposes real-time cluster health, task loads, and node statuses — giving developers the visibility needed for debugging, performance tuning, and production readiness.

📊 Exposing Cluster State with Actuator Endpoints

Spring Boot Actuator endpoints provide a natural interface to expose cluster state. This framework adds two custom indicators:

✅ `/actuator/health`

This includes:

"batchCluster": {
  "status": "UP",
  "details": {
    "Total Active Nodes": "3",
    "Total Nodes in Cluster": "3"
  }
},
"batchClusterNode": {
  "status": "UP",
  "details": {
    "Current Load (number of live tasks)": "2",
    "Node Status": "ACTIVE",
    "Node Id": "worker-1",
    "Last Heartbeat Update Time": "2025-08-03T02:43:07.133+00:00",
    "Start Time": "2025-08-03T02:42:32.092+00:00"
  }
}

This helps you instantly verify:

Overall cluster health, how many nodes available in total.
If the current node is active and responsive
How many tasks it's executing
When it last sent a heartbeat

🌐 `/actuator/batch-cluster` – Full Cluster Snapshot

This custom endpoint provides a full view of the entire cluster state:

{
  "nodes": [
    {
      "Started At": "2025-08-03T02:42:32.092+00:00",
      "Node Id": "worker-1",
      "Current Load (# of tasks)": 2,
      "Host": "worker-1.company.local",
      "Last Heartbeat": "2025-08-03T02:43:07.133+00:00",
      "Status": "ACTIVE"
    }
  ],
  "totalNodes": 3
}

It includes:

All registered nodes
Current task count
Last heartbeat timestamp
Node status (ACTIVE, UNREACHABLE, etc.)

📈 Node Load Metrics

Cluster load is computed based on:

Active tasks being executed per node
Heartbeat freshness
Task reassignment if a node becomes unreachable

This allows external monitoring tools to:

Detect load imbalance
Alert on stale heartbeats
Audit execution trends over time

📌 Best Practices

📦 Use static node IDs (e.g., worker-1, worker-2) for easier observability
🛠️ Integrate with Prometheus/Grafana using custom endpoints or intermediate exporters
🧪 Monitor node health to detect failures before partitions get stuck

✅ What’s Next (Part 6 Preview)

In the next part, we’ll cover:

⚠️ Failure handling and retries
🧯 What happens when a node crashes mid-job
🔄 How tasks are reassigned or resumed
🧠 Retry strategies to ensure data consistency

⭐ Want More?

Explore the code: GitHub – spring-batch-db-cluster-partitioning
Read earlier parts in the series: Dev.to article series

Distributed Spring Batch Coordination, Part 4: Getting Started with the Framework

Janardhan Chejarla — Sun, 03 Aug 2025 12:30:55 +0000

In this part of the series, we'll walk through how to set up and run a distributed Spring Batch job using the open-source database-backed coordination framework we introduced earlier.

✅ By the end of this guide, you’ll have a working multi-node Spring Batch cluster that can process large datasets in parallel — with no messaging queues or external coordination middleware required.

📦 Step 1: Add the Framework Dependency

First, include the published artifact from Maven Central.

<dependency>
  <groupId>io.github.jchejarla</groupId>
  <artifactId> spring-batch-db-cluster-core</artifactId>
  <version>2.0.0</version>
</dependency>

<!-- XML Writing Support -->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-oxm</artifactId>
</dependency>

<dependency>
  <groupId>com.thoughtworks.xstream</groupId>
  <artifactId>xstream</artifactId>
  <version>1.4.21</version>
</dependency>

Make sure you're using Spring Batch 5+ and Spring Boot 3+.

🧠 Step 2: Configure Your Cluster

Each node in the cluster is a regular Spring Boot application that shares access to the same relational database (PostgreSQL, Oracle, or MySQL).

In your application.yml, configure node identity and coordination settings:

spring:
   batch:
      enabled: true
      node-id: worker-${random.uuid} 
      grid-size: 4 # This is 
      heartbeat-interval: 5000
      task-polling-interval: 2000

📝 Note: While using worker-${random.uuid} helps during local testing, it's recommended to assign static node IDs (e.g., worker-1, worker-2, etc.) in production environments. This improves observability, debugging, and partition tracking across runs.

Make sure all nodes point to the same database and include the required tables.

⚠️ Important: This framework extends Spring Batch — so make sure to initialize the core Spring Batch schema before applying the coordination schema.

You can find the official Spring Batch schema files here.

Then, execute the coordination framework's schema available here.

📝 Note: We currently support Oracle, MySQL, PostgreSQL, and H2 (for testing purposes).

If you need support for another database, please open an issue on the Feature Request page.

⚙️ Step-by-Step: CSV to XML Job

⚙️ Step 1: Sample CSV File

We’ll use a sample file with 10,000 customer records. You can download the CSV from the GitHub example repo:

📄 customers_example_data.csv

customer_id,first_name,last_name,email,signup_date
C00001,Michael,Miller,michael.miller1@test.net,2023-01-03
C00002,David,Brown,david.brown2@test.net,2023-10-01
...

⚙️ Step 2: Define the Customer POJO

@Setter
@Getter
@ToString
public class Customer {
    private String customerId;
    private String firstName;
    private String lastName;
    private String email;
    private String signupDate;
}

⚙️ Step 3: ItemReader Setup

 @Bean("customerReader")
    @StepScope
    public FlatFileItemReader<Customer> customerReader(
            @Value("#{stepExecutionContext['startRow']}") long startIndex,
            @Value("#{stepExecutionContext['endRow']}") long endIndex,
            @Value("#{jobParameters['inputFile']}") String inputFile,
            LineMapper<Customer> lineMapper) {
        FlatFileItemReader<Customer> reader = new FlatFileItemReader<Customer>() {
            private int currentLine = 0;

            @Override
            public Customer read() throws Exception {
                Customer customer;
                while ((customer = super.read()) != null) {
                    currentLine++;
                    if (currentLine < startIndex) {
                        continue;
                    }
                    if (currentLine > endIndex) {
                        return null;
                    }
                    return customer;
                }
                return null;
            }
        };

        reader.setResource(new FileSystemResource(inputFile));
        reader.setLinesToSkip(1); // Skip header
        reader.setLineMapper(lineMapper);

        return reader;
    }

    @Bean
    public LineMapper<Customer> lineMapper() {
        DefaultLineMapper<Customer> mapper = new DefaultLineMapper<Customer>();
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames("customer_id", "first_name", "last_name", "email", "signup_date");

        BeanWrapperFieldSetMapper<Customer> fieldSetMapper = new BeanWrapperFieldSetMapper<Customer>();
        fieldSetMapper.setTargetType(Customer.class);

        mapper.setLineTokenizer(tokenizer);
        mapper.setFieldSetMapper(fieldSetMapper);
        return mapper;
    }

⚙️ Step 4: Optional Processor

@Bean
public ItemProcessor<Customer, Customer> customerProcessor() {
    return item -> item; // No-op for now
}

⚙️ Step 5: XML Writer

@StepScope
@Bean
public StaxEventItemWriter<Customer> customerXmlWriter(
    @Value("#{jobParameters['outputDir']}") String outputDir,
    @Value("#{stepExecutionContext['partitionId']}") String partitionId
) {
    StaxEventItemWriter<Customer> writer = new StaxEventItemWriter<>();
    writer.setRootTagName("customers");
    writer.setResource(new FileSystemResource(outputDir + "/customers-part-" + partitionId + ".xml"));

    XStreamMarshaller marshaller = new XStreamMarshaller();
    marshaller.setAliases(Map.of("customer", Customer.class));
    writer.setMarshaller(marshaller);

    return writer;
}

⚙️ Step 6: Job Config with Partitioning Strategy

Use round-robin or fixed-node allocation. Each ExecutionContext contains:

  context.putInt("startRow", i * range + 1); // Skip header
  context.putInt("endRow", (i + 1) * range);
  context.putInt("partitionId", i);

Complete Job Config:

import io.github.jchejarla.springbatch.clustering.api.ClusterAwarePartitioner;
import io.github.jchejarla.springbatch.clustering.api.PartitionStrategy;
import io.github.jchejarla.springbatch.clustering.autoconfigure.conditions.ConditionalOnClusterEnabled;
import io.github.jchejarla.springbatch.clustering.partition.ClusterAwarePartitionHandler;
import io.github.jchejarla.springbatch.clustering.partition.PartitionTransferableProp;
import io.github.jchejarla.springbatch.clustering.partition.PartitioningMode;
import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.core.partition.support.StepExecutionAggregator;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.xml.StaxEventItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.PlatformTransactionManager;

import java.util.ArrayList;
import java.util.List;

@Slf4j
@Configuration
@EnableBatchProcessing
@ConditionalOnClusterEnabled
public class ETLJobConfig {

    @Autowired
    private FlatFileItemReader<Customer> customerItemReader;
    @Autowired
    private CustomerProcessor customerProcessor;
    @Autowired
    private StaxEventItemWriter<Customer> customerXmlWriter;

    @Autowired
    @Qualifier("etlJobPartitioner")
    private Partitioner partitioner;
    @Autowired
    private ClusterAwarePartitionHandler partitionHandler;

    @Bean("etlClusteredJob")
    public Job clusteredJob(JobRepository jobRepository, PlatformTransactionManager platformTransactionManager, @Qualifier("multiStepAggregator") StepExecutionAggregator clusterAwareAggregator) {
        return new JobBuilder("etl-clustered-job", jobRepository)
                .incrementer(new RunIdIncrementer())
                .preventRestart()
                .start(multiNodeExecutionStep(jobRepository, platformTransactionManager, clusterAwareAggregator))
                .build();
    }

    public Step multiNodeExecutionStep(JobRepository jobRepository, PlatformTransactionManager txnManager, StepExecutionAggregator clusterAwareAggregator) {
        return new StepBuilder("etlStep.manager", jobRepository)
                .partitioner("multiNodeExecStep", partitioner)
                .partitionHandler(partitionHandler)
                .step(etlReaderWriterStep(jobRepository, txnManager))
                .aggregator(clusterAwareAggregator)
                .build();
    }

    @Bean
    public Step etlReaderWriterStep(JobRepository jobRepository, PlatformTransactionManager txnManager) {
        return new StepBuilder("etlReaderWriterStep", jobRepository).<Customer, Customer>chunk(100, txnManager)
                .reader(customerItemReader)
                .processor(customerProcessor)
                .writer(customerXmlWriter)
                .build();
    }

    @Bean("etlJobPartitioner")
    @StepScope
    public Partitioner etlJobPartitioner(@Value("#{jobParameters['rows']}") Integer rows) {
        return new ClusterAwarePartitioner() {

            @Override
            public List<ExecutionContext> createDistributedPartitions(int availableNodeCount) {
                int range = rows / availableNodeCount;
                List<ExecutionContext> partitions = new ArrayList<>(availableNodeCount);

                for (int i = 0; i < availableNodeCount; i++) {
                    ExecutionContext context = new ExecutionContext();
                    context.putInt("startRow", i * range + 1); // Skip header
                    context.putInt("endRow", (i + 1) * range);
                    context.putInt("partitionId", i);
                    partitions.add(context);
                }

                return partitions;
            }

            @Override
            public PartitionTransferableProp arePartitionsTransferableWhenNodeFailed() {
                return PartitionTransferableProp.YES;
            }

            @Override
            public PartitionStrategy buildPartitionStrategy() {
                return PartitionStrategy.builder().partitioningMode(PartitioningMode.ROUND_ROBIN).build();
            }
        };

    }
}

⚙️ Step 7: Launching Job with parameters

JobParameters parameters = new JobParametersBuilder()
                .addString("RUN_TIME", LocalDateTime.now().toString(), true)
                .addLong("rows", rows)
                .addString("inputFile", ETL_JOB_INPUT_FILE)
                .addString("outputDir", ETL_JOB_OUTPUT_DIR)
                .toJobParameters();
        Job job = applicationContext.getBean("etlClusteredJob", Job.class);
try {
      JobExecution jobExecution = jobLauncher.run(job, parameters);
     } catch(Exception e) {
         throw new RuntimeException("Exception occurred when launching the Job", e);
     }

✅ What’s Next (Part 5 Preview)

In the next part, we’ll cover:

📈 Monitoring batch executions (e.g., via /actuator/batch-cluster)
⚠️ Failure handling and retries
📊 Best practices for large-scale ingestion
🚀 Future roadmap (e.g., metrics, better dashboards)

Distributed Spring Batch Coordination – Part 3: Fault Tolerance and Node Recovery

Janardhan Chejarla — Sun, 03 Aug 2025 12:30:44 +0000

In this part of the series, we’ll dive into how the framework ensures reliability and resilience by detecting failures and automatically rebalancing the workload across a dynamic cluster of nodes. With no centralized scheduler or messaging broker dependency, node health and task recovery are entirely database-coordinated, lightweight, and highly configurable.

🔥 Failure Happens — Let’s Handle It Gracefully

In traditional Spring Batch setups, node failure often leads to partial execution or requires manual intervention. With this coordination framework, failure is a first-class citizen, and node recovery is built-in.

The system uses a two-step failure detection mechanism:

✅ Step 1: Detecting Unreachable Nodes

Every active node updates its timestamp in the BATCH_NODES table at regular intervals (heartbeat).

If a node fails to update within a configurable timeout (e.g., 30 seconds), it is marked as UNREACHABLE.

This state gives the node time to recover from temporary issues like GC pauses or transient network glitches.

✅ Step 2: Deregistering Stale Nodes

If the node remains unreachable beyond a second threshold (e.g., 2–3 minutes), it is considered stale and removed from the coordination table.

At this point, any tasks (partitions) it was executing become eligible for reassignment — but only if they were marked as is_transferrable = true when created. This allows for fine-grained control over which partitions can move between nodes and which are sticky by design.

🔄 Node Rejoin Logic

If the node recovers after being marked unreachable or deregistered, it can:

Re-register itself into the BATCH_NODES table
Participate in future partition assignment rounds
Remain fully stateless from a master node’s perspective

🧠 Masterless but Intelligent Coordination

This design avoids a single point of failure by decentralizing the intelligence:

The node that launches the job acts as the temporary “master” for coordination
Other nodes participate in a shared decision-making protocol using the database
No Zookeeper, Redis, or Kafka needed

📊 Built-in Observability in v2.0.0

This version includes Actuator-based monitoring endpoints:

/actuator/health:
- batchCluster: shows total and active nodes
- batchClusterNode: current node ID, load, heartbeat, status, uptime
/actuator/batch-cluster:
- Lists all registered nodes with status, load, heartbeat timestamps, and host info

This gives real-time visibility into the state of the cluster without requiring any additional dashboards or monitoring tools.

💡 What's Next?

In Part 4, we’ll walk through how to build a real-world distributed Spring Batch job using this coordination framework. You'll learn:

How to integrate the framework into your Spring Boot project using Maven
How to launch a multi-node cluster and observe active node discovery
How round-robin and fixed-node partitioning work in practice
How task distribution is calculated dynamically based on the current cluster size

We'll demonstrate this with a practical ETL example, where the master node splits a large CSV file into logical partitions (e.g., by line number or ID range), and each worker node processes its assigned chunk by transforming it into XML.

This simulates real-world scenarios like customer data migration or financial transaction reporting, helping you confidently apply the framework to production-scale workloads.

Stay tuned — the next part is hands-on and code-heavy!

Distributed Spring Batch Coordination, Part 2: How Database-Backed Partitioning Works

Janardhan Chejarla — Sun, 03 Aug 2025 12:30:23 +0000

📘 Part 2: How Database-Backed Partitioning Works

In Part 1, we discussed the challenges with traditional Spring Batch scaling — especially when relying on Kafka or RabbitMQ. In this part, let’s explore how we can simplify distributed coordination using a relational database as the central source of truth.

💡 Key Idea

Rather than broadcasting partition instructions via messaging middleware, the master node writes coordination state into the database. Worker nodes read from the database to discover which partitions they are responsible for — no messaging layer required.

⚙️ Core Coordination Tables

This model relies on three lightweight tables:

BATCH_NODES: Registers active nodes in the cluster
BATCH_JOB_COORDINATION: Tracks coordination for each partitioned step
BATCH_PARTITIONS: Stores partition metadata and execution state (assigned node, status, result)

These tables allow for real-time visibility into job execution without external queues or in-memory state.

🔁 Execution Flow

Master node receives the job request
It queries BATCH_NODES to find all currently active nodes
Using either:
- 🌀 Round-Robin, or
- 🎯 Fixed-Node allocation the master assigns partitions and stores them in BATCH_PARTITIONS
Workers poll for tasks where assigned_node = self
Once complete, they update their partition status
The master monitors for completion and performs final aggregation, if needed

✅ This works with any JDBC-compatible database and integrates well with Kubernetes, CI/CD workflows, or container-based clusters.

📊 Visual Overview

🧠 Why This Works

This architecture avoids common pitfalls like:

Tight coupling to messaging infrastructure
Delayed worker availability
Lack of visibility into node state
Hard-to-debug coordination failures

Instead, all coordination is transparent, queryable, and resilient to restarts.

🏁 What’s Next

In the next part of the series, we’ll cover:

🔹 Failure handling strategies
🔹 Retry logic
🔹 Node liveness and rebalancing

📘 Read Part 1 here

🌟 GitHub Repo: jchejarla/spring-batch-db-cluster-partitioning

Distributed Spring Batch Coordination, Part 1: The Problem with Traditional Spring Batch Scaling

Janardhan Chejarla — Sat, 02 Aug 2025 18:19:38 +0000

📘 Part 1: The Problem with Traditional Spring Batch Scaling

Scaling Spring Batch across multiple nodes typically involves complex setup — often requiring messaging middleware like RabbitMQ or Apache Kafka to enable remote partitioning. While effective, this approach introduces tight infrastructure coupling, deployment overhead, and runtime fragility.

Here’s the crux of the problem:

You need message brokers to send partition instructions to remote workers.
The master has no real visibility into how many workers are available at job launch.
Late-arriving or unavailable nodes may lead to skewed partitioning, failures, or idle workers.
The coordination state lives in memory or is dispersed across the cluster.

For modern, container-based workloads, this makes orchestration harder — especially when trying to run Spring Batch inside Kubernetes, CI/CD workflows, or ephemeral cloud environments.

🔧 Why I Built This Project

To simplify this, I built an open-source framework:

➡️ spring-batch-db-cluster-partitioning

This framework replaces message brokers with a relational database as the central coordination hub. It supports:

✅ Round-robin and fixed-node partition assignment
✅ Dynamic node discovery before the job starts
✅ Fully stateless master logic, with all orchestration handled via SQL

It’s lightweight, easy to plug into your Spring Batch step, and ready to run in Docker, Kubernetes, or CI pipelines.

🧠 Design Note: Master Node Uptime

While the coordination model is stateless and database-driven, the node that initiates a job acts as its master for the duration of execution. This node:

Launches the job and assigns partitions
Monitors worker progress and failures
Executes final aggregation or post-partition steps, if any
Writes final job completion status

To preserve job integrity, the master node must remain available while the job is running. However, since coordination state is fully persisted in the database, this node can be any eligible participant — making the model decentralized, resilient, and cloud-native in spirit.

📚 What You’ll Learn in This Series

In the coming parts of this series, I’ll walk you through:

The architecture and coordination flow
Partitioning strategies (round-robin, fixed-node)
Failure handling and node resilience
How to build and run distributed jobs with this framework
A real-world ETL use case (CSV to XML conversion) to demonstrate end-to-end job orchestration

If you’ve ever struggled with Spring Batch scaling or want a more DevOps-friendly model, this series is for you.

🔜 Coming Up Next:

Part 2 – How Database-Backed Partitioning Works in Spring Batch

Stay tuned — and ⭐️ the repo if you're excited:

👉 github.com/jchejarla/spring-batch-db-cluster-partitioning

Released v1.0.0 of a Spring Batch clustering library — pure DB-based coordination, no Kafka, no ZooKeeper. Feedback welcome!

Janardhan Chejarla — Tue, 08 Jul 2025 02:15:23 +0000

Janardhan Chejarla

Jul 8 '25

Spring Batch Clustering With Zero Messaging: Introducing spring-batch-db-cluster-partitioning (v1.0.0)

#springboot #springbatch #java #opensource

Comments 1

3 min read

Spring Batch Clustering With Zero Messaging: Introducing spring-batch-db-cluster-partitioning (v1.0.0)

Janardhan Chejarla — Tue, 08 Jul 2025 02:09:02 +0000

⚡ TL;DR

A production-ready Spring Batch extension for dynamic clustering using only the database — no Kafka, no RabbitMQ, no coordination servers. Version 1.0.0 is now released on Maven Central!

🚀 Why I Built This
Distributed batch processing often requires complex infrastructure — think messaging systems, zookeeper, or centralized schedulers. In many real-world deployments, especially in financial institutions, this adds cost, risk, and tight operational coupling.

To solve this, I built a lightweight, pluggable Spring Batch extension that enables cluster-aware partitioned step execution using nothing but a shared database as the coordination layer.

🛠️ Key Features

✅ Cluster Coordination Using DB Only
Each node registers, heartbeats, and participates in job execution using lightweight database tables — no brokers required.

✅ Dynamic Partition Assignment
Partitions are assigned and rebalanced at runtime based on node availability — perfect for ephemeral cloud-native deployments.

✅ Failover & Recovery
If a node dies mid-job, remaining nodes detect the loss and reassign unfinished partitions.

✅ Pluggable Coordination Tables
Custom schemas (BATCH_NODES, BATCH_PARTITIONS, BATCH_JOB_COORDINATION) provide fine-grained visibility and control.

✅ Zero External Dependencies
Built purely on Spring Batch 5.x, Spring JDBC, and standard transaction semantics.

📦 Released Artifacts

<dependency>
  <groupId>io.github.jchejarla</groupId>
  <artifactId>clustering-core</artifactId>
  <version>1.0.0</version>
</dependency>

Available now on Maven Central.

🔍 Architecture Overview

Master Node — Automatically elected (no Zookeeper!) based on job launcher.
Coordinator Tables — Shared DB tables used to track:
- Active nodes
- Step partition states
- Ongoing executions
PartitionHandler — Custom implementation that uses SQL to dynamically assign work.

✅ Example Code: GitHub - examples directory for ready-to-run Spring Boot projects demonstrating the cluster partitioning in action.

💡 Use Cases

Spring Batch jobs in Kubernetes (nodes scale up/down)
FinTech ETL pipelines where messaging systems are overkill
On-prem enterprise environments with restricted tech stacks
Batch workloads on serverless compute (like AWS Fargate or Google Cloud Run)

🧪 Example: Dynamic Step Execution

@Bean
public Step partitionedStep() {
    return stepBuilderFactory.get("partitionedStep")
        .partitioner("workerStep", customPartitioner())
        .partitionHandler(clusterAwarePartitionHandler())
        .build();
}

The clusterAwarePartitionHandler is where the DB magic happens.

📈 What’s Next
🔹 Monitoring endpoints
🔹 Retry policies and smarter failure detection

If you’re interested in how this framework works under the hood, I’ve started a detailed article series:
📘 Distributed Spring Batch Coordination: Part 1 – The Problem with Traditional Scaling

More parts coming soon!

🙌 Contribute or Follow

⭐ GitHub Repo
📦 Maven Central
💬 Issues and PRs welcome — especially around test cases and Kubernetes integrations.

🧠 Behind the Scenes

This project was born from practical needs in building scalable ETL pipelines in a large financial services ecosystem. By removing the messaging layer, we reduced infra cost and simplified failover handling — while maintaining full reliability.

👋 Final Thoughts
If you're tired of spinning up Kafka just to run partitioned jobs — this is for you.

This library is designed for engineers, architects, and teams who want reliability without orchestration overload. Try it out, share feedback, and help shape the roadmap.

DEV Community: Janardhan Chejarla

Distributed Spring Batch Coordination, Part 7: Best Practices for Production

🚀 Introduction

⚙️ Configuration Best Practices

✅ Use Static Node IDs in Production

📅 Tune Heartbeat and Failure Detection Intervals

🔁 Enable Task Reassignment Safely

📡 Observability and Monitoring

🩺 Use Built-in Health Indicators

📊 Track Load Per Node

🛡️ Fault Tolerance Tips

🚨 Plan for Network Glitches

🧠 Node Self-Recovery

📁 Job Design Tips

🔗 Keep Partition Logic Simple and Stateless

🧩 Isolate Shared Resources

🧭 Final Thoughts

⭐️ Support the Project

Distributed Spring Batch Coordination, Part 6: Developer Control and Partitioning Strategies

🔍 Introduction

💥 Why Failure Handling Matters

🧠 Node Failure Detection

🔄 Reassignment of Tasks

✅ What’s Next (Part 7 Preview)

👨‍💻 About the Author

Distributed Spring Batch Coordination, Part 5: Monitoring, Observability, and Health Checks

🔍 Introduction

📊 Exposing Cluster State with Actuator Endpoints

✅ /actuator/health

🌐 /actuator/batch-cluster – Full Cluster Snapshot

📈 Node Load Metrics

📌 Best Practices

✅ What’s Next (Part 6 Preview)

⭐ Want More?

Distributed Spring Batch Coordination, Part 4: Getting Started with the Framework

📦 Step 1: Add the Framework Dependency

🧠 Step 2: Configure Your Cluster

⚙️ Step-by-Step: CSV to XML Job

⚙️ Step 1: Sample CSV File

⚙️ Step 2: Define the Customer POJO

⚙️ Step 3: ItemReader Setup

⚙️ Step 4: Optional Processor

⚙️ Step 5: XML Writer

⚙️ Step 6: Job Config with Partitioning Strategy

⚙️ Step 7: Launching Job with parameters

✅ What’s Next (Part 5 Preview)

Distributed Spring Batch Coordination – Part 3: Fault Tolerance and Node Recovery

🔥 Failure Happens — Let’s Handle It Gracefully

✅ Step 1: Detecting Unreachable Nodes

✅ Step 2: Deregistering Stale Nodes

🔄 Node Rejoin Logic

🧠 Masterless but Intelligent Coordination

📊 Built-in Observability in v2.0.0

💡 What's Next?

Distributed Spring Batch Coordination, Part 2: How Database-Backed Partitioning Works

📘 Part 2: How Database-Backed Partitioning Works

💡 Key Idea

⚙️ Core Coordination Tables

🔁 Execution Flow

📊 Visual Overview

🧠 Why This Works

🏁 What’s Next

Distributed Spring Batch Coordination, Part 1: The Problem with Traditional Spring Batch Scaling

📘 Part 1: The Problem with Traditional Spring Batch Scaling

🔧 Why I Built This Project

🧠 Design Note: Master Node Uptime

📚 What You’ll Learn in This Series

🔜 Coming Up Next:

Released v1.0.0 of a Spring Batch clustering library — pure DB-based coordination, no Kafka, no ZooKeeper. Feedback welcome!

Spring Batch Clustering With Zero Messaging: Introducing spring-batch-db-cluster-partitioning (v1.0.0)

Spring Batch Clustering With Zero Messaging: Introducing spring-batch-db-cluster-partitioning (v1.0.0)

✅ `/actuator/health`

🌐 `/actuator/batch-cluster` – Full Cluster Snapshot