Spring Batch: Processing Large Data Volumes Efficiently (2026-09-03 17:31)

Spring Batch: Processing Large Data Volumes

Processing millions of records is a common enterprise requirement—think end-of-day financial reconciliation, ETL pipelines, or bulk data migrations. Spring Batch is a robust framework built precisely for these scenarios, offering transaction management, chunk-based processing, restartability, and scaling primitives out of the box.

In this post, we'll explore how to build and tune Spring Batch jobs for high-volume workloads.

Core Concepts

A Spring Batch Job is composed of one or more Steps. Each chunk-oriented step follows a read-process-write cycle:

• ItemReader — reads one item at a time from a source.
• ItemProcessor — transforms or filters each item (optional).
• ItemWriter — writes items in batches.

The magic for large volumes lies in chunk-oriented processing: items are read one-by-one, accumulated into a chunk, then written together within a single transaction.

A Basic Chunk-Oriented Step

@Bean
public Step processStep(JobRepository jobRepository,
                        PlatformTransactionManager txManager,
                        ItemReader<Transaction> reader,
                        ItemProcessor<Transaction, ReportRow> processor,
                        ItemWriter<ReportRow> writer) {
    return new StepBuilder("processStep", jobRepository)
            .<Transaction, ReportRow>chunk(1000, txManager)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .faultTolerant()
            .skipLimit(100)
            .skip(FlatFileParseException.class)
            .build();
}

Here, the chunk size of 1000 means 1000 items are read and processed in memory, then committed together. Choosing the right chunk size is critical—too small increases transaction overhead, too large increases memory pressure.

Reading Large Datasets

Database Cursor vs. Paging

When reading from a database, avoid loading everything into memory. Spring Batch provides two strategies:

JdbcCursorItemReader streams rows using a single connection and a JDBC cursor:

@Bean
public JdbcCursorItemReader<Transaction> cursorReader(DataSource dataSource) {
    return new JdbcCursorItemReaderBuilder<Transaction>()
            .name("txCursorReader")
            .dataSource(dataSource)
            .sql("SELECT id, amount, account_id FROM transactions")
            .rowMapper(new TransactionRowMapper())
            .fetchSize(1000)
            .build();
}

JdbcPagingItemReader issues repeated paginated queries, which is safer for restartability and works well across multiple threads:

@Bean
public JdbcPagingItemReader<Transaction> pagingReader(DataSource dataSource,
                                                      PagingQueryProvider queryProvider) {
    return new JdbcPagingItemReaderBuilder<Transaction>()
            .name("txPagingReader")
            .dataSource(dataSource)
            .queryProvider(queryProvider)
            .pageSize(1000)
            .rowMapper(new TransactionRowMapper())
            .build();
}

Tip: Set fetchSize/pageSize to match your chunk size to reduce round trips.

Scaling Strategies

A single-threaded step eventually hits a ceiling. Spring Batch offers several scaling models.

1. Multi-threaded Step

The simplest scaling option runs chunks concurrently on a thread pool:

@Bean
public Step multiThreadedStep(JobRepository jobRepository,
                              PlatformTransactionManager txManager,
                              ItemReader<Transaction> reader,
                              ItemWriter<ReportRow> writer) {
    return new StepBuilder("multiThreadedStep", jobRepository)
            .<Transaction, ReportRow>chunk(1000, txManager)
            .reader(reader)
            .writer(writer)
            .taskExecutor(new SimpleAsyncTaskExecutor("batch-"))
            .throttleLimit(8)
            .build();
}

Caveat: Your reader must be thread-safe. JdbcCursorItemReader is not, whereas JdbcPagingItemReader is. Alternatively, wrap the reader in SynchronizedItemStreamReader.

2. Partitioning

For massive datasets, partitioning splits the data into ranges processed by separate step instances—each with its own reader, processor, and writer. This is ideal for parallel execution across threads or even remote nodes.

@Bean
public Step masterStep(JobRepository jobRepository,
                       Step workerStep,
                       Partitioner partitioner) {
    return new StepBuilder("masterStep", jobRepository)
            .partitioner("workerStep", partitioner)
            .step(workerStep)
            .gridSize(10)
            .taskExecutor(new SimpleAsyncTaskExecutor("part-"))
            .build();
}

A Partitioner divides the workload—for example, by ID ranges:

java
public class RangePartitioner implements Partitioner {
    @Override
    public Map<String, ExecutionContext> partition(int gridSize) {
        Map<String, ExecutionContext> result = new HashMap<>();
        long min = 1, max = 1_000_000;
        long targetSize = (max - min) / gridSize + 1;

        long start = min;
        for (int i = 0; i < gridSize; i++) {
            ExecutionContext ctx = new ExecutionContext