DEV Community: Shaogat Alam

🛠️ Understanding the Relationship Between chunk() Size and fetchSize in Spring Batch 🔄

Shaogat Alam — Wed, 11 Sep 2024 15:37:03 +0000

In Spring Batch, the chunk() size and fetchSize in the JdbcPagingItemReader serve different purposes. Here's how they interact and what happens when one is larger than the other:

1. `chunk()` Size (Chunk-Oriented Processing)

The chunk() size defines the number of items that will be processed (read, processed, and written) in a single transaction.
When the chunk size is reached, Spring Batch will commit the transaction, and a new chunk begins.

2. `fetchSize` (Database Fetch Size)

The fetchSize controls the number of rows retrieved from the database in one query execution (or one "fetch" from the database cursor).
It is a performance optimization that helps reduce the number of database round-trips, especially for large datasets.

Relationship Between `fetchSize` and `chunk()` Size

If chunk() size > fetchSize:
- Spring Batch will fetch data from the database in smaller batches (based on the fetchSize) but will still process and commit data in larger chunks.
- For example, if fetchSize = 100 and chunk() = 200, Spring Batch will first fetch 100 records, then another 100, and process all 200 records in a single chunk before committing.
- There will be more database round-trips compared to a scenario where fetchSize equals or exceeds chunk() size.
If fetchSize > chunk() size:
- Spring Batch will fetch more records than it needs for one chunk, but it will only process the chunk size before committing the transaction.
- For example, if fetchSize = 500 and chunk() = 200, Spring Batch will fetch 500 records from the database but only process 200 before committing. The remaining 300 will stay in memory for the next chunks.
- This can be more efficient in terms of reducing database round-trips but may consume more memory because the remaining records will be kept in memory until processed.

Ideal Configuration

Match chunk() size and fetchSize if possible: This ensures that Spring Batch fetches exactly the number of records needed for each chunk, minimizing round-trips while avoiding excessive memory usage.
Adjust based on database and memory constraints:
- If your database can handle large fetch sizes without performance degradation, you can set a higher fetchSize than chunk() size.
- If memory consumption is a concern, setting fetchSize equal to or lower than chunk() size ensures that only the necessary records are held in memory at any time.

Scenarios

Chunk Size > Fetch Size Example:

   stepBuilderFactory.get("userEmailStep")
       .<User, Email>chunk(500)  // Process 500 records per chunk (per transaction)
       .reader(userReader())  // Fetch 200 records at a time from the database
       .processor(emailProcessor())
       .writer(emailWriter())
       .build();

Fetches 200 records from the database.
Processes the first 200, then fetches another 200, and so on until 500 records are processed in the current chunk.
The transaction is committed after processing the chunk of 500 records.

Fetch Size > Chunk Size Example:

   JdbcPagingItemReader<User> reader = new JdbcPagingItemReader<>();
   reader.setFetchSize(1000);  // Fetch 1000 records from the database

Fetches 1000 records from the database.
Processes 500 records at a time (assuming chunk(500)), and the remaining 500 records are stored in memory for the next chunk.
This reduces the number of database fetches but increases memory usage.

Summary

If chunk() size is larger than fetchSize, it leads to multiple database fetches to process one chunk.
If fetchSize is larger than chunk() size, the fetched data will stay in memory until fully processed, reducing database fetches but consuming more memory.

SelectPaginated: Handle Millions of Options Quickly and Efficiently.

Shaogat Alam — Mon, 08 Jul 2024 11:41:52 +0000

Hello everyone,

I'd like to introduce SelectPaginated, a paginated select component for React that can handle large datasets and provide API call and pagination functionality.

The key features of SelectPaginated include:

Large Dataset Handling: By fetching data in small, manageable chunks and utilizing local storage to cache the fetched data, SelectPaginated effectively handles large datasets without compromising performance.
API Call Integration: The component seamlessly integrates API calls, allowing you to fetch data from external sources and present it to your users.
Pagination: SelectPaginated incorporates pagination functionality, enabling your users to navigate through large datasets with ease.
Local Storage Support: The component uses the browser's local storage to persist the fetched data, reducing the need for repeated API calls and improving the overall user experience.
Static Data Support: SelectPaginated also supports the use of static data, providing flexibility in the way you handle your application's data requirements.

To install the select-paginated package, you can use the following command:

npm i select-paginated

npmjs - select-paginated

Usage

import React from 'react'
import SelectPaginated from 'select-paginated';

function Test() {
    const options = [
        { id: 1, name: 'Option 1', description: 'This is the first option' },
        { id: 2, name: 'Option 2', description: 'This is the second option' }
    ];

  return (
    <>
        <SelectPaginated 
          // Provide `options` prop when `api` prop is not being used
          options={options} 

          // Provide `api` prop when `options` prop is not being used
          api={{
                resourceUrl: "https://jsonplaceholder.typicode.com/comments",
                pageParamKey: "_page", 
                limitParamKey: "_limit",
                // Final endpoint: "https://jsonplaceholder.typicode.com/comments?_page=1&_limit=50"
            }}
            displayKey="name"
            pageSize={50}
            isLinearArray={false}
            onSelect={(selectedItems) => {
                console.log('selected items :: ', JSON.stringify(selectedItems));
            }}
            onRemove={(removedItem) => {
                console.log('Removed items :: ', JSON.stringify(removedItem));
            }}
            multiSelect={true}
            searchPlaceholder="Search..."
            localStorageKey="SelectFetchedData"
        />
    </>
  )
}

export default Test

Props

options(array,required when api prop is not provided )

Description: An array of pre-defined options to be used instead of fetching data from an API.This can be particularly useful for small dataset, static datasets or for data that is already available on the client side.
Example :
- Simple linear array - [ "Item 1", "Item 2", "Item 3", // ...more items],
- Array of objects - [ { id: 1, name: "Item 1" }, { id: 2, name: "Item 2" }, // ...more items ]

pageSize(number,default:50) :

The number of items to show and fetch(in-case of fetching data) per page.

isLinearArray(boolean,default:false) :

Set {true} when :
- The fetched data or value of options prop is a simple linear array of primitive values (e.g., strings, numbers).
- No displayKey is needed.
- Example: ["item1", "item2", "item3"]

Set {false} when:

The fetched data or value of options prop is an array of objects.
A displayKey must be specified to indicate which property to display. c. Example:

[
    {"name": "id labore ex et quam laborum","email": "Eliseo@gardner.biz",},
    {"name": "quo vero reiciendis velit similique earum","email": "Jayne_Kuhic@sydney.com"},
]

In this case, set displayKey to the property you want to display, e.g., "email".

displayKey(string,default:'name',required only when isLinearArray is false) :

Description : Specifies the property of the objects in the array to be displayed.
For instance, consider the following response from an API:
```
[
    {"name": "id labore ex et quam laborum","email": "Eliseo@gardner.biz",},
    {"name": "quo vero reiciendis velit similique earum","email": "Jayne_Kuhic@sydney.com"},
]
```
- To display the "email" field, set displayKey to "email".

api(object,required when options prop is not provided) :

Properties :
- resourceUrl (string, required):
  - The URL from which data will be fetched.
- pageParamKey (string, optional, default: "_page") :
  - This is the query parameter key used by your backend API to specify the page number. It should match what your backend expects for pagination.Common defaults include "page", "pageNumber", "p".
  - Example : If pageParamKey is set to "page", the API request URL might include ?page=1, ?page=2, etc.
- limitParamKey (string, optional, default: "_limit") :
  - This is the query parameter key used by your backend API to specify the number of items per page. Similar to pageParamKey, it should align with your backend's pagination configuration. Common defaults include "limit", "pageSize", "size".
  - Example: If limitParamKey is set to "size", the API request URL might include ?size=10, ?size=20, etc.

onSelect (function, optional) :

A callback function invoked when items are selected.

onRemove (function, optional) :

A callback function invoked when items are removed.

multiSelect (boolean, optional, default: true) :

Enables or disables multi-selection mode.

searchPlaceholder (string, optional, default: "Search...") :

Placeholder text for the search input field.

localStorageKey (string, optional, default: "SelectFetchedData") :

The unique key used to store data in local storage for persistence.

SelectPaginated: Handle Millions of Options Quickly and Efficiently.

Shaogat Alam — Fri, 28 Jun 2024 15:10:21 +0000

Hello everyone,

I'd like to introduce SelectPaginated, a paginated select component for React that can handle large datasets and provide API call and pagination functionality.

The key features of SelectPaginated include:

Large Dataset Handling: By fetching data in small, manageable chunks and utilizing local storage to cache the fetched data, SelectPaginated effectively handles large datasets without compromising performance.
API Call Integration: The component seamlessly integrates API calls, allowing you to fetch data from external sources and present it to your users.
Pagination: SelectPaginated incorporates pagination functionality, enabling your users to navigate through large datasets with ease.
Local Storage Support: The component uses the browser's local storage to persist the fetched data, reducing the need for repeated API calls and improving the overall user experience.
Static Data Support: SelectPaginated also supports the use of static data, providing flexibility in the way you handle your application's data requirements.

To install the select-paginated package, you can use the following command:

npm i select-paginated

npmjs - select-paginated

Usage

import React from 'react'
import SelectPaginated from 'select-paginated';

function Test() {
    const options = [
        { id: 1, name: 'Option 1', description: 'This is the first option' },
        { id: 2, name: 'Option 2', description: 'This is the second option' }
    ];

  return (
    <>
        <SelectPaginated 
          // Provide `options` prop when `api` prop is not being used
          options={options} 

          // Provide `api` prop when `options` prop is not being used
          api={{
                resourceUrl: "https://jsonplaceholder.typicode.com/comments",
                pageParamKey: "_page", 
                limitParamKey: "_limit",
                // Final endpoint: "https://jsonplaceholder.typicode.com/comments?_page=1&_limit=50"
            }}
            displayKey="name"
            pageSize={50}
            isLinearArray={false}
            onSelect={(selectedItems) => {
                console.log('selected items :: ', JSON.stringify(selectedItems));
            }}
            onRemove={(removedItem) => {
                console.log('Removed items :: ', JSON.stringify(removedItem));
            }}
            multiSelect={true}
            searchPlaceholder="Search..."
            localStorageKey="SelectFetchedData"
        />
    </>
  )
}

export default Test

Props

options(array,required when api prop is not provided )

Description: An array of pre-defined options to be used instead of fetching data from an API.This can be particularly useful for small dataset, static datasets or for data that is already available on the client side.
Example :
- Simple linear array - [ "Item 1", "Item 2", "Item 3", // ...more items],
- Array of objects - [ { id: 1, name: "Item 1" }, { id: 2, name: "Item 2" }, // ...more items ]

pageSize(number,default:50) :

The number of items to show and fetch(in-case of fetching data) per page.

isLinearArray(boolean,default:false) :

Set {true} when :
- The fetched data or value of options prop is a simple linear array of primitive values (e.g., strings, numbers).
- No displayKey is needed.
- Example: ["item1", "item2", "item3"]

Set {false} when:

The fetched data or value of options prop is an array of objects.
A displayKey must be specified to indicate which property to display. c. Example:

[
    {"name": "id labore ex et quam laborum","email": "Eliseo@gardner.biz",},
    {"name": "quo vero reiciendis velit similique earum","email": "Jayne_Kuhic@sydney.com"},
]

In this case, set displayKey to the property you want to display, e.g., "email".

displayKey(string,default:'name',required only when isLinearArray is false) :

Description : Specifies the property of the objects in the array to be displayed.
For instance, consider the following response from an API:
```
[
    {"name": "id labore ex et quam laborum","email": "Eliseo@gardner.biz",},
    {"name": "quo vero reiciendis velit similique earum","email": "Jayne_Kuhic@sydney.com"},
]
```
- To display the "email" field, set displayKey to "email".

api(object,required when options prop is not provided) :

Properties :
- resourceUrl (string, required):
  - The URL from which data will be fetched.
- pageParamKey (string, optional, default: "_page") :
  - This is the query parameter key used by your backend API to specify the page number. It should match what your backend expects for pagination.Common defaults include "page", "pageNumber", "p".
  - Example : If pageParamKey is set to "page", the API request URL might include ?page=1, ?page=2, etc.
- limitParamKey (string, optional, default: "_limit") :
  - This is the query parameter key used by your backend API to specify the number of items per page. Similar to pageParamKey, it should align with your backend's pagination configuration. Common defaults include "limit", "pageSize", "size".
  - Example: If limitParamKey is set to "size", the API request URL might include ?size=10, ?size=20, etc.

onSelect (function, optional) :

A callback function invoked when items are selected.

onRemove (function, optional) :

A callback function invoked when items are removed.

multiSelect (boolean, optional, default: true) :

Enables or disables multi-selection mode.

searchPlaceholder (string, optional, default: "Search...") :

Placeholder text for the search input field.

localStorageKey (string, optional, default: "SelectFetchedData") :

The unique key used to store data in local storage for persistence.

DEV Community: Shaogat Alam

🛠️ Understanding the Relationship Between chunk() Size and fetchSize in Spring Batch 🔄

1. chunk() Size (Chunk-Oriented Processing)

2. fetchSize (Database Fetch Size)

Relationship Between fetchSize and chunk() Size

Ideal Configuration

Scenarios

Summary

SelectPaginated: Handle Millions of Options Quickly and Efficiently.

To install the select-paginated package, you can use the following command:

npmjs - select-paginated

Usage

Props

SelectPaginated: Handle Millions of Options Quickly and Efficiently.

To install the select-paginated package, you can use the following command:

npmjs - select-paginated

Usage

Props

1. `chunk()` Size (Chunk-Oriented Processing)

2. `fetchSize` (Database Fetch Size)

Relationship Between `fetchSize` and `chunk()` Size