DEV Community: SagarTrimukhe

Zero downtime, lossless migration of records to ElasticSearch indexes

SagarTrimukhe — Mon, 26 Jan 2026 07:59:41 +0000

Migrating nearly 2TB of audit log data to new Elasticsearch (ES) indexes on an AWS OpenSearch cluster is no small feat—especially when the system is live, handling thousands of new logs every minute, and strict consistency is non-negotiable. Here’s how we tackled this challenge, ensuring zero downtime, no data loss, and minimal impact on other services.

Why Migrate?

Our existing ES schema was no longer meeting the needs of our growing, scalable requirements. Query performance was lagging, and new use cases demanded a more flexible schema. We needed to redesign our ES indexes to support faster retrieval and richer queries, all while keeping response times low.

Challenges

Live System: The audit log service is mission-critical, with other services depending on it for workflow completion. Downtime was not an option.
High Throughput: Thousands of logs are ingested every minute. Missing even a single log could mislead customers, as logs track everything from device additions to user logins.
Shared ES Cluster: Multiple services use the same AWS OpenSearch (Elasticsearch-compatible) cluster, so heavy migration scripts could not be allowed to degrade API latencies.
Large Data Volume: Each month had its own ES index, with four indexes active at any time. We needed to migrate all data from each to new indexes.
No Data Loss, No Duplicates: Consistency was paramount. We could not afford to lose or duplicate any audit logs.

Planning the Migration

1. Understanding the Scale

We started by gathering production-scale data: index sizes, average and maximum latencies, average record size and CPU, memory and latency tolerance thresholds. Since we’d be copying (not moving) data, we also ensured we had enough resources to hold the duplicated data during migration.

To ensure a smooth OpenSearch data migration without overwhelming the cluster, it is important to operate within safe resource limits. Keep CPU utilization of each data node below 70% and maintain JVM memory pressure below 80% throughout the migration process.

When planning the migration indexing rate, also account for existing indexing load, search traffic, and search latency. New indexing throughput should be introduced gradually, based on calculated cluster headroom, assuming a near-linear impact since indexing is both CPU and I/O bound.

The safe migration indexing rate can be estimated using the following model:

CPU safe threshold (CPU_safe): 70% (matches the target data-node ceiling)

Current CPU usage (CPU_now): measured as x

JVM safe threshold (JVM_safe): 80% (keeps pressure under the GC risk zone)

Current JVM pressure (JVM_now): observed JVM memory pressure

JVM growth factor (JVM_growth per 1K): ≈1.2% additional JVM pressure per extra 1,000 docs/sec indexed

On the storage side, although gp3 EBS volumes support 250 MB/sec throughput per node, indexing can consume disk bandwidth. A conservative estimate is:

40 MB/sec disk throughput per 1,000 docs/sec of indexing load

To maintain additional safety, reserve a 30% buffer to absorb unpredictable load increases and avoid resource saturation (multiply by 0.7).

Safe Migration Indexing Formula

R_{safe} = R_{current} \times \min\left(\frac{CPU_{safe}}{CPU_{now}}, \frac{JVM_{safe} - JVM_{now}}{JVM_{growth/1K}}\right) \times 0.7

Practical Example

If the current cluster handles 20,000 records per minute (RPM) and both CPU and JVM show 50% available headroom, you can safely plan to migrate an additional 10,000 records per minute without overloading the system (effective rate ≈30,000 RPM, ≈43 million records per day). This keeps:

CPU remains under 55–60%

JVM pressure stays below 80–82%

Disk throughput remains well within safe limits.

This approach ensures stable search performance and a reliable migration process without compromising cluster health.

2. Efficient Data Reading

Reading data from ES indexes had to be done serially. Parallel reads would add too much query overhead. We considered migrating customer-by-customer, but with 400k+ accounts and the need to fetch account IDs from another microservice, this approach was too complex and would require excessive checkpoint management.

Instead, we used the ES scroll API (search_after/point-in-time works similarly), which allowed us to read data in serial batches using a cursor value. This cursor was stored in Redis as a checkpoint for each month’s index.

3. Designing for Reliability

Given the massive data size, number of records and resource constraints, migration would take time. We needed a process that could be paused or stopped at any point, and resumed from the last checkpoint in case of failure—without starting over. For this, we used Redis to store migration checkpoints, with a 7-day TTL for safety.

Checkpointing Strategy: When to Save the checkpoint in Redis?

Instead of writing the ES cursor to Redis every time after reading data from ES, we decided to write it at 5-minute intervals. This approach kept the write load low on Redis. Since our Redis cluster is also a shared infrastructure, we wanted to avoid any spikes or unnecessary load on Redis during the migration process. A restart could replay up to ~50k docs (5 minutes × 10k/min), which was acceptable because writes were idempotent (same document IDs) and bulk replays were safe for downstream services.

4. Deployment Strategy

The migration script was containerized and deployed as a Kubernetes Job. On startup, it checked Redis for an existing checkpoint. If none was found, it started a fresh migration. Depending on the system load we spanned multiple job instances, each handling a different month’s index. The script was designed to take the month as a parameter, allowing multiple instances to run concurrently without conflict.

5. Batch Processing and Parallel Writes

Each ES call fetched 10,000 records. We split these into ten batches of 1,000 records each, processing them in parallel. Each batch was converted to the new schema, sometimes requiring calls to other internal services for additional data. Once processed, each batch was written back to ES using bulk writes.

We could have increased parallelism, but since each record conversion involved external service calls, we prioritized system stability over speed. The migration was intentionally slow to avoid impacting customers or other services. Bulk writers honored backpressure—if CPU/JVM/disk exceeded thresholds or bulk responses returned rejections, the job applied exponential backoff before resuming.

6. Handling Duplicates and Consistency

A key risk was partial batch processing: if the migration was interrupted after fetching 10,000 records but before all were written, restarting from the last checkpoint could result in duplicate writes. To prevent this, we used the original ES document ID as the ID in the new index. Elasticsearch overwrites documents with the same ID, ensuring no duplicates.

Results

600 Million Logs Migrated: Over four months, we processed about 600 million audit logs.
20 Million Records/Day: On average, 20 million records were migrated daily.
Zero Downtime, No Data Loss: The migration was seamless, with no impact on customers or dependent services.

Once all the data was copied to the index in the updated schema, only then did we enable the system to read from the new indexes. This ensured there was no impact to existing system features during the migration.

As data is continuously flowing into the system for the current month index, we had another module that was writing to the new index continuously, alongside writing to the existing old index. There was no issue with older month indexes, as data cannot be added to those indexes. The system is write- or append-only—no edits or deletes are allowed.

Key Takeaways

Checkpoints are Critical: Storing ES cursor values in Redis allowed safe, resumable migrations.
Batching and Parallelism: Careful batching and controlled parallelism balanced speed and system stability.
Idempotency: Using document IDs to prevent duplicates was essential for data integrity.
Preparation Pays Off: Understanding production data and system constraints upfront made all the difference.

Progress bar to show content read/remaining.

SagarTrimukhe — Sun, 13 Nov 2022 15:37:10 +0000

I found this website where a progress bar is shown which helps users to know how much content they have read or is still remaining.

I tried to recreate that in react and it is easy and involves a bit of maths!

How to get the percentage?

Simple maths :)

Code:

Demo:

Generate YAML files in Golang.

SagarTrimukhe — Wed, 25 Aug 2021 14:55:36 +0000

This is post is about converting go struct/map into a yaml using this amazing go package go-yaml

We will be using yaml.Marshal method to convert a struct into yaml.

Each of the examples will have complete code, so that users can copy paste and quickly run and experiment.

Example 1: Very basic example of converting a struct to yaml.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type Student struct {
    Name string
    Age  int
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML ---")
    fmt.Println(string(yamlData))  // yamlData will be in bytes. So converting it to string.
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML ---
name: Sagar
age: 23

Example 2: Providing custom tags.

Let's say we want to have different key names in output yaml. Then we can do that by adding tags. This very similar to JSON tags.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type Student struct {
    Name string `yaml:"student-name"`
    Age  int    `yaml:"student-age"`
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
    }

    yamlData, err := yaml.Marshal(&s1)
    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML with custom tags---")
    fmt.Println(string(yamlData))
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML with custom tags---
student-name: Sagar
student-age: 23

Tip: Make sure there is no gap between yaml: and "tag". (Wasted 30 mins of my time to figure out why tags are not coming in output)

Example 3: Structure with Arrays and Maps.

Here I have used another structure to store marks. A simple map also works.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type MarksStruct struct {
    Sceince     int8 `yaml:"science"`
    Mathematics int8 `yaml:"mathematics"`
    English     int8 `yaml:"english"`
}

type Student struct {
    Name   string      `yaml:"student-name"`
    Age    int8        `yaml:"student-age"`
    Marks  MarksStruct `yaml:"subject-marks"`
    Sports []string
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
        Marks: MarksStruct{
            Sceince:     95,
            Mathematics: 90,
            English:     90,
        },
        Sports: []string{"Cricket", "Football"},
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML with maps and arrays ---")
    fmt.Println(string(yamlData))
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML with maps and arrays ---
student-name: Sagar
student-age: 23
subject-marks:
  science: 95
  mathematics: 90
  english: 90
sports:
- Cricket
- Football

Example 4: Sometimes we want to actually create a YAML file.

package main
import (
    "fmt"
    "io/ioutil"

    "gopkg.in/yaml.v2"
)

type MarksStruct struct {
    Sceince     int8 `yaml:"science"`
    Mathematics int8 `yaml:"mathematics"`
    English     int8 `yaml:"english"`
}

type Student struct {
    Name   string      `yaml:"student-name"`
    Age    int8        `yaml:"student-age"`
    Marks  MarksStruct `yaml:"subject-marks"`
    Sports []string
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
        Marks: MarksStruct{
            Sceince:     95,
            Mathematics: 90,
            English:     90,
        },
        Sports: []string{"Cricket", "Football"},
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fileName := "test.yaml"
    err = ioutil.WriteFile(fileName, yamlData, 0644)
    if err != nil {
        panic("Unable to write data into the file")
    }
}

Output will be a file named test.yaml file.

Thank you for reading. Hope this helps.

Complete guide to deploy a simple full stack application in Docker

SagarTrimukhe — Thu, 05 Aug 2021 17:03:48 +0000

Create a simple todo UI using React.
Create a simple backend server using Express.
Connect Frontend and Backend.
Create UI bundle and serve it through Express server.
Run the application in Docker

Creating a simple TODO app using React.

We will be using create-react-app to quickly setup a react application with basic dependencies installed.

Command to create the app

npx create-react-app frontend

This will create folder named frontend containing all the basic files with dependencies installed.

Two more dependencies are required:

axios : To make API calls. fetch call also be used.
uuid : To generate random IDs for todo tasks.

commands to install the above

npm install --save axios
npm install --save uuid

Below is the simple UI code which has

A text input box to take the task name.
A "Add" button to add new tasks.
List of previously created tasks with "Delete" button to delete the tasks.

This is how it will look (No fancy colors or animations)

API handlers are maintained in a separate file.

Creating a server using Express.js

Let's start with a folder creation named backend.

cd into the folder
cd backend
Run "npm init" command to create a package.json file
When the above command is ran, it will ask for few details. All can be skipped by hitting enter.
Run "npm install --save express" to install the express js dependency.
By default the entry point of the application will be pointing to "index.js". It can be changed by editing the package.json file
"main": "your_file_name.js"
Create a file index.js (or your_file_name.js)
Following is the simple express js code with 3 APIs.

It has
a. GET /tasks endpoint to get the list of tasks.
b. POST /tasks to create a new task.
c. DELETE /tasks to delete a task.

All the tasks are stored in-memory. The tasks data will be lost as soon the server is stopped.
(So, NOT exactly a FULL STACK application)

To start the server run following command

node index.js

You can test APIs using a REST Client like Postman or simple CURL commands.

Connecting frontend and backend.

Even though the correct endpoints are mentioned in UI, it will not be able to reach the backend APIs due to CORS.

To solve this we need to use a proxy.
It is very simple to proxy the calls by just updating the UI package.json file.

Add the below configuration.

 "proxy": "http://localhost:4000"

Now the UI should be able to connect to backend APIs.

Generating the UI bundle and serving it through express.

Generating production UI bundle is dead simple
Run the below command.

npm run build

This is will create a folder named build in your frontend root directory containing index.html file along with JavaScript and CSS files.

This can be served using a static server like "serve" package.

BUT, the UI will not be able to reach backend servers.
BECAUSE, proxy feature is available only in development mode.

To solve this issue, we can serve the UI using same express server. YES you read it right. a single server to server both frontend and backend.

To do so,

Copy the build folder to backend folder
Add the following line in index.js file

app.use(express.static('build'));

We can give the absolute path also, just keeping it simple here :)

Now start the express server. At '/' path the UI will be served and at other paths, the APIs will be available.

Deploying the application in a container.

Till now we have developed and deployed the application on local machine.

If you are a docker guy, then we can quickly deploy this application in a container as well.

Create a Dockerfile. Dockerfile is a simple text file containing all the commands to create a docker image. The following is a docker file which uses alpine OS as a base image and exposes the port 4000.

Create a docker image Run the command to build the image

docker image build -t todoapp:1.0 .

Start the container Once the image is created, next step is to create a container. Run the command to create and start the container.

docker container run -d -p 8000:4000 todoapp:1.0

If the docker is running on a VM then the application can be accessed at VM-IP-Address:8000 eg. http://192.168.43.18:8000

Complete project is available at: https://github.com/SagarTrimukhe/todo-app

No Internet Connection wrapper for React apps

SagarTrimukhe — Fri, 14 May 2021 07:36:26 +0000

Imagine,
We have a web application that heavily depends on the backend server for information (eg. records in a table) and that information needs to be constantly updated. We might think to use some polling mechanism.

But if the data received from the server is directly stored in a React state variable and if the user loses the internet connection then there are chances of updating the state with empty data.

So, instead of showing empty data, we can show a message, something like "No internet connection."

How can we do that?

We can write a wrapper component and wrap the entry-level component. So whenever the internet connection is lost, a custom page/message can be shown.

Here I have used the navigator.onLine API to get the network status.

enough story, show me the code :)

Main component



import './App.css';
import NoInternetConnection from './NoInternetConnection'

function App() {
  return (
    <div className="App">
        <NoInternetConnection>
        <h1>My first post on DEV!!</h1>
        </NoInternetConnection>
    </div>
  );
}

export default App;

Wrapper component



import React, {useState, useEffect} from 'react';

const NoInternetConnection = (props) => {
    // state variable holds the state of the internet connection
    const [isOnline, setOnline] = useState(true);

    // On initization set the isOnline state.
    useEffect(()=>{
        setOnline(navigator.onLine)
    },[])

    // event listeners to update the state 
    window.addEventListener('online', () => {
        setOnline(true)
    });

    window.addEventListener('offline', () => {
        setOnline(false)
    });

    // if user is online, return the child component else return a custom component
    if(isOnline){
    return(
        props.children
    )
    } else {
        return(<h1>No Interner Connection. Please try again later.</h1>)
    }
}

export default NoInternetConnection;

Here is the demo.

That's it. By the way, this is my first post on DEV (damn! on the internet :)). Feedback is appreciated.

DEV Community: SagarTrimukhe

Zero downtime, lossless migration of records to ElasticSearch indexes

Why Migrate?

Challenges

Planning the Migration

1. Understanding the Scale

2. Efficient Data Reading

3. Designing for Reliability

4. Deployment Strategy

5. Batch Processing and Parallel Writes

6. Handling Duplicates and Consistency

Results

Key Takeaways

Progress bar to show content read/remaining.

How to get the percentage?

Code:

Demo:

Generate YAML files in Golang.

Example 1: Very basic example of converting a struct to yaml.

Example 2: Providing custom tags.

Example 3: Structure with Arrays and Maps.

Example 4: Sometimes we want to actually create a YAML file.

Complete guide to deploy a simple full stack application in Docker

Table of contents

Creating a simple TODO app using React.

Creating a server using Express.js

Connecting frontend and backend.

Generating the UI bundle and serving it through express.

Deploying the application in a container.

No Internet Connection wrapper for React apps

How can we do that?

Main component

Wrapper component