DEV Community: SagarTrimukhe

Why `livez` and `readyz` Matter for Kubernetes Health Probes

SagarTrimukhe — Sat, 09 May 2026 06:42:35 +0000

When a service handles a high volume of requests, health probes are not just a small deployment detail anymore. They become part of the reliability strategy.

In Kubernetes, liveness and readiness probes help the platform decide two different things:

Is this container still alive and capable of making forward progress?
Is this pod ready to receive production traffic right now?

This distinction matters. Even one minute of unavailability in a busy service can mean thousands of dropped requests. In loss-sensitive systems like Audit trail, logging systems, and financial services, dropped requests are not just temporary failures. They can become permanent data loss.

What we will cover

In this article, we will look at:

what livez and readyz actually mean
why both probes are needed and why one cannot replace the other
how dependencies like Kafka, Redis, and OpenSearch affect probe design
how to think about readiness failures vs liveness failures
how probe frequency, timeout, and thresholds affect recovery behavior
common mistakes that make Kubernetes restart pods too aggressively

`livez` vs `readyz`

Both probes solve different problems.

Liveness

Liveness answers a simple question: should Kubernetes restart this container?

If the process is stuck, deadlocked, or unable to make progress, the liveness probe should fail so Kubernetes can recreate the pod.

Readiness

Readiness answers another question: should this pod receive traffic?

If the process is alive but temporarily unable to serve requests safely, the readiness probe should fail so Kubernetes removes the pod from the set of ready endpoints without killing it.

This separation is very important in distributed systems. Not every dependency issue should cause a restart. Some failures are transient, and restarting the service can actually make things worse by creating churn, reconnection storms, or repeated cold starts.

One more important point: for automated health decisions, the HTTP status code is what matters. Human-readable details are useful for debugging, but machines should rely on the status code instead of parsing response text.

Visual summary

Why Probe Design Matters in Real Services

Let's take an example of a production service that depends on external infrastructure such as Kafka, OpenSearch, databases, and Redis. If those dependencies are not reachable, the service may no longer be able to accept, process, store, or retrieve requests correctly.

Some dependencies are critical. Others are optional or only affect performance. That is exactly why probe design has to be deliberate:

If every transient Redis failure returns 500 from readiness, pods may flap in and out of service too aggressively.
If a dependency is essential for correctness, masking the failure may keep serving bad or incomplete responses.
If a dependency is only an optimization, such as Redis caching, keeping the pod ready may be the right decision while the service falls back to a slower path.
If a dependency failure causes the process to become unresponsive or completely fail, liveness should fail so Kubernetes can restart it.

There is no single rule that fits every dependency. Health checks should reflect what role that dependency plays in the request path.

A Practical Implementation Pattern

One practical pattern is to keep liveness and readiness intentionally different.

`livez` pattern

A good livez endpoint focuses on forward progress. For read-heavy services, that may mean checking the critical read path. For write-heavy services, it may mean checking process responsiveness and whether repeated Kafka publish failures have pushed the service into an unhealthy state.

`readyz` pattern

readyz should answer one question: Can this pod safely receive traffic right now? If a critical dependency, such as Kafka is unavailable or not yet ready, readiness should fail so Kubernetes stops routing traffic to the pod without restarting it.

Example Kubernetes Probe Configuration

Here is a typical probe configuration:

livenessProbe:
  httpGet:
    path: /livez
    port: app-port
  initialDelaySeconds: 15
  failureThreshold: 3
  periodSeconds: 60
  timeoutSeconds: 30

readinessProbe:
  httpGet:
    path: /readyz
    port: app-port
  initialDelaySeconds: 5
  periodSeconds: 30
  timeoutSeconds: 30

These settings control when probing starts, how often it runs, how long Kubernetes waits, and how many failures it tolerates before acting. In many cases, readiness is checked more frequently than liveness so traffic can be drained quickly without restarting pods too aggressively.

What Can Go Wrong in Distributed Systems

Failures are expected in distributed systems. They are not edge cases. They are normal operating conditions.

A dependency can become unavailable for many reasons:

network partitions between services
DNS lookup failures
broker node failures in Kafka
leader elections or replica rebalancing
intermittent packet loss or latency spikes
TLS handshake or certificate issues
connection pool exhaustion
CPU starvation or event loop stalls
storage latency spikes in downstream systems
OpenSearch cluster pressure or shard unavailability
authentication token expiry or authorization failures
service mesh or load balancer routing problems

So the real job of probe design is to distinguish between a pod that is temporarily degraded and a pod that genuinely needs to be removed or restarted.

Intermittent Failures Need Intelligent Handling

For example, suppose Redis disconnects briefly:

If Redis is only an optimization and requests can still be served correctly, failing readiness may be unnecessary.
If Redis is required to prevent overload or enforce correctness, readiness may need to fail.
Liveness should almost never fail for a short Redis outage unless the service is truly wedged and cannot recover without a restart.

Many systems get probe behavior wrong by treating every dependency failure as a reason to restart. A better pattern is this:

use liveness for stuck or unrecoverable local process failure
use readiness for temporary inability to serve traffic safely
use internal retry logic, circuit breakers, and connection recovery for transient downstream issues

Threshold-based liveness logic follows this principle by reacting to repeated failure instead of a single failed operation.

Tuning Probe Frequency and Thresholds

If probes run too frequently:

they add unnecessary load to the service
they can add load to downstream systems
they may amplify transient failures into noisy operational events

If probes run too infrequently:

unhealthy pods remain in rotation longer
recovery is delayed
downtime or bad responses last longer than necessary

There is no universal number that works for every service. The right settings depend on:

how quickly the service must stop receiving traffic after a failure
how expensive the probe logic is
whether downstream systems can tolerate frequent health checks
whether restarts are cheap or expensive
whether the service is stateless, stateful, or loss-sensitive

For a loss-sensitive service, probe tuning should be conservative enough to avoid false positives, but still responsive enough to stop sending traffic to a pod that cannot safely process requests.

Practical Guidelines

When designing livez and readyz endpoints for a production service, these rules help:

Keep livez focused on process health and forward progress.
Use readyz to reflect whether the pod can safely serve traffic now.
Do not make liveness fail for every transient downstream issue.
Avoid expensive checks inside high-frequency probes.
Use thresholds and recovery logic for intermittent failures.
Treat critical dependencies differently from optional or degradable ones.
Tune probe intervals and thresholds based on service criticality, not guesswork.

Closing Thoughts

Liveness and readiness probes look simple, but they strongly influence availability, recovery behavior, and data safety in Kubernetes.

In simple words:

livez checks whether the process is still functioning in its assigned role
readyz decides whether the pod should continue receiving traffic

That separation is what makes Kubernetes probes useful. They should not just report status. They should reflect operational intent.

If you design them carefully, they help Kubernetes make the right decision during failures. If you design them poorly, they can turn small dependency blips into wider outages.

Zero downtime, lossless migration of records to ElasticSearch indexes

SagarTrimukhe — Mon, 26 Jan 2026 07:59:41 +0000

Migrating nearly 2TB of audit log data to new Elasticsearch (ES) indexes on an AWS OpenSearch cluster is no small feat—especially when the system is live, handling thousands of new logs every minute, and strict consistency is non-negotiable. Here’s how we tackled this challenge, ensuring zero downtime, no data loss, and minimal impact on other services.

Why Migrate?

Our existing ES schema was no longer meeting the needs of our growing, scalable requirements. Query performance was lagging, and new use cases demanded a more flexible schema. We needed to redesign our ES indexes to support faster retrieval and richer queries, all while keeping response times low.

Challenges

Live System: The audit log service is mission-critical, with other services depending on it for workflow completion. Downtime was not an option.
High Throughput: Thousands of logs are ingested every minute. Missing even a single log could mislead customers, as logs track everything from device additions to user logins.
Shared ES Cluster: Multiple services use the same AWS OpenSearch (Elasticsearch-compatible) cluster, so heavy migration scripts could not be allowed to degrade API latencies.
Large Data Volume: Each month had its own ES index, with four indexes active at any time. We needed to migrate all data from each to new indexes.
No Data Loss, No Duplicates: Consistency was paramount. We could not afford to lose or duplicate any audit logs.

Planning the Migration

1. Understanding the Scale

We started by gathering production-scale data: index sizes, average and maximum latencies, average record size and CPU, memory and latency tolerance thresholds. Since we’d be copying (not moving) data, we also ensured we had enough resources to hold the duplicated data during migration.

To ensure a smooth OpenSearch data migration without overwhelming the cluster, it is important to operate within safe resource limits. Keep CPU utilization of each data node below 70% and maintain JVM memory pressure below 80% throughout the migration process.

When planning the migration indexing rate, also account for existing indexing load, search traffic, and search latency. New indexing throughput should be introduced gradually, based on calculated cluster headroom, assuming a near-linear impact since indexing is both CPU and I/O bound.

The safe migration indexing rate can be estimated using the following model:

CPU safe threshold (CPU_safe): 70% (matches the target data-node ceiling)

Current CPU usage (CPU_now): measured as x

JVM safe threshold (JVM_safe): 80% (keeps pressure under the GC risk zone)

Current JVM pressure (JVM_now): observed JVM memory pressure

JVM growth factor (JVM_growth per 1K): ≈1.2% additional JVM pressure per extra 1,000 docs/sec indexed

On the storage side, although gp3 EBS volumes support 250 MB/sec throughput per node, indexing can consume disk bandwidth. A conservative estimate is:

40 MB/sec disk throughput per 1,000 docs/sec of indexing load

To maintain additional safety, reserve a 30% buffer to absorb unpredictable load increases and avoid resource saturation (multiply by 0.7).

Safe Migration Indexing Formula

R_{safe} = R_{current} \times \min\left(\frac{CPU_{safe}}{CPU_{now}}, \frac{JVM_{safe} - JVM_{now}}{JVM_{growth/1K}}\right) \times 0.7

Practical Example

If the current cluster handles 20,000 records per minute (RPM) and both CPU and JVM show 50% available headroom, you can safely plan to migrate an additional 10,000 records per minute without overloading the system (effective rate ≈30,000 RPM, ≈43 million records per day). This keeps:

CPU remains under 55–60%

JVM pressure stays below 80–82%

Disk throughput remains well within safe limits.

This approach ensures stable search performance and a reliable migration process without compromising cluster health.

2. Efficient Data Reading

Reading data from ES indexes had to be done serially. Parallel reads would add too much query overhead. We considered migrating customer-by-customer, but with 400k+ accounts and the need to fetch account IDs from another microservice, this approach was too complex and would require excessive checkpoint management.

Instead, we used the ES scroll API (search_after/point-in-time works similarly), which allowed us to read data in serial batches using a cursor value. This cursor was stored in Redis as a checkpoint for each month’s index.

3. Designing for Reliability

Given the massive data size, number of records and resource constraints, migration would take time. We needed a process that could be paused or stopped at any point, and resumed from the last checkpoint in case of failure—without starting over. For this, we used Redis to store migration checkpoints, with a 7-day TTL for safety.

Checkpointing Strategy: When to Save the checkpoint in Redis?

Instead of writing the ES cursor to Redis every time after reading data from ES, we decided to write it at 5-minute intervals. This approach kept the write load low on Redis. Since our Redis cluster is also a shared infrastructure, we wanted to avoid any spikes or unnecessary load on Redis during the migration process. A restart could replay up to ~50k docs (5 minutes × 10k/min), which was acceptable because writes were idempotent (same document IDs) and bulk replays were safe for downstream services.

4. Deployment Strategy

The migration script was containerized and deployed as a Kubernetes Job. On startup, it checked Redis for an existing checkpoint. If none was found, it started a fresh migration. Depending on the system load we spanned multiple job instances, each handling a different month’s index. The script was designed to take the month as a parameter, allowing multiple instances to run concurrently without conflict.

5. Batch Processing and Parallel Writes

Each ES call fetched 10,000 records. We split these into ten batches of 1,000 records each, processing them in parallel. Each batch was converted to the new schema, sometimes requiring calls to other internal services for additional data. Once processed, each batch was written back to ES using bulk writes.

We could have increased parallelism, but since each record conversion involved external service calls, we prioritized system stability over speed. The migration was intentionally slow to avoid impacting customers or other services. Bulk writers honored backpressure—if CPU/JVM/disk exceeded thresholds or bulk responses returned rejections, the job applied exponential backoff before resuming.

6. Handling Duplicates and Consistency

A key risk was partial batch processing: if the migration was interrupted after fetching 10,000 records but before all were written, restarting from the last checkpoint could result in duplicate writes. To prevent this, we used the original ES document ID as the ID in the new index. Elasticsearch overwrites documents with the same ID, ensuring no duplicates.

Results

600 Million Logs Migrated: Over four months, we processed about 600 million audit logs.
20 Million Records/Day: On average, 20 million records were migrated daily.
Zero Downtime, No Data Loss: The migration was seamless, with no impact on customers or dependent services.

Once all the data was copied to the index in the updated schema, only then did we enable the system to read from the new indexes. This ensured there was no impact to existing system features during the migration.

As data is continuously flowing into the system for the current month index, we had another module that was writing to the new index continuously, alongside writing to the existing old index. There was no issue with older month indexes, as data cannot be added to those indexes. The system is write- or append-only—no edits or deletes are allowed.

Key Takeaways

Checkpoints are Critical: Storing ES cursor values in Redis allowed safe, resumable migrations.
Batching and Parallelism: Careful batching and controlled parallelism balanced speed and system stability.
Idempotency: Using document IDs to prevent duplicates was essential for data integrity.
Preparation Pays Off: Understanding production data and system constraints upfront made all the difference.

Progress bar to show content read/remaining.

SagarTrimukhe — Sun, 13 Nov 2022 15:37:10 +0000

I found this website where a progress bar is shown which helps users to know how much content they have read or is still remaining.

I tried to recreate that in react and it is easy and involves a bit of maths!

How to get the percentage?

Simple maths :)

Code:

Demo:

Generate YAML files in Golang.

SagarTrimukhe — Wed, 25 Aug 2021 14:55:36 +0000

This is post is about converting go struct/map into a yaml using this amazing go package go-yaml

We will be using yaml.Marshal method to convert a struct into yaml.

Each of the examples will have complete code, so that users can copy paste and quickly run and experiment.

Example 1: Very basic example of converting a struct to yaml.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type Student struct {
    Name string
    Age  int
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML ---")
    fmt.Println(string(yamlData))  // yamlData will be in bytes. So converting it to string.
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML ---
name: Sagar
age: 23

Example 2: Providing custom tags.

Let's say we want to have different key names in output yaml. Then we can do that by adding tags. This very similar to JSON tags.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type Student struct {
    Name string `yaml:"student-name"`
    Age  int    `yaml:"student-age"`
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
    }

    yamlData, err := yaml.Marshal(&s1)
    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML with custom tags---")
    fmt.Println(string(yamlData))
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML with custom tags---
student-name: Sagar
student-age: 23

Tip: Make sure there is no gap between yaml: and "tag". (Wasted 30 mins of my time to figure out why tags are not coming in output)

Example 3: Structure with Arrays and Maps.

Here I have used another structure to store marks. A simple map also works.

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
)

type MarksStruct struct {
    Sceince     int8 `yaml:"science"`
    Mathematics int8 `yaml:"mathematics"`
    English     int8 `yaml:"english"`
}

type Student struct {
    Name   string      `yaml:"student-name"`
    Age    int8        `yaml:"student-age"`
    Marks  MarksStruct `yaml:"subject-marks"`
    Sports []string
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
        Marks: MarksStruct{
            Sceince:     95,
            Mathematics: 90,
            English:     90,
        },
        Sports: []string{"Cricket", "Football"},
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fmt.Println(" --- YAML with maps and arrays ---")
    fmt.Println(string(yamlData))
}

Output:

PS D:\Programming Languages\Go\src\yaml_conversion> go run .\main.go
 --- YAML with maps and arrays ---
student-name: Sagar
student-age: 23
subject-marks:
  science: 95
  mathematics: 90
  english: 90
sports:
- Cricket
- Football

Example 4: Sometimes we want to actually create a YAML file.

package main
import (
    "fmt"
    "io/ioutil"

    "gopkg.in/yaml.v2"
)

type MarksStruct struct {
    Sceince     int8 `yaml:"science"`
    Mathematics int8 `yaml:"mathematics"`
    English     int8 `yaml:"english"`
}

type Student struct {
    Name   string      `yaml:"student-name"`
    Age    int8        `yaml:"student-age"`
    Marks  MarksStruct `yaml:"subject-marks"`
    Sports []string
}

func main() {
    s1 := Student{
        Name: "Sagar",
        Age:  23,
        Marks: MarksStruct{
            Sceince:     95,
            Mathematics: 90,
            English:     90,
        },
        Sports: []string{"Cricket", "Football"},
    }

    yamlData, err := yaml.Marshal(&s1)

    if err != nil {
        fmt.Printf("Error while Marshaling. %v", err)
    }

    fileName := "test.yaml"
    err = ioutil.WriteFile(fileName, yamlData, 0644)
    if err != nil {
        panic("Unable to write data into the file")
    }
}

Output will be a file named test.yaml file.

Thank you for reading. Hope this helps.

Complete guide to deploy a simple full stack application in Docker

SagarTrimukhe — Thu, 05 Aug 2021 17:03:48 +0000

Create a simple todo UI using React.
Create a simple backend server using Express.
Connect Frontend and Backend.
Create UI bundle and serve it through Express server.
Run the application in Docker

Creating a simple TODO app using React.

We will be using create-react-app to quickly setup a react application with basic dependencies installed.

Command to create the app

npx create-react-app frontend

This will create folder named frontend containing all the basic files with dependencies installed.

Two more dependencies are required:

axios : To make API calls. fetch call also be used.
uuid : To generate random IDs for todo tasks.

commands to install the above

npm install --save axios
npm install --save uuid

Below is the simple UI code which has

A text input box to take the task name.
A "Add" button to add new tasks.
List of previously created tasks with "Delete" button to delete the tasks.

This is how it will look (No fancy colors or animations)

API handlers are maintained in a separate file.

Creating a server using Express.js

Let's start with a folder creation named backend.

cd into the folder
cd backend
Run "npm init" command to create a package.json file
When the above command is ran, it will ask for few details. All can be skipped by hitting enter.
Run "npm install --save express" to install the express js dependency.
By default the entry point of the application will be pointing to "index.js". It can be changed by editing the package.json file
"main": "your_file_name.js"
Create a file index.js (or your_file_name.js)
Following is the simple express js code with 3 APIs.

It has
a. GET /tasks endpoint to get the list of tasks.
b. POST /tasks to create a new task.
c. DELETE /tasks to delete a task.

All the tasks are stored in-memory. The tasks data will be lost as soon the server is stopped.
(So, NOT exactly a FULL STACK application)

To start the server run following command

node index.js

You can test APIs using a REST Client like Postman or simple CURL commands.

Connecting frontend and backend.

Even though the correct endpoints are mentioned in UI, it will not be able to reach the backend APIs due to CORS.

To solve this we need to use a proxy.
It is very simple to proxy the calls by just updating the UI package.json file.

Add the below configuration.

 "proxy": "http://localhost:4000"

Now the UI should be able to connect to backend APIs.

Generating the UI bundle and serving it through express.

Generating production UI bundle is dead simple
Run the below command.

npm run build

This is will create a folder named build in your frontend root directory containing index.html file along with JavaScript and CSS files.

This can be served using a static server like "serve" package.

BUT, the UI will not be able to reach backend servers.
BECAUSE, proxy feature is available only in development mode.

To solve this issue, we can serve the UI using same express server. YES you read it right. a single server to server both frontend and backend.

To do so,

Copy the build folder to backend folder
Add the following line in index.js file

app.use(express.static('build'));

We can give the absolute path also, just keeping it simple here :)

Now start the express server. At '/' path the UI will be served and at other paths, the APIs will be available.

Deploying the application in a container.

Till now we have developed and deployed the application on local machine.

If you are a docker guy, then we can quickly deploy this application in a container as well.

Create a Dockerfile. Dockerfile is a simple text file containing all the commands to create a docker image. The following is a docker file which uses alpine OS as a base image and exposes the port 4000.

Create a docker image Run the command to build the image

docker image build -t todoapp:1.0 .

Start the container Once the image is created, next step is to create a container. Run the command to create and start the container.

docker container run -d -p 8000:4000 todoapp:1.0

If the docker is running on a VM then the application can be accessed at VM-IP-Address:8000 eg. http://192.168.43.18:8000

Complete project is available at: https://github.com/SagarTrimukhe/todo-app

No Internet Connection wrapper for React apps

SagarTrimukhe — Fri, 14 May 2021 07:36:26 +0000

Imagine,
We have a web application that heavily depends on the backend server for information (eg. records in a table) and that information needs to be constantly updated. We might think to use some polling mechanism.

But if the data received from the server is directly stored in a React state variable and if the user loses the internet connection then there are chances of updating the state with empty data.

So, instead of showing empty data, we can show a message, something like "No internet connection."

How can we do that?

We can write a wrapper component and wrap the entry-level component. So whenever the internet connection is lost, a custom page/message can be shown.

Here I have used the navigator.onLine API to get the network status.

enough story, show me the code :)

Main component

import './App.css';
import NoInternetConnection from './NoInternetConnection'

function App() {
  return (
    <div className="App">
        <NoInternetConnection>
        <h1>My first post on DEV!!</h1>
        </NoInternetConnection>
    </div>
  );
}

export default App;

Wrapper component

import React, {useState, useEffect} from 'react';

const NoInternetConnection = (props) => {
    // state variable holds the state of the internet connection
    const [isOnline, setOnline] = useState(true);

    // On initization set the isOnline state.
    useEffect(()=>{
        setOnline(navigator.onLine)
    },[])

    // event listeners to update the state 
    window.addEventListener('online', () => {
        setOnline(true)
    });

    window.addEventListener('offline', () => {
        setOnline(false)
    });

    // if user is online, return the child component else return a custom component
    if(isOnline){
    return(
        props.children
    )
    } else {
        return(<h1>No Interner Connection. Please try again later.</h1>)
    }
}

export default NoInternetConnection;

Here is the demo.

That's it. By the way, this is my first post on DEV (damn! on the internet :)). Feedback is appreciated.

DEV Community: SagarTrimukhe

Why `livez` and `readyz` Matter for Kubernetes Health Probes

What we will cover

livez vs readyz

Liveness

Readiness

Visual summary

Why Probe Design Matters in Real Services

A Practical Implementation Pattern

livez pattern

readyz pattern

Example Kubernetes Probe Configuration

What Can Go Wrong in Distributed Systems

Intermittent Failures Need Intelligent Handling

Tuning Probe Frequency and Thresholds

Practical Guidelines

Closing Thoughts

Zero downtime, lossless migration of records to ElasticSearch indexes

Why Migrate?

Challenges

Planning the Migration

1. Understanding the Scale

2. Efficient Data Reading

3. Designing for Reliability

4. Deployment Strategy

5. Batch Processing and Parallel Writes

6. Handling Duplicates and Consistency

Results

Key Takeaways

Progress bar to show content read/remaining.

How to get the percentage?

Code:

Demo:

Generate YAML files in Golang.

Example 1: Very basic example of converting a struct to yaml.

Example 2: Providing custom tags.

Example 3: Structure with Arrays and Maps.

Example 4: Sometimes we want to actually create a YAML file.

Complete guide to deploy a simple full stack application in Docker

Table of contents

Creating a simple TODO app using React.

Creating a server using Express.js

Connecting frontend and backend.

Generating the UI bundle and serving it through express.

Deploying the application in a container.

No Internet Connection wrapper for React apps

How can we do that?

Main component

Wrapper component

`livez` vs `readyz`

`livez` pattern

`readyz` pattern