DEV Community: Basu

Why Elasticsearch Should Not Be Preferred for Writing Data

Basu — Tue, 17 Dec 2024 18:26:18 +0000

Elasticsearch is one of the most popular tools for searching and analyzing data. Its ability to process massive datasets and deliver near real-time search results has made it a cornerstone for applications ranging from e-commerce platforms to monitoring systems. However, when it comes to writing data, Elasticsearch falls short in several critical areas. In this article, we’ll explore the challenges of writing data to Elasticsearch, explain why it’s not ideal for write-heavy use cases, and discuss better alternatives. 📊💾🚫

Understanding Elasticsearch’s Core Strength: Read-Heavy Workloads

Elasticsearch is optimized for fast querying and analytics, powered by its inverted index architecture. This makes it a perfect choice for use cases where:

Data is read and searched frequently.

Queries involve complex filtering, aggregations, or full-text search.

However, Elasticsearch was not built for heavy or continuous data writing. Its architecture and design decisions, while excellent for search, introduce inefficiencies for write-intensive applications. 🧠🔍💡

Challenges of Writing Data to Elasticsearch

High Resource Consumption

Indexing Overhead: Every time data is written to Elasticsearch, it goes through a series of processes:

Analysis: Text fields are broken into tokens using analyzers.

Inverted Index Creation: Tokens are mapped to their locations in documents.

Segment Management: Data is written to immutable segments on disk.

These steps consume significant CPU, memory, and disk resources. 📈🛠️🔄

Write Amplification: Elasticsearch continuously creates and merges segments to optimize queries. These operations amplify disk I/O, leading to slower writes and higher infrastructure costs. 🚀📂📉

Eventual Consistency

Elasticsearch is eventually consistent. This means that after writing data to one node, it may take time for the data to replicate across the cluster and become available for querying.

For applications requiring immediate consistency (e.g., financial transactions or inventory updates), this delay is unacceptable. ⏱️⚠️🔄

Frequent Updates Are Costly

Unlike traditional databases, Elasticsearch doesn’t modify existing documents directly.

Instead, it marks the old document as deleted and writes a new version of the document to a new segment. This process:

Consumes more disk space.

Triggers reindexing and segment merges, further taxing the system. 🗂️🖋️🔄

Limited Transactional Capabilities

Elasticsearch does not support ACID (Atomicity, Consistency, Isolation, Durability) transactions.

Concurrent writes or updates can lead to data conflicts or inconsistencies, making it unsuitable for applications requiring strict transactional guarantees. ❌🔒🔗

Performance Degradation Under Heavy Write Load

Write-heavy workloads can overwhelm Elasticsearch, causing cluster instability. Symptoms include:

High latencies for both writes and reads.

Increased memory usage leading to out-of-memory errors.

Node failures and slower query performance. 🐢🔥💥

Disk Usage Overhead

The inverted index and additional metadata (e.g., for replicas and segments) result in significant disk usage. Frequent writes, updates, and deletes exacerbate this problem, leading to higher storage requirements and costs. 💾📉📛

Example: Writing Data to Elasticsearch

Scenario

Imagine a real-time analytics system for tracking user interactions on a website. The system logs every page view, button click, and transaction as a separate document in Elasticsearch. With millions of interactions recorded daily, the following challenges arise: 🌐📱📊

High Write Throughput:

Each interaction generates a new document.

Elasticsearch’s indexing process struggles to keep up, resulting in slower writes and increased resource usage.

Frequent Updates:

If user interactions need updates (e.g., adding session details), Elasticsearch marks the old document as deleted and writes a new version. This doubles the write effort and bloats disk usage. 📈📝📛

Delayed Availability:

Newly indexed data isn’t immediately available for queries due to the refresh interval (default: 1 second). For real-time applications, this delay is problematic. 🕒⚡🚫

Better Alternatives for Write-Heavy Workloads

If your application involves high write throughput or frequent updates, consider these alternatives: 💡🔄💼

Relational Databases

Examples: MySQL, PostgreSQL

Advantages:

ACID-compliant transactions ensure data consistency.

Optimized for frequent updates and transactional writes.

Use Cases: Financial systems, inventory management, and applications requiring strong consistency.

NoSQL Databases

Examples: MongoDB, Apache Cassandra, DynamoDB

Advantages:

High scalability and fault tolerance.

Efficient for high write throughput.

Use Cases: Real-time analytics, distributed systems, and high-availability applications.

Message Queues or Streaming Systems

Examples: Apache Kafka, Amazon Kinesis

Advantages:

Handle massive write loads efficiently.

Provide durability and scalability for event-driven architectures.

Use Cases: Event logging, real-time data pipelines, and buffering writes before processing.

Time-Series Databases

Examples: InfluxDB, TimescaleDB

Advantages:

Designed for time-series data with high write speeds.

Built-in support for time-based queries and retention policies.

Use Cases: IoT data, monitoring systems, and performance analytics. 📊📆🚀

Optimizing Writes to Elasticsearch (If You Must)

For scenarios where Elasticsearch is necessary for search and analytics but still requires frequent data writes, consider the following optimizations: 🛠️📊✨

Use Bulk API:

Batch multiple write operations into a single request to reduce indexing overhead.

Adjust Refresh Interval:

Increase the refresh interval to delay making new data searchable, reducing resource usage during writes.

Shard Configuration:

Optimize the number of shards and replicas to balance performance and storage.

Pre-process Data:

Use tools like Apache Kafka or AWS Lambda to aggregate and transform data before writing to Elasticsearch.

Monitor and Scale:

Use Elasticsearch’s monitoring tools to identify bottlenecks and scale resources as needed. 📊📈🖥️

Conclusion

Elasticsearch is an exceptional tool for search and analytics, but it is not designed to handle write-heavy workloads efficiently. Its resource-intensive indexing process, eventual consistency model, and limited transactional capabilities make it unsuitable for applications that prioritize high write throughput or frequent updates. 📈🚀❌

Instead, consider using purpose-built databases and systems for writing data, and use Elasticsearch as a secondary layer for search and analytics. This approach ensures better performance, scalability, and cost-efficiency for your application. 💼💡✔️

Have you faced challenges with writing data to Elasticsearch? Share your experiences in the comments! 💬📣🤔

Records in java

Basu — Tue, 26 Mar 2024 05:27:59 +0000

Whenever we talk about boilerplate code in Java, there is always an argument that now IDEs are intelligent enough to generate sources. The problem comes when there is an addition in the fields. Record in Java 14 solves this problem at least for the Data Classes.
A record class in Java automatically has:

A private final field for each component of the record.
A public constructor whose signature matches the record components.
Public read accessor methods for each component of the record, with the same name and type as the component.
Implementations of equals(), hashCode(), and toString() methods.

Here is an example of how to declare and use a record:
public record Employee(String name, int age) {}

This record class Employee has two components: name and age. You can create an instance of Employee and access its components like this:

Employee employee = new Employee("John Doe", 30); System.out.println(employee.name()); // prints "John Doe" System.out.println(employee.age()); // prints 30

Note that records are intended to be used as simple data carriers, so they are not suitable for every situation. If you need more complex behavior, you should use a regular class.

Records in Java, while useful for creating simple data carrier classes, do have some limitations:

No Inheritance: Records cannot extend any other class, and they are implicitly final, so no other class can extend a record.
Limited Instance Variables: All instance variables must be declared in the record header, and these variables are automatically final. You cannot declare additional fields in the body of the record.
No Argument-less Constructor: Records automatically provide a constructor with arguments for all fields, but they do not provide a no-argument constructor. This can be a limitation when working with frameworks that rely on no-argument constructors.
No Mutability: Since all fields in a record are final, records are inherently immutable. This means once a record object is created, you cannot change its state.
Limited Use of Annotations: Annotations that are applicable to classes, methods, or fields may not be applicable to record components.
Limited Scope: Records are intended to be used as simple data carriers. If you need more complex behavior, such as methods that operate on the fields, you should use a regular class.

few good reads -

Records in java

Basu — Tue, 26 Mar 2024 05:25:54 +0000

A private final field for each component of the record.
A public constructor whose signature matches the record components.
Public read accessor methods for each component of the record, with the same name and type as the component.
Implementations of equals(), hashCode(), and toString() methods.

Here is an example of how to declare and use a record:
public record Employee(String name, int age) {}

This record class Employee has two components: name and age. You can create an instance of Employee and access its components like this:

Employee employee = new Employee("John Doe", 30); System.out.println(employee.name()); // prints "John Doe" System.out.println(employee.age()); // prints 30

Note that records are intended to be used as simple data carriers, so they are not suitable for every situation. If you need more complex behavior, you should use a regular class.

Records in Java, while useful for creating simple data carrier classes, do have some limitations:

No Inheritance: Records cannot extend any other class, and they are implicitly final, so no other class can extend a record.
Limited Instance Variables: All instance variables must be declared in the record header, and these variables are automatically final. You cannot declare additional fields in the body of the record.
No Argument-less Constructor: Records automatically provide a constructor with arguments for all fields, but they do not provide a no-argument constructor. This can be a limitation when working with frameworks that rely on no-argument constructors.
No Mutability: Since all fields in a record are final, records are inherently immutable. This means once a record object is created, you cannot change its state.
Limited Use of Annotations: Annotations that are applicable to classes, methods, or fields may not be applicable to record components.
Limited Scope: Records are intended to be used as simple data carriers. If you need more complex behavior, such as methods that operate on the fields, you should use a regular class.

few good reads -

Java groupingBy, Mapping, Counting

Basu — Sun, 24 Mar 2024 09:32:22 +0000

In my earlier post, we talked about various methods provided by java.util.stream.Collectors.
This post, I am going to talk about few other methods which are very powerful and useful.

Here is my previous post -
https://dev.to/bsknath/exploring-java-collectors-utilities-10ne

Exploring Java Collectors utilities

Basu — Fri, 22 Mar 2024 12:22:56 +0000

In Java, the java.util.stream.Collectors class provides a number of useful methods for performing reductions on streams. Here are some commonly used ones:

toList(): Collects all Stream elements into a List instance.

List<String> list = stream.collect(Collectors.toList());

toSet(): Collects all Stream elements into a Set instance.

Set<String> set = stream.collect(Collectors.toSet());

toMap(): Creates a Map instance from the Stream elements, with the first function serving as the map's keys and the second function as the values.

Map<String, Integer> map = stream.collect(Collectors.toMap(Function.identity(), String::length));

joining(): Concatenates all Stream elements into a String.

String joined = stream.collect(Collectors.joining(", "));

counting(): Counts the number of elements in the Stream.

Long count = stream.collect(Collectors.counting());

summingInt(), summingLong(), summingDouble(): Sums the Stream elements.

Integer sum = stream.collect(Collectors.summingInt(Integer::intValue));

maxBy(), minBy(): Finds the maximum or minimum Stream element according to a provided Comparator.

Optional<String> max = stream.collect(Collectors.maxBy(Comparator.naturalOrder()));

collectingAndThen(): Performs an additional finishing transformation.

List<String> unmodifiableList = stream.collect(Collectors.collectingAndThen(Collectors.toList(), Collections::unmodifiableList));

partitioningBy(): Partitions the Stream elements into a Map according to a Predicate.

Map<Boolean, List<String>> partitioned = stream.collect(Collectors.partitioningBy(s -> s.length() > 5));

These are just a few examples. The Collectors class provides many more utility methods for common tasks.

Stop, think about it. Are you using the right Data Structure?

Basu — Sat, 24 Jun 2023 04:40:19 +0000

A very trivial thing but knowing which data structure or implementation maintains the order of insertion, allows duplicates, or does not accept null helps prevent unexpected outcomes. Had a similar first-hand experience a few months ago. 💀

For example, lets take Map interface. There are three classes that implements Map in java - HashMap, LinkedHashMap and TreeMap.

♨️ HashMap doesn't maintain insertion order, LinkedHashMap does, TreeMap maintains order in sorted manner.
♨️ HashMap and LinkedHashMap allows one null key object (K,V), TreeMap doesn't (it may have multiple null values, never the keys).
♨️ HashMap and LinkedHashMap both have O(1) insertion and search. TreeMap, has O(logN). Bonus - TreeMap is implemented using RB-Tree, HM - LinkedLists, LHM - Doubley-LL. These implementation are the reason for these TC.

Coding standards that everyone must follow

Basu — Thu, 15 Dec 2022 18:23:05 +0000

Introduction

First rule of coding - write clean, beautiful and elegant code
Second rule of coding - write clean, beautiful and elegant code
Take time and effort to keep our code clean and pleasing for everyone to read
Follow these basic coding principles:
KISS – Keep It Simple, Stupid
SOLID Principles
Inversion of Control
DRY – Don’t Repeat Yourself.
YAGNI – You Ain’t Gonna Need It.
Law of Demeter.
Follow Boy’s Scout rule – leave code better than you found it.
Use “Design Patterns” whenever it makes sense to do it.
Avoid Premature Optimization – Premature Optimization is the root of all evil (or at least most of it) in programming. It adds unnecessary complexity in code and makes it hard to maintain in the long run. Always measure, analyze and then optimize.
Pay attention to all IDE generated warnings/suggestions

General

Favor composition over inheritance.
Create abstraction that separate higher level concepts from lower level details.
Variables and functions should be defined close to where they are used.
When a method uses getter/setter excessively from other class, consider moving that logic to that class.
Make sure all DTOs and Value Objects are immutable.
Avoid magic numbers, Hungarian notations in code. Use named constant instead of magic numbers

Core Java

Leverage Java 8’s stream API as much as you can instead of processing collection/array explicitly. For simple operations see if makes sense to process the collection without using stream API.
Use Lombok to generate boilerplate code. But do not overuse it to generate code that you don’t need
Avoid annotations like @SneakyThrow, @data, @synchronized from Lombok
Before using @Value annotation make sure you really need all functionality provided by @Value
Create new type instead of passing around complex collection types around e.g. instead of using List> create a type that represents that collection at higher abstraction
Use enums instead of constants
Investigate if there’s a reliable third-party library available before writing your own utility methods. Apache commons, Google Guava and Spring Utils are good example of third-party libraries that provides tons of useful functionality
Use meaningful and explanatory name that reveals the intention of variable/class/method
If name requires comment, it doesn’t reveal its intention. Consider re-naming such variables
Use long and descriptive name but do not use very long variable name that it starting to look like sentence
Avoid unnecessarily acronymization of variable name. A variable called “timeSheet” is much more descriptive and meaningful than “ts”
Class Names – Classes and Objects should have noun or noun phrase names. E.g. TimeSheet, WorkEntry. It should not be a verb.
Method Names – Method should have verb or verb phrase names. E.g. approveTimeSheet, submitWorkEntry
Event Class Names - Should be verb in past tense E.g.TimeSheetSubmitted

Comments

Avoid comments. We should strive at writing self-explanatory code that requires minimum to no comments. Ideally, our code should not have any comments. If your code needs commenting, you might be doing something wrong
Do not comment change history, user story in source code – it should go as part of your commit message. All meta-data about code like author, last modified time, US number, Defect Numbers should not be part of code
Do not use Javadoc comment that says nothing more than function signature
Javadoc style comments which are mainly used for documentation should be limited for library functionality
Never ever comment out code. Just delete it.
Comment should be used only when code cannot say for itself clearly

Functions

Functions should be small. This applies to code blocks like if, while, switch blocks etc.
Function should do one thing only
If your function has more than a few arguments, see if your method has tight coupling with other classes. If you do need those arguments, try to wrap them in separate class
Avoid Boolean flag as method parameter that is used as a flag to choose behavior in method
Return java.util.Optional instead of "return null" from method

Tests

Make sure your tests cover all scenarios specified.
Write unit test for everything that could possibly break
Use IDE coverage tool to make sure you are covering all critical code path and branches
Test all boundary conditions
Write test (or tests) for each reported bug