DEV Community

mohamed Tayel
mohamed Tayel

Posted on • Edited on

2

Elasticsearch Guide on Docker: A Comprehensive Walkthrough

Introduction

Elasticsearch is a powerful, open-source search and analytics engine designed for handling large volumes of data in real-time. Whether you need full-text search, structured search, or analytics, Elasticsearch offers a scalable solution capable of managing and analyzing large datasets quickly and efficiently.

Why Use Elasticsearch?

  1. Speed and Scalability: Elasticsearch is built on top of Apache Lucene and is known for its speed and scalability. It can handle large amounts of data and provide near real-time search results.
  2. Full-Text Search Capabilities: With Elasticsearch, you can perform complex full-text searches, including natural language processing, filtering, and ranking, making it ideal for applications that require advanced search functionalities.
  3. Real-Time Analytics: Elasticsearch allows for real-time data analysis, making it a popular choice for logging, monitoring, and observability use cases.
  4. Distributed Nature: Elasticsearch is designed to scale horizontally across multiple nodes, making it resilient and highly available.

How to Use Elasticsearch

Elasticsearch is commonly used in:

  • Web and Application Search: Enhancing search capabilities in websites, e-commerce platforms, and applications.
  • Log and Event Data Management: Storing, searching, and analyzing log and event data in real-time.
  • Data Analytics: Performing complex queries and aggregations on large datasets.

Installing Elasticsearch on Docker

Step 1: Set Up Elasticsearch with Docker

  1. Pull the Elasticsearch Docker Image:
   docker pull elasticsearch:8.10.2
Enter fullscreen mode Exit fullscreen mode
  1. Run Elasticsearch in a Docker Container:
   docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:8.10.2
Enter fullscreen mode Exit fullscreen mode

Step 2: Troubleshooting Common Issues

During the setup, several issues can arise. Here’s a summary of the troubleshooting steps:

  1. Port Conflict:

    • If Docker fails to start due to a port conflict, ensure that no other process is using ports 9200 or 9300. Use the following commands to find and stop conflicting processes:
     netstat -ano | findstr :9200
     taskkill /PID <PID> /F
    
  2. Empty Reply from Server:

    • If you encounter an "Empty reply from server" error, it may be due to Elasticsearch not responding properly. Ensure that the container is running:
     docker ps
    
  • Check the logs for any issues:

     docker logs elasticsearch
    
  1. Authentication Issues:

    • If you receive a security exception when accessing Elasticsearch, reset the password for the elastic user:
     docker exec -it elasticsearch bin/elasticsearch-reset-password -u elastic
    
  2. Security Configuration:

    • Elasticsearch may require HTTPS and authentication. Use the curl command with the -k flag to bypass certificate warnings:
     curl -k -u elastic:your_password https://localhost:9200
    
  3. Disk Space Warning:

    • If Elasticsearch logs a warning like low disk watermark [85%] exceeded, it means the disk space is running low. To resolve this, you can increase the available disk space using the following command in your Docker environment:
     wsl -d docker-desktop sysctl -w vm.max_map_count=262144
    
  • Additionally, you can adjust the disk watermark settings in the elasticsearch.yml configuration file:

     cluster.routing.allocation.disk.watermark.low: "5%"
     cluster.routing.allocation.disk.watermark.high: "10%"
     cluster.routing.allocation.disk.watermark.flood_stage: "2%"
    
  • After making changes, restart the Elasticsearch container to apply the settings:

     docker restart elasticsearch
    

Building a .NET Core POC with Elasticsearch

Step 1: Set Up the .NET Core Project

  1. Create a New Console Application:
   dotnet new console -n ElasticsearchPOC
   cd ElasticsearchPOC
Enter fullscreen mode Exit fullscreen mode
  1. Install the NEST Client:
   dotnet add package NEST --version 7.17.0
Enter fullscreen mode Exit fullscreen mode

Step 2: Connect to Elasticsearch and Index Data

  1. Connect to Elasticsearch:
   var settings = new ConnectionSettings(new Uri("https://localhost:9200"))
                  .DefaultIndex("my_index")
                  .BasicAuthentication("elastic", "your_password")
                  .ServerCertificateValidationCallback(CertificateValidations.AllowAll);

   var client = new ElasticClient(settings);
Enter fullscreen mode Exit fullscreen mode
  1. Index Sample Data:
   var person = new { Id = 1, Name = "John Doe", Age = 30, Occupation = "Software Developer" };
   var indexResponse = client.IndexDocument(person);
Enter fullscreen mode Exit fullscreen mode
  1. Search the Data:
   var searchResponse = client.Search<object>(s => s
       .Query(q => q.Match(m => m.Field("name").Query("John Doe")))
   );
Enter fullscreen mode Exit fullscreen mode

Step 3: Run the Application

dotnet run
Enter fullscreen mode Exit fullscreen mode

Conclusion

Elasticsearch is a powerful tool for search and analytics, and integrating it with .NET Core provides a robust solution for managing and querying data. By running Elasticsearch on Docker and connecting it with .NET Core, you can quickly deploy a scalable search solution with minimal setup.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up