DEV Community

mohamed Tayel
mohamed Tayel

Posted on • Updated on

Elasticsearch Guide on Docker: A Comprehensive Walkthrough

Introduction

Elasticsearch is a powerful, open-source search and analytics engine designed for handling large volumes of data in real-time. Whether you need full-text search, structured search, or analytics, Elasticsearch offers a scalable solution capable of managing and analyzing large datasets quickly and efficiently.

Why Use Elasticsearch?

  1. Speed and Scalability: Elasticsearch is built on top of Apache Lucene and is known for its speed and scalability. It can handle large amounts of data and provide near real-time search results.
  2. Full-Text Search Capabilities: With Elasticsearch, you can perform complex full-text searches, including natural language processing, filtering, and ranking, making it ideal for applications that require advanced search functionalities.
  3. Real-Time Analytics: Elasticsearch allows for real-time data analysis, making it a popular choice for logging, monitoring, and observability use cases.
  4. Distributed Nature: Elasticsearch is designed to scale horizontally across multiple nodes, making it resilient and highly available.

How to Use Elasticsearch

Elasticsearch is commonly used in:

  • Web and Application Search: Enhancing search capabilities in websites, e-commerce platforms, and applications.
  • Log and Event Data Management: Storing, searching, and analyzing log and event data in real-time.
  • Data Analytics: Performing complex queries and aggregations on large datasets.

Installing Elasticsearch on Docker

Step 1: Set Up Elasticsearch with Docker

  1. Pull the Elasticsearch Docker Image:
   docker pull elasticsearch:8.10.2
Enter fullscreen mode Exit fullscreen mode
  1. Run Elasticsearch in a Docker Container:
   docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:8.10.2
Enter fullscreen mode Exit fullscreen mode

Step 2: Troubleshooting Common Issues

During the setup, several issues can arise. Here’s a summary of the troubleshooting steps:

  1. Port Conflict:

    • If Docker fails to start due to a port conflict, ensure that no other process is using ports 9200 or 9300. Use the following commands to find and stop conflicting processes:
     netstat -ano | findstr :9200
     taskkill /PID <PID> /F
    
  2. Empty Reply from Server:

    • If you encounter an "Empty reply from server" error, it may be due to Elasticsearch not responding properly. Ensure that the container is running:
     docker ps
    
  • Check the logs for any issues:

     docker logs elasticsearch
    
  1. Authentication Issues:

    • If you receive a security exception when accessing Elasticsearch, reset the password for the elastic user:
     docker exec -it elasticsearch bin/elasticsearch-reset-password -u elastic
    
  2. Security Configuration:

    • Elasticsearch may require HTTPS and authentication. Use the curl command with the -k flag to bypass certificate warnings:
     curl -k -u elastic:your_password https://localhost:9200
    
  3. Disk Space Warning:

    • If Elasticsearch logs a warning like low disk watermark [85%] exceeded, it means the disk space is running low. To resolve this, you can increase the available disk space using the following command in your Docker environment:
     wsl -d docker-desktop sysctl -w vm.max_map_count=262144
    
  • Additionally, you can adjust the disk watermark settings in the elasticsearch.yml configuration file:

     cluster.routing.allocation.disk.watermark.low: "5%"
     cluster.routing.allocation.disk.watermark.high: "10%"
     cluster.routing.allocation.disk.watermark.flood_stage: "2%"
    
  • After making changes, restart the Elasticsearch container to apply the settings:

     docker restart elasticsearch
    

Building a .NET Core POC with Elasticsearch

Step 1: Set Up the .NET Core Project

  1. Create a New Console Application:
   dotnet new console -n ElasticsearchPOC
   cd ElasticsearchPOC
Enter fullscreen mode Exit fullscreen mode
  1. Install the NEST Client:
   dotnet add package NEST --version 7.17.0
Enter fullscreen mode Exit fullscreen mode

Step 2: Connect to Elasticsearch and Index Data

  1. Connect to Elasticsearch:
   var settings = new ConnectionSettings(new Uri("https://localhost:9200"))
                  .DefaultIndex("my_index")
                  .BasicAuthentication("elastic", "your_password")
                  .ServerCertificateValidationCallback(CertificateValidations.AllowAll);

   var client = new ElasticClient(settings);
Enter fullscreen mode Exit fullscreen mode
  1. Index Sample Data:
   var person = new { Id = 1, Name = "John Doe", Age = 30, Occupation = "Software Developer" };
   var indexResponse = client.IndexDocument(person);
Enter fullscreen mode Exit fullscreen mode
  1. Search the Data:
   var searchResponse = client.Search<object>(s => s
       .Query(q => q.Match(m => m.Field("name").Query("John Doe")))
   );
Enter fullscreen mode Exit fullscreen mode

Step 3: Run the Application

dotnet run
Enter fullscreen mode Exit fullscreen mode

Conclusion

Elasticsearch is a powerful tool for search and analytics, and integrating it with .NET Core provides a robust solution for managing and querying data. By running Elasticsearch on Docker and connecting it with .NET Core, you can quickly deploy a scalable search solution with minimal setup.

Top comments (0)