DEV Community

Cover image for ELK Stack Mastery: Building a Scalable Log Management System
Cey
Cey

Posted on

ELK Stack Mastery: Building a Scalable Log Management System

Youtube Tutorial

Watch the tutorial

Overview

This project sets up an Elastic Cluster with 3 nodes using Virtualbox virtual machines. It includes the setup of Elasticsearch, Logstash, and Kibana (ELK stack) for log management and analysis.

Project Goals

  • Set up Elastic Cluster with all necessary components.
  • Create an index with a retention period of 10 days in Hot, 10 days in Cold, and 10 days in Frozen tiers.
  • Load logs using one of the methods listed in the setup.
  • Create a Dashboard with drilldown capabilities.

Prerequisites

  • VirtualBox installed on your system
  • Debian 12 ISO image
  • Sufficient system resources to run 3 VMs

VM Configuration

Create 3 VMs with the following specifications:

  1. elktest1 (Master + Data_Hot + Data_Content, Kibana, Logstash)
    • 8 GB RAM, 4 CPU, 40 GB storage
  2. elktest2 (Data_Cold, Logstash)
    • 8 GB RAM, 4 CPU, 40 GB storage
  3. elktest3 (Data_Frozen, Logstash)
    • 6 GB RAM, 3 CPU, 50 GB storage

Setup Instructions

1. VM Installation

  1. Download Debian 12 ISO:

    <https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.7.0-amd64-netinst.iso>
    
  2. Install Debian on each VM.

  3. In VM settings, change network from NAT to Bridged Adapter.

2. SSH Setup

Install SSH on each VM:

su -
apt-get update
apt-get install openssh-server
systemctl start ssh
systemctl enable ssh
Enter fullscreen mode Exit fullscreen mode

Reboot and get IP addresses:

reboot now
ip addr show
Enter fullscreen mode Exit fullscreen mode

Connect from host machine:

ssh <username>@<your_ip_address>
Enter fullscreen mode Exit fullscreen mode

3. Elasticsearch Installation

On all VMs:

apt install curl
curl -fsSL <https://artifacts.elastic.co/GPG-KEY-elasticsearch> | gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] <https://artifacts.elastic.co/packages/8.x/apt> stable main" | tee -a /etc/apt/sources.list.d/elastic-8.x.list
apt update
apt install elasticsearch
Enter fullscreen mode Exit fullscreen mode

4. Elasticsearch Configuration

On elktest1:

  1. Edit /etc/elasticsearch/elasticsearch.yml:

    cluster.name: elktestcluster
    node.name: elktest1
    node.roles: ["master","data_hot","data_content"]
    cluster.initial_master_nodes: ["elktest1"]
    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch
    network.host: 0.0.0.0
    http.port: 9200
    discovery.seed_hosts: ["elktest1"]
    xpack.security.enabled: true
    xpack.security.enrollment.enabled: true
    xpack.security.http.ssl:
      enabled: true
      keystore.path: certs/http.p12
    xpack.security.transport.ssl:
      enabled: true
      verification_mode: certificate
      keystore.path: certs/transport.p12
      truststore.path: certs/transport.p12
    http.host: 0.0.0.0
    
  2. Start Elasticsearch:

    systemctl start elasticsearch
    
  3. Reset elastic user password:

    /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic
    
  4. Generate enrollment tokens for other nodes:

    cd /usr/share/elasticsearch/bin
    ./elasticsearch-create-enrollment-token -s node
    

On elktest2 and elktest3:

  1. Reconfigure node with enrollment token:

    cd /usr/share/elasticsearch/bin
    ./elasticsearch-reconfigure-node --enrollment-token <your_enrollment_token>
    
  2. Edit /etc/elasticsearch/elasticsearch.yml:

    For elktest2:

    cluster.name: elktestcluster
    node.name: elktest2
    node.roles: ["data_cold"]
    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch
    network.host: 0.0.0.0
    http.port: 9200
    

    For elktest3:

    cluster.name: elktestcluster
    node.name: elktest3
    node.roles: ["data_frozen"]
    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch
    network.host: 0.0.0.0
    http.port: 9200
    xpack.searchable.snapshot.shared_cache.size: 30%
    
  3. Start Elasticsearch on both nodes:

    systemctl start elasticsearch
    

5. Index Lifecycle Management

Create ILM policy:

PUT _ilm/policy/elktestcluster_logs_policy
{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_size": "40gb",
                        "max_age": "10d"
                    }
                }
            },
            "warm": {
                "min_age": "10d",
                "actions": {
                    "forcemerge": {
                        "max_num_segments": 1
                    },
                    "allocate": {
                        "require": {
                            "data": "cold"
                        }
                    }
                }
            },
            "cold": {
                "min_age": "20d",
                "actions": {
                    "freeze": {},
                    "allocate": {
                        "require": {
                            "data": "frozen"
                        }
                    }
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Assign policy to index template:

PUT _index_template/elktestcluster_logs_template
{
  "index_patterns": ["elktestcluster-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "elktestcluster_logs_policy",
      "index.lifecycle.rollover_alias": "elktestcluster-logs"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

6. Logstash Setup

Install Logstash on all VMs:

apt install logstash -y
Enter fullscreen mode Exit fullscreen mode

Add logstash user to elasticsearch group:

sudo usermod -aG elasticsearch logstash
Enter fullscreen mode Exit fullscreen mode

Create Logstash pipeline configuration:

nano /etc/logstash/conf.d/elktestcluster-logs.con
Enter fullscreen mode Exit fullscreen mode

Add the following content:

input {
  file {
    path => [
      "/var/log/elasticsearch/elktestcluster*.json"
    ]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => ["<https://elktest1:9200>", "<https://elktest2:9200>", "<https://elktest3:9200>"]
    index => "elktestcluster-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "elastic"
    ssl => true
    cacert => "/etc/elasticsearch/certs/http_ca.crt"
  }
}
Enter fullscreen mode Exit fullscreen mode

Start Logstash on all VMs:

systemctl start logstash
Enter fullscreen mode Exit fullscreen mode

7. Kibana Setup

Install Kibana on one VM (preferably elktest1 or elktest2):

apt install kibana -y
Enter fullscreen mode Exit fullscreen mode

Reset kibana_system user password:

/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u kibana_system
Enter fullscreen mode Exit fullscreen mode

Configure Kibana:

nano /etc/kibana/kibana.yml
Enter fullscreen mode Exit fullscreen mode

Add/edit the following:

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["<https://elktest1:9200>", "<https://elktest2:9200>", "<https://elktest3:9200>"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana"
elasticsearch.ssl.verificationMode: none
Enter fullscreen mode Exit fullscreen mode

Start Kibana:

systemctl start kibana
Enter fullscreen mode Exit fullscreen mode

8. Accessing Kibana

Open a web browser and go to:

http://<your_kibana_machine_ip>:5601
Enter fullscreen mode Exit fullscreen mode

Use the Elasticsearch credentials:

  • Username: elastic
  • Password: elastic

Final Steps

  1. Create a data view from cluster logs in Kibana.
  2. Create a dashboard from the data view.

Kibana Snapshot

Congratulations! You have now set up a complete ELK stack for log management and analysis.

👋 While you are here

Reinvent your career. Join DEV.

It takes one minute and is worth it for your career.

Get started

Top comments (0)

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay