DEV Community: Kaoutar

Building a Scalable & Secure ELK Stack Infrastructure – A Practical Guide

Kaoutar — Fri, 14 Mar 2025 14:09:06 +0000

Managing logs efficiently is critical for monitoring, troubleshooting, and security compliance in any modern IT environment. The ELK stack (Elasticsearch, Logstash, Kibana) provides a powerful, scalable, and real-time logging solution, whether deployed on-premise or in the cloud.

In this guide, I’ll walk you through how to design and deploy a centralized log management system using ELK, covering architecture, best practices, and key optimizations.

Why Centralized Logging?

Handling logs across multiple applications and servers can be a nightmare. A centralized logging system helps:

Aggregate logs from multiple sources
Ensure real-time monitoring and alerting
Improve security compliance (e.g., encryption, access control)
Optimize performance and storage

Architecture Overview: Key Components

A robust ELK architecture consists of multiple components working together:

Filebeat → Collects logs from various sources
Logstash → Processes, filters, and enriches log data
Elasticsearch → Stores and indexes logs for fast retrieval
Kibana → Provides real-time dashboards and analytics
Backup & Security Measures → Ensures compliance and disaster recovery

Example Deployment (On-Premise or Cloud-based)

A large-scale financial institution handling millions of transactions daily requires centralized log management to track system activity, detect fraud, and ensure compliance with regulations like PSI-DSS and GDPR. The logging infrastructure must be scalable, resilient, and secure, capable of processing high volumes of structured and unstructured logs from multiple applications, security tools, and databases.

To meet these demands, the ELK stack is deployed across dedicated virtual machines (VMs) or containers with optimized resource allocation:

🖥 Virtual Machine for Elasticsearch
Elasticsearch is the core of the ELK stack, handling indexing and search. It is resource-intensive, especially when processing a large volume of logs.

Recommended Specs:

vCPU: 16 vCPUs (scalable based on log volume and queries)
RAM: 64 GB (benefits from large heap size)
Storage: 2 TB SSD (adjustable based on retention and log volume)
OS: RHEL, Ubuntu, or any Linux-based production-optimized OS

-> Justification:
Elasticsearch requires high memory and fast storage for indexing and queries. Depending on data volume, adding more storage or clustering nodes can enhance fault tolerance and scalability.

🖥 Virtual Machine for Logstash
Logstash processes, filters, and enriches logs before forwarding them to Elasticsearch. Complex pipelines can be CPU and memory-intensive.

Recommended Specs:

vCPU: 8 vCPUs
RAM: 16 GB
Storage: 500 GB SSD (lower storage needs than Elasticsearch)
OS: RHEL or Ubuntu

-> Justification:
CPU and RAM usage depends on log volume and pipeline complexity. Heavy filtering or data enrichment increases resource consumption.

🖥 Virtual Machine for Kibana
Kibana provides visualization and analytics, but it requires fewer resources compared to Elasticsearch and Logstash.

Recommended Specs:

vCPU: 4 vCPUs
RAM: 8 GB
Storage: 100 GB SSD (minimal storage requirements)
OS: RHEL, Ubuntu, or any Linux-based system

-> Justification:
Kibana mainly handles dashboard rendering and visualization queries. Resource needs increase with more users and complex visualizations.

📌 Filebeat (Lightweight Log Shipper)
Filebeat is a lightweight agent that collects and forwards logs to Logstash or Elasticsearch.
-> Justification:
Filebeat is resource-efficient and has minimal processing overhead. It can be deployed on multiple servers depending on the log sources.

🖥 Virtual Machine for Backup Server
A backup server stores Elasticsearch snapshots and automated backups to ensure data integrity and recovery.

Recommended Specs:

vCPU: 4 vCPUs
RAM: 16 GB (low RAM requirements since backups are storage-intensive)
Storage: 4 TB HDD (encrypted)
OS: Ubuntu Server, CentOS, or RHEL

-> Justification:

Backup storage does not require fast SSDs—high-capacity HDDs are sufficient.
Backup strategies include snapshots, rotation policies, and periodic recovery tests.

Key Benefits of This Architecture

Resource Isolation → Each component runs on its own VM, ensuring one service’s workload doesn’t impact others.
Scalability → Each component can be scaled independently (e.g., Elasticsearch can be expanded as log volume grows).
High Availability & Fault Tolerance → Elasticsearch can run as a cluster with multiple nodes, and backups ensure data security.
Security Best Practices → Separate VMs allow granular firewall rules and network policies, restricting communication between components.

Data Flow & Processing

Step 1: Log Collection with Filebeat

Filebeat runs on source servers (applications, databases, containers).
Sends logs to Logstash or directly to Elasticsearch.

Step 2: Processing with Logstash

Filters and enriches logs (e.g., adds metadata, geo-location, or obfuscates sensitive data).
Outputs to Elasticsearch for storage.

Step 3: Indexing & Storage in Elasticsearch

Optimized for fast queries and scalable storage.
Index lifecycle management ensures log retention policies.

Step 4: Visualization & Alerting with Kibana

Dashboards provide real-time insights.
Alerts notify teams of anomalies or system failures.

Security & Compliance Considerations

Ensuring a secure logging infrastructure is crucial for data protection and regulatory compliance (e.g., SOC2, GDPR, PSI-DSS). Key practices include:

Data Encryption

In Transit: All communication between Filebeat, Logstash, Elasticsearch, and Kibana is secured using SSL/TLS certificates to prevent data interception.
At Rest: Log indices in Elasticsearch are encrypted using the X-Pack security module, and physical disks on servers are also encrypted to protect stored data.

Authentication & Access Control

Role-Based Access Control (RBAC): Permissions are managed based on user roles, ensuring restricted access to logs only for authorized personnel.
LDAP / Active Directory Integration: Centralized authentication management allows seamless user provisioning and control.
Multi-Factor Authentication (MFA): Enforced for Elasticsearch and Kibana administrators to enhance security against credential theft.

Access Logging & Monitoring

Audit Logging: User activities and system interactions in Elasticsearch and Kibana are logged for traceability.
Unauthorized Access Monitoring: Failed login attempts and unusual modifications are actively monitored to detect potential security threats.

Additional Security Measures

Log Retention Policies: Automatic log purging using Index Lifecycle Management (ILM) ensures compliance with data retention regulations.
Backup & Disaster Recovery: Regular Elasticsearch snapshots ensure data availability and protection against loss or corruption.
Network Security & Isolation: Strict firewall rules and network segmentation prevent unauthorized access between components.

Deployment & Automation Best Practices

Infrastructure as Code (IaC)

Automate deployment using Ansible, Terraform, or CI/CD tools.

Scaling Strategy

Deploy Elasticsearch as a cluster for high availability.
Optimize sharding & indexing for better performance.
Use log filtering to reduce unnecessary data ingestion.

Monitoring & Performance Tuning

Leverage Elasticsearch monitoring tools (Kibana Stack Monitoring, Grafana).
Tune heap size & JVM settings for optimal resource allocation.

Why You Should Build a Scalable ELK Stack?

A well-designed ELK stack enables organizations to streamline log management, improve security, and gain valuable insights from their data. Whether deployed on premise or cloud, following best practices ensures scalability, performance, and compliance.

Terraform

Kaoutar — Mon, 27 May 2024 11:27:32 +0000

Utilizing reusable, shareable, human-readable configuration files, HashiCorp Terraform is an infrastructure as code (IaC) software solution that enables DevOps teams to automate infrastructure provisioning. Infrastructure provisioning may be automated using this technology in both on-premises and cloud scenarios.

Infrastructure as Code

The process of supplying and controlling IT infrastructure using coding is known as "infrastructure as code" IaC enables DevOps teams to programmatically and automatically manage, monitor, and provide the resources they require, as opposed to manual infrastructure management, when each necessary resource is manually set by a human.

Teams may use Terraform to describe and provision all of the infrastructure's parts using code. Config files, which are readily shared, reused, and versioned, contain the code. In order to manage the whole cloud or data center infrastructure and its resources over the course of their lifespan, the files aid in the creation of a standardized workflow.

Declarative configuration files for Terraform define the final state of the infrastructure. Instead of having to give detailed instructions, which is a laborious and time-consuming procedure, to construct the necessary infrastructure resources, the tool handles the underlying
logic itself.

It is simple for DevOps teams to accomplish the following since the files codify the application programming interfaces (APIs) for cloud platforms and other services:

Any cloud provider may be used to provision resources.
Put up compliance and security barriers to harmonize the infrastructure.
Use defined and dependable procedures to ensure consistency in the provisioning, sharing, and reuse of infrastructure.
Integrate VCS, ITSM, and CI/CD with the self-service infrastructure. Terraform is capable of managing low-level components like DNS records as well as highlevel infrastructure elements like computation, storage, and networking resources.

Additionally, it may be used to automatically setup servers, databases, and firewall settings. Teams may manage infrastructure using their favorite programming language, including TypeScript, Python, Go, C#, and Java, with the use of a Cloud Development Kit for Terraform (CDKTF).

How Terraform works

The ability to construct declarative configuration files using Terraform is made possible by the widely used APIs that are accessible from all major cloud service providers. The Terraform Registry has a list of these suppliers.

Teams may utilize the modules, policy libraries, and tasks included in the Registry to easily install standard infrastructure setups and maintain them automatically with code. The process for Terraform consists of three steps:

Write
A user defines the necessary resources in configuration files at this step. These resources might be spread out throughout several on-premises or cloud settings, as well as between various suppliers and services.
Plan
This step starts once the user examines and approves the necessary phases. The steps that will be taken to develop or upgrade the infrastructure are described in the execution plan that Terraform produces in this case.
Apply
Before Terraform makes modifications to the infrastructure, the plan must be approved by the user. After receiving permission, Terraform executes the suggested procedures in the specified sequence. Before making changes, it will always consider resource dependencies.
For example, in the event that a user decides to increase the number of virtual machines in a VPC (virtual private cloud), Terraform will first rebuild the VPC before scaling up the VMs

Use Cases

IaC is Terraform's most popular use case. Terraform infrastructure deployments are simple to integrate with current CI/CD procedures.
Teams may use Terraform, for instance, to automatically update member pools for load balancing and other crucial networking activities.
For provisioning across many clouds, Terraform is also helpful. Development teams may use Terraform to provide load balancers in Google Cloud, manage Active Directory (AD) resources in Microsoft Azure, and deploy serverless operations in AWS.
Manage Kubernetes clusters in any public cloud (AWS, Azure, Google).
Enforce policy-as-code before infrastructure components are developed and deployed.
Use secrets and credentials in Terraform setups automatically.
Import current infrastructure into a blank Terraform workspace to codify it.
Transfer state to Terraform to protect it and make it simple for authorized collaborators to access it.

Kibana Fundamentals

Kaoutar — Mon, 27 May 2024 11:06:27 +0000

A data visualization platform that is primarily used to analyze massive volumes of logs in the form of line graphs, bar graphs, pie charts, heat maps, region maps, coordinate maps, gauge, goals, and other visual representations. It is simple to foresee or notice changes in trends of mistakes or other noteworthy events of the input source thanks to the display.

Key features

Visualization There are several simple methods to view data with Kibana. Examples of some of the ones that are frequently used include heat maps, pie charts, line graphs, vertical bar charts, and horizontal bar charts.

Dashboard When the visualizations are prepared, they may all be arranged on the Dashboard, which is a single board. You can get a good sense of what is going overall by watching many portions at once.

Dev Tools Using dev tools, you may work with your indexes. Dummy indexes may be added using dev tools by beginners, and they can also add, amend, and remove data and utilize the indexes to generate visualization.

Reports
You may export all of the data from dashboards and visualizations as reports (in CSV format), embed them in code, or share them with others through URLs.
Search and Filter Query
You may use filters and search queries to find the information you need from a dashboard or visualization tool for a certain input.

Plugins
Third-party plugins can be added to Kibana to bring new visualizations or other UI additions.
Regional and Coordinate Maps
In Kibana, a coordinate and region map aids in displaying the visualization on the geographical map while providing a truthful representation of the data.

Timelion Another visualization tool that is generally used for time-based data analysis is Timelion, sometimes known as a timeline. When working with a timeline, we must employ a straightforward expression language that enables us to connect to the index and do computations on the data to provide the desired results. Comparing data to the prior cycle in terms of week, month, etc. is more helpful.

Canvas Another useful element of Kibana is canvas. You may visualize your data using a canvas by using different color schemes, shapes, messages, and numerous pages, also known as a workpad

Logstash Fundamentals
Elasticsearch Fundamentals

Logstash Fundamentals

Kaoutar — Mon, 27 May 2024 10:41:49 +0000

Logstash is the next generation logging framework, it functions as a centralized framework for log collecting, processing, storage, and search, it can normalize data from several sources and dynamically combine them into the destinations of your choosing.
With a wide range of input, filter, and output plugins, Logstash enables any sort of event to be enhanced and altered with many native codecs that simplify the ingestion process. By utilizing more data, both in terms of volume and diversity, Logstash offers insights.
Files, Syslog, TCP/UDP, stdin, and many other input methods are just a few of the input sources that Logstash may accept. In order to change the events in the gathered logs, a wide variety of filters may be used.

One or more pipelines are used to arrange the processing. One or more plug-ins receive or gather data in each pipeline, which is subsequently added to an internal queue that is typically minimal and only kept temporarily in memory, but you may customize it to be bigger and permanently saved to disk to increase dependability and resilience.

The processing threads read the data from the waiting file in micro-lots and process it using one of the configured plug-in filters sequentially. Logstash is ready for use and has a wide variety of plug-ins that focus on particular sorts of processing. This is how data is analyzed,
processed, and expanded.
Once the data has been processed, the processing threads send the data to the appropriate plug-ins for output, which are responsible for formatting and transferring the data (for example, to Elasticsearch).

Key features

Event Object
It is Logstash's primary object and contains all of the data flow for the pipeline. This object is used by Logstash to retain the incoming data and add any new fields produced during the filtering process.
Pipeline
It consists of Logstash's data flow phases from input to output. The pipeline receives the input data and processes it in the form of an event. then transmits to an output location in the preferred format for the user or end system.
Input
The data must first pass through this step of the Logstash pipeline before it can be processed further. To obtain data from many systems, Logstash provides a variety of plugins. File, Syslog, Redis, and Beats are some of the plugins that are utilized the most frequently.
Filter
The intermediate step of Logstash is where the real event processing happens. To assist the developer in parsing and transforming the events into a desired format, Logstash provides a variety of plugins. Grok, Mutate, Drop, Clone, and Geoip are some of the filter plugins that are most often used.
Output
The output events from Logstash may now be formatted into the structure needed by the destination systems at this last step of the pipeline. Finally, it uses plugins to deliver the output event to the target when processing is finished. Elasticsearch, File, Graphite, Statsd, and other plugins are some of the ones that are most frequently utilized.

Kibana Fundamentals
Elasticsearch Fundamentals

Elasticsearch Fundamentals

Kaoutar — Mon, 27 May 2024 10:34:21 +0000

Elasticsearch is an Apache Lucene-based search engine. It is a real-time, distributed, multitenant-capable full-text search engine, it offers a RESTful API based on JSON documents, it can be used for full-text search, structured search, analytics, or all three. One of its most important advantages is the capacity to search quickly by indexing the text to be searched. Many search engines have long been available with the option to search by timestamp or precise quantities, Elasticsearch distinguishes itself by running full-text searches, managing synonyms, and evaluating items based on relevancy.
Furthermore, it may provide real-time analytics and aggregation from the same data, it outperforms other search engines in this area.
Elasticsearch is widely used in many large corporations. Here are some examples of applications:

Elasticsearch is used by Netflix to deliver millions of messages to clients every day via various channels like as email, push alerts, text messaging, phone calls, and so on.
Salesforce has created a bespoke plugin on top of Elasticsearch that collects Salesforce log data, allowing for insights on organizational usage trends and user activity.
Elasticsearch is used by the New York Times to store all 15 million articles written over the previous 160 years. This allows for fantastic archival search capabilities.
Elasticsearch is used by Microsoft for search and analytics capabilities in a variety of products, including MSN, Microsoft Social Listening, and Azure Search.
Elasticsearch was utilized by eBay to provide a versatile search platform. Elasticsearch is not solely utilized by major enterprises, it is also used by many startups and small businesses. Elasticsearch's appeal is that it can operate on a laptop or expand up to hundreds of servers and petabytes of data.

Key features

It offers statistics and real-time search for your data
Elasticsearch can run on anything from a basic laptop to hundreds of nodes and is a distributed system.
It may be used to deploy multitenant, highly available clusters. It automatically rearranges and rebalances data upon the addition of a new node or the failure of a node.
Elasticsearch distributes the processing of queries and data storage among many data nodes. Scalability, dependability, and performance are all improved.
Data in an Elasticsearch cluster is duplicated across several nodes, so even if one node fails, it is still accessible.

Elasticsearch can comprehend and search natural language text since it is built on top of Lucene, a full-text search technology.
Rather of storing documents as rows in a table, Elasticsearch saves them as JSON.

Elasticsearch makes use of a JSON-based query language rather than a SQL-based one

Elasticsearch does not enable JOINS between tables, in contrast to relational databases.
Word aggregations, geographic searches, and support for scripting languages are just a few of Elasticsearch's built-in analytical features.
In relational databases, a schema is the equivalent of a mapping in Elasticsearch. Elasticsearch will automatically assign a data type to a document field if one isn't explicitly specified if it hasn't before.

Key Components

Cluster
A cluster is a grouping of one or more nodes that collectively contains all of the data and offers federated indexing and search capabilities across all nodes. Each node in a cluster should be given a distinct name.
Node
Node generally refers to a server that functions as a cluster member. A node is an instance, not a machine, in Elasticsearch context. This implies that several nodes can operate on a single machine. An Elasticsearch instance consists of one or more cluster-based nodes. By
default, a node also starts up when an Elasticsearch instance does.
A distinctive name is used to identify each node. A random UUID is used as the node identification at initialization if the node identifier is not supplied. The 'cluster.name' field is a part of every node setup. The cluster will automatically create, with each node having the same 'cluster.name' upon launch.
A node must carry out several tasks:
- Storing data
- Processing data (indexing, searching, aggregation, etc.)
- Preserving the cluster's health

In a cluster, all these operations are available to every node. Elasticsearch offers the option to distribute duties among several nodes. This makes scaling, optimizing, and maintaining the cluster simple.
The three primary ways to setup an Elasticsearch node are as follows:
Elasticsearch master node controls the Elasticsearch cluster by processing one cluster state at a time and broadcasting the state to all other nodes. The master node is in charge of all clusterwide operations, including the creation and deletion of indexes.
Elasticsearch data node contains data and the inverted index. This is the default configuration for nodes.
Elasticsearch client node serves as a load balancer that routes incoming requests to various cluster nodes.

Port 9200 and Port 9300
Two primary ports are used by the Elasticsearch architecture for communication:
Filtering queries originating from outside the cluster is done using port 9200. This procedure responds to queries sent using REST APIs, which are used for querying, indexing, and other functions.
For inter-node communication, use port 9300. The transport layer is where this happens.
Shards of Elasticsearch
Shards are the fundamental pieces of indexing that make up the Elasticsearch architecture. They are compact and scalable.
You may store an unlimited number of documents on each index. Elasticsearch might, however, break if an index exceeds the hosting server's storage restrictions. Sharding, or dividing indexes into smaller parts, solves this problem. You can spread activities among shards to increase performance as a whole. The number of shards you produce after generating an index is up to you. Every shard functions as a separate Lucene index that can be hosted anywhere in the cluster.

Elasticsearch Replicas
Replicas in Elasticsearch are copies of index shards. For backup and recovery reasons, replicas are utilized as a fail-safe strategy. Duplicates are never added on the node hosting the primary (original) shards, replicas are kept at several places to assure availability. After the index is formed, replicas may be defined and made in any number and because of this, you can store more replicas than primary shards.
Index
Index is a container to store data similar to a database in the relational databases. An index contains a collection of documents that have similar characteristics or are logically related. If we use an e-commerce website as an example, there will be indexes for customers, items, and so on.
We can create as many indexes as necessary inside a single cluster, depending on our needs.
Elasticsearch searches an index rather than the text directly. As a result, it enables quick search results. Instead of searching every word on every page of the book, you may scan the index at the back of a book to find pages in the book that are relevant to a term.
The name "Inverted Index" refers to this form of index because it converts a word-centric data structure (words->pages) to a page-centric data structure (pages->words). Elasticsearch has support for inverted indexes, which are built and maintained using Apache Lucene

Document Document is the piece indexed by Elasticsearch in the JSON format. Any number of documents can be added to an index.

Kibana Fundamentals
Logstash Fundamentals