DEV Community

Cover image for SOP for Installation of ELK Stack with Filebeat on AWS Servers
kaustubh yerkade
kaustubh yerkade

Posted on

SOP for Installation of ELK Stack with Filebeat on AWS Servers

Standard Operating Procedure (SOP): Installation of ELK Stack with Filebeat on AWS Servers

1. Purpose

This SOP outlines the step-by-step process for installing and configuring the ELK (Elasticsearch, Logstash, and Kibana) stack with Filebeat on AWS servers. The document includes prerequisites, system requirements, cluster setup, and AWS-specific configurations.

2. Prerequisites

  • AWS Account: Access to an active AWS account with appropriate IAM permissions.
  • Networking:
    • VPC created for deployment.
    • Subnets for public/private resources.
    • Security groups with the required ports opened (refer to the Ports section).
  • System Access:
    • SSH key pair for accessing EC2 instances.
    • Admin/root privileges on servers.
  • Software Tools:
    • AWS CLI installed and configured locally.
    • Terraform/CloudFormation (optional for automation).
    • Remote terminal such as PuTTY or a compatible SSH client.

3. System Requirements

Compute Requirements

Component Instance Type Minimum vCPU Minimum Memory
Elasticsearch t3.medium or higher 2 8 GB
Logstash t3.medium or higher 2 4 GB
Kibana t3.small or higher 1 2 GB
Filebeat Agents t2.micro or higher 1 1 GB

Storage Requirements

Component Disk Type Minimum Storage
Elasticsearch SSD (gp3) 50 GB
Logstash SSD (gp3) 10 GB
Kibana General HDD 10 GB

Ports to Open

Service Protocol Port
Elasticsearch HTTP/HTTPS 9200
Kibana HTTP 5601
Logstash TCP/UDP 5044
Filebeat Outbound HTTP 9200

4. Cluster Creation

AWS Perspective

  1. VPC Setup:

    • Create a VPC and enable DNS hostnames.
    • Set up two subnets (public and private) for different components.
    • Create an Internet Gateway (IGW) and attach it to the VPC.

  2. Security Groups:

    • Allow inbound traffic for required ports in the security group associated with EC2 instances.
    • Restrict access to trusted IP ranges for Kibana and Elasticsearch.

  3. EC2 Instances:

    • Launch EC2 instances for Elasticsearch, Logstash, Kibana, and Filebeat.
    • Use Amazon Linux 2 or Ubuntu 20.04 for compatibility.
    • Allocate Elastic IPs for external access if required.

  4. IAM Role:

    • Attach an IAM role with S3 and CloudWatch permissions for backup and monitoring.

Landing Zone Perspective

  1. Use AWS Landing Zone (if applicable) to standardize account setup and ensure governance.
  2. Configure logging and monitoring through AWS CloudTrail and AWS CloudWatch.
  3. Ensure tagging standards are applied for resource identification.

5. Installation Steps

Elasticsearch

  1. Install Elasticsearch: 
   sudo apt update
   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   sudo apt install apt-transport-https
   echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
   sudo apt update && sudo apt install elasticsearch
Enter fullscreen mode Exit fullscreen mode

  1. Configure Elasticsearch:

    • Update /etc/elasticsearch/elasticsearch.yml with the following: 
     network.host: 0.0.0.0
 cluster.name: elk-cluster

  • Restart the service:

     sudo systemctl enable elasticsearch
 sudo systemctl start elasticsearch

Logstash

  1. Install Logstash: 
   sudo apt install logstash
Enter fullscreen mode Exit fullscreen mode

  1. Configure Logstash:

    • Create a configuration file at /etc/logstash/conf.d/logstash.conf: 
     input {
     beats {
         port => 5044
     }
 }

 output {
     elasticsearch {
         hosts => ["http://<elasticsearch-IP>:9200"]
         index => "logs-%{+YYYY.MM.dd}"
     }
 }

  • Start the service:

     sudo systemctl enable logstash
 sudo systemctl start logstash

Kibana

  1. Install Kibana: 
   sudo apt install kibana
Enter fullscreen mode Exit fullscreen mode

  1. Configure Kibana:

    • Update /etc/kibana/kibana.yml: 
     server.host: "0.0.0.0"
 elasticsearch.hosts: ["http://<elasticsearch-IP>:9200"]

  • Restart the service:

     sudo systemctl enable kibana
 sudo systemctl start kibana

Filebeat

  1. Install Filebeat: 
   sudo apt install filebeat
Enter fullscreen mode Exit fullscreen mode

  1. Configure Filebeat:

    • Update /etc/filebeat/filebeat.yml: 
     filebeat.inputs:
 - type: log
   paths:
     - /var/log/*.log

 output.logstash:
   hosts: ["<logstash-IP>:5044"]

  • Start Filebeat:

     sudo systemctl enable filebeat
 sudo systemctl start filebeat

6. Post-Installation Verification

  1. Elasticsearch:

    • Verify connectivity: 
     curl http://<elasticsearch-IP>:9200

  2. Logstash:

    • Check for log ingestion in Elasticsearch.

  3. Kibana:

    • Access the Kibana UI at http://<kibana-IP>:5601 and configure an index pattern.

  4. Filebeat:

    • Confirm logs are being sent to Logstash and indexed in Elasticsearch.

7. Maintenance and Monitoring

  1. Backup:
    • Configure snapshot backups to an S3 bucket.
  2. Monitoring:
    • Use Elastic Stack’s monitoring features or integrate with AWS CloudWatch.
  3. Scaling:
    • Use AWS Auto Scaling Groups for horizontal scaling of Elasticsearch nodes.

8. Troubleshooting

  • Logs:
    • Check service logs for Elasticsearch (/var/log/elasticsearch), Logstash (/var/log/logstash), and Kibana (/var/log/kibana).
  • Network Issues:
    • Verify VPC routing tables and security group configurations.
  • Resource Bottlenecks:
    • Monitor instance metrics and upgrade instance types or increase storage as needed.

This SOP ensures a standardized deployment of the ELK stack with Filebeat on AWS, facilitating efficient log management and monitoring.

Top comments (0)