Deploy Kafka + Filebeat + ELK - Docker Edition - Part 2

Introduction

This article is the last part of a two part series where we will deploy ELK stack using docker/docker-compose.
In this article, we will be configuring Logstash, Elasticsearch and Kibana. If you haven't gone through the previous article, check out this link

Logstash

Logstash is a server-side data processing pipeline that consumes data from different sources and send it to elasticsearch. We touched on its importance when comparing with filebeat in the previous article.
Now to install logstash, we will be adding three components

• a pipeline config - logstash.conf
• a setting config - logstash.yml
• docker-compose file

Pipeline configuration will include the information about your input (kafka in our case), any filteration that needs to be done, and output (aka elasticsearch).
Create a folder named pipeline and add this configuration file to it.

Logstash pipeline - logstash.conf

input {
    kafka{
        bootstrap_servers => "KAFKA_SERVERS_IP:KAFKA_SERVERS_PORT"
        topics => ["applogs"]
    }
}
filter {
    json {
        source => "message"
      }
}
## Add your filters / logstash plugins configuration here
output {
        elasticsearch {
    hosts => ["ELASTICSEARCH_IP:9200"]
    user => 'elastic'
    password => 'somesecretpassword'
    }

As you can see, In the input section, we are listening to kafka on topic applogs.
In my case, I have added a filter which will jsonify the data that comes under the key named "message". There are several plugin filters to choose from.
The output is redirected towards elastic search, with username password configured for authentication.
Create a folder named settings and add this configuration file to it

logstash settings - logstash.yml

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
path.logs: /var/log/logstash
config.reload.automatic: true
log.level: debug
xpack.monitoring.enabled: false

The docker-compose file looks like this:

logstash docker-compose

version: '2'

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:5.6.3
    ports:
      - "10000:10000"
    volumes:
      - ./settings/:/usr/share/logstash/config/
      - ./pipeline/:/usr/share/logstash/pipeline/
    container_name: logstash                          

Here the configuration files mentioned above have been mounted.

To run the above file

docker-compose up -d

Now, You will get an error as we have not setup elastic search yet. Now lets go ahead and install Elasticsearch and Kibana

Elasticsearch and Kibana

Kibana is an open source user interface that helps you monitor and visualise data, which in our case, is being provided by elasticsearch.
Elasticsearch, as the name suggests, is a search and analytics engine for different types of data.

Elasticsearch and Kibana docker-compose

version: "2"
services:
 elasticsearch:
  image: "docker.elastic.co/elasticsearch/elasticsearch:7.13.2"
  container_name: elasticsearch
  environment:
   - discovery.type=single-node
   - cluster.routing.allocation.disk.threshold_enabled=true
   - cluster.routing.allocation.disk.watermark.low=65%
   - cluster.routing.allocation.disk.watermark.high=70%
   - xpack.security.enabled=true
   - xpack.security.audit.enabled=true
   - ELASTIC_PASSWORD=somethingsecret
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   - bootstrap.memory_lock=true

  ulimits:
    memlock:
      soft: -1
      hard: -1
  volumes:
    - ./data:/usr/share/elasticsearch/data
  ports:
   - "9200:9200"
  networks:
   - eknetwork

 kibana:
  depends_on:
   - elasticsearch
  image: "docker.elastic.co/kibana/kibana:7.13.2"
  container_name: kibana
  ports:
   - "5601:5601"
  environment:
   - ELASTICSEARCH_URL=http://localhost:9200
   - ELASTICSEARCH_USERNAME=elastic
   - ELASTICSEARCH_PASSWORD=somethingsecretpassword
  networks:
   - eknetwork

networks:
 eknetwork:
  driver: bridge

Make sure to add same elasticsearch password as the one you have provided in your logstash pipeline configuration file.

To run the above file

docker-compose up -d

Now that all our processes are running, You would need to configure Kibana's source from the UI so that you are able to see data on the UI. If you have done this setup on the server and want to configure UI immediately, I would suggest you to create a firewall rule for your public IP on port 5601

ufw allow from YOUR_PUBLIC_IP to any port 5601

Now go to your favorite browser, type server's ip and port and you will see something like this:

Enter the credentials as specified in the docker-compose above and you should be able to enter the application.

And Voila! Your setup is complete. Just don't forget to add log-rotation to your docker instances.
I have added all the configuration and docker file here