Rsyslog kafka and elk

Intro.

The elk stack is popular, but I have not found any suitable articles about connecting Rsyslog to Kafka and ELK. You can found somewhere about Kafka somewhere about Logstash or Rsyslog, but not altogether.

In this article, we will make a docker-compose file that will launch the entire system, and build an image that simulates an application with logs. We will also consider how you can check each system separately.

I don't want to create a detailed description of each application. It is just starting point for you to learn rsyslog and ELK

Github with full project: https://github.com/ArtemMe/rsyslog_kafka_elk

I split the project on release (tag). Every release is a new service like Rsyslog(tag 0.1) or Kibana(tag 0.4). You can switch to desire release and start project to test build

Below in the article, I give a description of each service. You can download the project and go to the root of the project and enter the command:

docker-compose up

Rsyslog kafka logstash elasticsearch and kibana will be up. So you can go to kibana on localhost:5601 and check launch.
Also, each section contains tips on how to check the service %)

Rsyslog. (tag 0.1)

We will need two configuration files: one with the basic settings /etc/rsyslog.conf, the second/etc/rsyslog.d/kafka-sender.conf is optional with settings for our needs

We will not delve into the rsyslog settings because you can dig into them for a long time. Just remember basic instructions: module, template, action
Let's take a look at an example of the file /etc/rsyslog.d/kafka-sender.conf:

# load module which use for sending message to kafka
module(load="omkafka") 

# Declare template for log with name "json_lines" :
template(name="json_lines" type="list" option.json="on") {  
        constant(value="{")
        constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
        constant(value="\",\"message\":\"")     property(name="msg")
        constant(value="\",\"host\":\"")        property(name="hostname")
        constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
        constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
        constant(value="\",\"syslog-tag\":\"")  property(name="syslogtag")
        constant(value="\"}")
}

# Decalare action to send message to kafka broker in test_topic_1. Note how we use template json_lines and module omkafka
action(
        broker=["host.docker.internal:9092"]
        type="omkafka"
        template="json_lines"
        topic="test_topic_1"
        action.resumeRetryCount="-1"
        action.reportsuspension="on"
)

Remember topic name: test_topic_1

You can find the full list of property names for templates there: https://www.rsyslog.com/doc/master/configuration/properties.html

Also note the main file /etc/rsyslog.conf contains a line like:$ IncludeConfig /etc/rsyslog.d/ *. Conf
This is a directive that tells us where else to read the settings for rsyslog. It is useful to separate common settings from specific ones

Create an image for generating logs

The image will essentially just start rsyslog. In the future, we will be able to enter this container and generate logs.

You can find the Docker file in the /rsyslog folder. Let's look at the chunk of that file where on the first and second lines we copy our config. On the third line, we mount a folder for logs which will be generated

COPY rsyslog.conf /etc/
COPY rsyslog.d/*.conf /etc/rsyslog.d/

VOLUME ["/var/log"]

Building. Go to /rsyslog folder and execute

docker build . -t rsyslog_kafka

Launch the container to check the image.

docker run rsyslog_kafka

To check that rsyslog are writing logs, go to our container and call the command:

docker run --rm --network=rsyslog_kafka_elk_elk rsyslog_kafka bash -c `logger -p daemon.debug "This is a test."`

Let's look at folder /logsand you should find a string like this This is a test..

Congratulations! You have configured rsyslog in your docker container!

A bit about networking in docker containers.

Let's create our network in the docker-compose.yml file. In the future, each service can be launched to different machines. This is no problem.

networks:
  elk:
    driver: bridge

Kafka (tag 0.2)

I took this repository as a basis: https://github.com/wurstmeister/kafka-docker
The resulting service is:

zookeeper:
  image: wurstmeister/zookeeper:latest
  ports:
    - "2181:2181"
  container_name: zookeeper
  networks:
    - elk

kafka:
  image: wurstmeister/kafka:0.11.0.1
  ports:
    - "9092:9092"
  environment:
    # The below only works for a macOS environment if you installed Docker for
    # Mac. If your Docker engine is using another platform/OS, please refer to
    # the relevant documentation with regards to finding the Host IP address
    # for your platform.
    KAFKA_ADVERTISED_HOST_NAME: docker.for.mac.localhost
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_CREATE_TOPICS: "logstash_logs:1:1"
  links:
    - zookeeper
  depends_on:
    - zookeeper
  container_name: kafka
  networks:
    - elk

Tips how to check kafka (you can do it after starting containers):

We will be able to see what is in the Kafka topic when we launch our containers. First, you need to download Kafka. Here is a cool tutorial https://kafka.apache.org/quickstart but if it's short download it here https://www.apache.org/dyn/closer.cgi?path=/kafka/2.7.0/kafka_2.13-2.7.0.tgz and unpack it to /app folder.
Actually, we need scripts in the /bin folder.

Now, we can connect to the container and execute a script to see if there are any entries inside the topic test_topic_1:

docker run --rm --network=rsyslog_kafka_elk_elk -v /app/kafka_2.13-2.7.0:/kafka wurstmeister/kafka:0.11.0.1 bash -c "/kafka/bin/kafka-console-consumer.sh --topic test_topic_1 --from-beginning --bootstrap-server 172.23.0.4:9092"

About the command itself: we connect to the rsyslog_kafka_elk_elk network, rsyslog_kafka_elk is the name of the folder where the docker-compose.yml file is located, and elk is the network that we specified. With the -v command, we mount scripts for Kafka into our container.

The result of command should be something like this:

{"timestamp":"2021-02-27T17:43:38.828970+00:00","message":" action 'action-1-omkafka' resumed (module 'omkafka') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]","host":"c0dcee95ffd0","severity":"info","facility":"syslog","syslog-tag":"rsyslogd:"}

Logstash (tag 0.3)

Configs are located in the /logstash folder. logstash.yml - here we specify parameters for connecting to Elasticsearch

In the config, there is a setting for Kafka as for an incoming stream and a setting for elasticsearch as for an outgoing stream

input {
    beats {
        port => 5044
    }

    tcp {
        port => 5000
    }
    kafka
    {
        bootstrap_servers => "kafka:9092"
        topics => "test_topic_1"
    }
}

## Add your filters / logstash plugins configuration here

output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        user => "elastic"
        password => "changeme"
        ecs_compatibility => disabled
    }

    file {

        path => "/var/logstash/logs/test.log"
        codec => line { format => "custom format: %{message}"}
    }
}

To monitor what goes into the Elastisearch and check the Logstesh is working properly, I created a file output stream so logs will be written to test.log file. The main thing is does not forget to add volume to docker-compose.yml

volumes:
  - type: bind
    source: ./logstash/config/logstash.yml
    target: /usr/share/logstash/config/logstash.yml
    read_only: true
  - type: bind
    source: ./logstash/pipeline
    target: /usr/share/logstash/pipeline
    read_only: true
  - ./logs:/var/logstash/logs

When you start the service check test.log file in your project. You should find logs from kafka

Elasticsearch (tag 0.3)

This is the simplest configuration. We will launch the trial version, but you can turn on the open source one if you wish. Configs as usual in /elasticsearch/config/

## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

Tips how to check elasticsearch (you can do it after starting containers):

Let's check the indexes of the elastic. Take as a basis a cool image of praqma/network-multitool and command curl:

docker run --rm --network=rsyslog_kafka_elk_elk praqma/network-multitool bash -c "curl elasticsearch:9200/_cat/indices?s=store.size:desc -u elastic:changeme"

As the result of command:

The directory /usr/share/nginx/html is not mounted.
Over-writing the default index.html file with some useful information.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0green  open .monitoring-es-7-2021.02.28       QP1RL9ezRwmCFLe38dnlTg 1 0 1337 442   1.4mb   1.4mb
green  open .monitoring-es-7-2021.03.07       z0f-K-g7RhqDEbqnupfzPA 1 0  576 428   1.2mb   1.2mb
green  open .monitoring-logstash-7-2021.03.07 rKMYIZE9Q6mSR6_8SG5kUw 1 0  382   0 340.4kb 340.4kb
green  open .watches                          nthHo2KlRhe0HC-8MuT6rA 1 0    6  36 257.1kb 257.1kb
green  open .monitoring-logstash-7-2021.02.28 x98c3c14ToSqmBSOX8gmSg 1 0  363   0 230.1kb 230.1kb
green  open .monitoring-alerts-7              nbdSRkOSSGuLTGYv0z2L1Q 1 0    3   5  62.4kb  62.4kb
yellow open logstash-2021.03.07-000001        22YB7SzYR2a-BAgDEBY0bg 1 1   18   0  10.6kb  10.6kb
green  open .triggered_watches                sp7csXheQIiH7TGmY-EiIw 1 0    0  12   6.9kb   6.9kb
100   784  100   784    0     0  14254      0 --:--:-- --:--:-- --:--:-- 14254

We can see that the indices are being created and our elastic is alive. Let's connect Kibana now

Kibana (tag 0.4)

This is what the service looks like

kibana:
  build:
    context: kibana/
    args:
      ELK_VERSION: $ELK_VERSION
  volumes:
    - type: bind
      source: ./kibana/config/kibana.yml
      target: /usr/share/kibana/config/kibana.yml
      read_only: true
  ports:
    - "5601:5601"
  networks:
    - elk
  depends_on:
    - elasticsearch

In the /kibana folder we have a docker file to build an image and also settings for kibana:

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

## X-Pack security credentials
#
elasticsearch.username: elastic
elasticsearch.password: changeme

To enter the Kibana UI, you need to log in to the browser localhost:5601 (login/password is elasctic/changeme)
In the left menu, find Discover, click on it and create an index. I suggest this logstash-*