JMeter Integration with Elastic

I work in Kibana almost every day at work to debug and monitor the test execution. I always wanted to integrate my JMeter logs with Kibana to deep-dive into JMeter logs in real time. This blog post will talk about JMeter Integration with Elastic suite of products ElasticSearch, Kibana, and Filebeat. We are going to deploy ElasticSearch, Kibana, and Filebeat components locally and visualize the logs in Kibana.

There are many other integrations possible to deep-dive into JMeter metrics. But using the ELK (Elastic, Logstash, and Kibana) stack, it is easy to implement and maintain.

About Elastic

Elastic offers various products and solutions for search, analytics, security, logging and more. Their solutions are powered by ElasticSearch and Kibana.

ElasticSearch is the search and analytics engine which is the heart of the Elastic Stack.

Kibana is the visualization layer of ELK.

Beats and Logstash are the Ingest engine.

JMeter Integration with Elastic - ELK Stack

Installations

We are going to install the following components to integrate ELK with JMeter.

1. ElasticSearch
2. Kibana
3. Filebeat

Let us begin with ElasticSearch

Go to https://www.elastic.co/start to begin downloading ElasticSearch. Click on the respective download links based on your operating system. This integration tested with Windows 10 Pro OS.

Download the zip file for Windows, extract it and place it in your favorite location. Once you extracted the zip file, you will get the below folders and files. By default, ElasticSearch uses localhost:9200. You can change the port and other configuration by editing the config\elasticsearch.yml

ElasticSearch in Windows 10

To launch ElasticSearch, open Powershell and navigate to bin folder and enter .\elasticsearch.bat

Installing Kibana

In https://www.elastic.co/start page, click on the appropriate download link for Kibana. In Windows, extract the downloaded zip file and place it in your favorite folder.

Kibana in Windows 10

By default, Kibana uses localhost:5601. To configure the properties, go to config\kibana.yml.

To start Kibana, open Powershell, navigate to bin folder and enter .\kibana.bat.

Installing Filebeat

Filebeat is the lightweight log analysis for Elastic. Go to https://www.elastic.co/downloads/beats/filebeat to download the appropriate package. For Windows, there is a MSI package available for Filebeat. Download the package and install it.

It will get install in C:\Program Files\Filebeat.

Install the Filebeat as a service which will help you to start the service for the unexpected restarts.

To install it as a service, open Powershell and navigate to C:\Program Files\Filebeat and run

.\install-service-filebeat.ps1

Configuring Filebeat

Now we completed all the installations which are required to integrate. Next step is to configure the Filebeat and Kibana.

To configure Filebeat, open Filebeat.yml from C:\Program Files\Filebeat location and enter the below configurations.

Below configuration is to mention the inputs and the JMeter log location. Multiline pattern is to identify the timestamp in the JMeter log file.

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:\Tools\apache-jmeter-5.4\bin\jmeter.log
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

Below configuration is for Kibana.

setup.kibana:
  host: "localhost:5601"

Below configuration is for ElasticSearch.

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "<username>"
  password: "<password>"

Save the yml file and start the service using the below command.

Start-Service filebeat

To stop the service, use Stop-Service filebeat.

Launch Kibana by navigating to http://localhost:5601. Go to Management > Stack Management page.

Configuring Kibana

Click on Index Patterns and make sure filebeat-* pattern is exists. Else, you can create one by clicking on Create Index pattern button. Index pattern helps to retrieve the data from ElasticSearch.

Index Patterns

JMeter Test Execution

Now, it is time to start the test in JMeter. Launch JMeter and open the test plan which you want to execute. In the below example, I have one simple JSR223 Sampler to log info.

Sample Test Plan

Start the execution in CLI mode. This test plan will simply log the string in the jmeter.log file.

Launch Kibana and click on Discover as shown below.

Kibana Discover

Change the time filter and click on message and then click on Refresh button.

Filter

This will filter out only the timestamp and the message and display the hits stats.

Output

Closing Words

By integrating JMeter with ElasticSearch, you can stream the performance stats real time to Kibana and visualize the data as per your requirement. This will help performance engineers to drill-down to the root cause when the test is running.

Kibana comes with powerful visualization, machine learning, canvas features which will expedite the process of performance testing and engineering.