loading...

Write test cases for your Apache Logstash filters

oivoodoo profile image Alexandr K ・3 min read

About 1 year ago I started to work on ELK(ElasticSearch, Logstash, Kibana) setup for BI platform. While I was working on processing the events coming to Logstash I found that it's pretty annoying to work on filter changes if you have complex regular expressions, structures or conditions.

I had a lot of experience in Ruby On Rails development the last years and decided to cover Logstash filters by RSpec test cases to save my time, because logstash is using jruby inside of their packages and gems to distribute the extensions.

Also it would be great to have everything in docker to skip Logstash and gem installations on local machine.

Lets start from the Dockerfile and then I will extend step by step the project structure.

FROM logstash:2.4

# Install Rspec related dependencies
RUN logstash-plugin install --development

# Install prod dependencies
RUN logstash-plugin install logstash-filter-prune

ARG ES_PLUGIN=logstash-output-elasticsearch-6.2.4-java.gem
ARG KAFKA_PLUGIN=logstash-input-kafka-7.0.0.gem

COPY gems/${ES_PLUGIN} /tmp/${ES_PLUGIN}
RUN logstash-plugin install /tmp/${ES_PLUGIN}

COPY gems/${KAFKA_PLUGIN} /tmp/${KAFKA_PLUGIN}
RUN logstash-plugin install /tmp/${KAFKA_PLUGIN}

gems/ contains freezed gem versions for logstash setup.

Faster way to run test cases lets define Makefile:

NAME = your_logstash

build:
    docker build -t $(NAME) .
.PHONY: build

clean:
    docker rmi --force $(NAME)
.PHONY: clean

test:
    @docker run --rm -t -i \
        -v `pwd`/../:/app \
        -w /app \
        $(NAME) \
        /bin/bash -c "rspec /app/logstash/spec/$(TEST_CASE)"
.PHONY: test

console:
    @docker run --rm -t -i \
        -v `pwd`/../:/app \
        -w /app \
        $(NAME) \
        /bin/bash
.PHONY: console

make test - run rspec inside of your container
make console - opens interactive terminal inside of docker container in case if you want to run everything manually and debug(binding.pry) your specs.

Now it would be great to start working on spec/spec_helper.rb that's root of rspec.

require "logstash/devutils/rspec/spec_helper"
require 'rspec'
require 'rspec/expectations'

require 'ostruct'
require 'erb'
require 'yaml'
require 'json'

# running the grok code outside a logstash package means
# LOGSTASH_HOME will not be defined, so let's set it here
# before requiring the grok filter
# coming from the original examples for logtash specs.
unless LogStash::Environment.const_defined?(:LOGSTASH_HOME)
  LogStash::Environment::LOGSTASH_HOME = File.expand_path("../", __FILE__)
end

module Helpers
  ROOT_PATH = File.dirname(File.expand_path(__FILE__))
  TEMPLATES_PATH = File.join(ROOT_PATH, '..', 'conf.d/')

  def load_fixture(filename, settings = {})
    message = File.read(File.join(ROOT_PATH, 'fixtures', filename))
    settings.merge('message' => message)
  end

  def load_filter(filename, render_vars = {})
    content = File.read(File.join(TEMPLATES_PATH, filename))

    render_vars = OpenStruct.new(render_vars)

    # it's not elegant enough :) but found it as the easiest way to handle
    # jinja2 simple variables replacement.
    template = ERB.new(content.gsub('{{', '<%=').gsub('}}', '%>'))
    template.result(render_vars.instance_eval { binding })
  end
end

require "logstash/filters/grok"

In my logstash filters I am using the statements from jinja2 python library {{ }} to replace variables on deploy by using ansible.

Now we are ready to define actual spec to test our logstash filter.

lets assume you want to regex line like username=<username>. Define your filter_spec.rb inside spec/filters folder. Filters should not deal with input and output logstash statements. Instead we should read fixture with example of expected line to process and then apply filter.

require_relative '../spec_helper'

describe 'elb' do
  extend Helpers

  # set config using our defined filter in `conf.d/01-filter.conf` file.
  config load_filter('01-filter.conf')

  # You can define in spec/fixtures/sample1.txt
  # Example: username=oivoodoo
  sample(load_fixture('sample1.txt', 'type' => 'your-source-data')) do
    insist { subject.get('username') } == 'oivoodoo'
  end
end
filter {
  if [type] == "{{ type }}" {
    grok {
      match => {
        "message" => [
          "%{WORD:username}"
        ]
      }
    }

    prune {
      blacklist_names => [
        "@version",
        "message"
      ]
    }
  }
}

I believe it should be enough to test your filters. Found it pretty useful to save time instead of deploying and waiting new coming data. Also it's easy to add ruby filter inside of filter and place binding.pry and verify event object.

Posted on by:

Discussion

pic
Editor guide