Something About Grok

#elasticsearch #backend #devops #dataanalysis

Here Grok, a filter plugin used in ELK stack. It converts unstructured logs to structured ones.

Grok is built upon regular expression. The syntax is %{SYNTAX:SEMANTIC}. For example:

A log line

10.0.1.13 GET /home HTTP/2.0 200 13969

A filter defined in logstash configure file as below

filter {
  grok {
    match => {
      "message" => '%{IPORHOST:ip} \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})'
    }
  }
}

The log line passes through the filter, it produces the result

ip: 10.0.1.13
method: GET
request: /home
response: 200
bytes: 13969

With structured data, you can search by specific field while using Elasticsearch.

Grok Debugger

If you are not so sure the pattern would work on your log or not, here are some grok debugger to help you verify the patterns.

Make log processing easier. :D

The Jargon File, which describes itself as a "Hacker's Dictionary" and has been published under that name three times, puts grok in a programming context:

When you claim to "grok" some knowledge or technique, you are asserting that you have not merely learned it in a detached instrumental way but that it has become part of you, part of your identity. For example, to say that you "know" Lisp is simply to assert that you can code in it if necessary — but to say you "grok" LISP is to claim that you have deeply entered the world-view and spirit of the language, with the implication that it has transformed your view of programming. Contrast zen, which is a similar supernatural understanding experienced as a single brief flash.
--Wikipedia