DEV Community

Aloisio Bilck
Aloisio Bilck

Posted on

1

Kubernetes + Nginx ingress enviando logs para o Graylog com fluent-bit

Neste tutorial será mostrado como centralizar os logs do Kubernetes + Nginx-ingress com Fluent-bit.

Existem outras maneiras talvez mais elegantes, mas optamos em enviar os logs (stdout/stderr) dos pods para o Graylog utilizando o fluent-bit. O fluent-bit será instalado por helm chart.

Documentação oficial

Graylog
Fluent-bit helm chart
Kubernetes

Versões utilizadas

Graylog >= 4.x
fluent-bit >= 0.20.8

Configuração do Fluent-bit

Não irei abordar a instalação do fluent-bit utilizando helm chart. Você pode seguir a documentação oficial para instalação. Irei deixar o custom-values.yaml que utilizei para instalação. Talvez para seu ambiente você precise customizar mais alguns parâmetros.
Estou enviando os logs de todos os namespaces do cluster para o Graylog, você pode customizar isso utilizando o parâmetro Exclude_Path.

# kind -- DaemonSet or Deployment
kind: DaemonSet

serviceMonitor:
  enabled: true

## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file
config:
  service: |
    [SERVICE]
        Flush 5
        Daemon Off
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        # Aqui estou excluindo o envio dos logs do namespace `logging`.
        Exclude_Path /var/log/containers/*logging*.log
        Parser docker
        Tag kube.*
        DB /var/log/flb_kube.db
        Mem_Buf_Limit 50MB
        Refresh_Interval 10
        Docker_Mode On
        Skip_Long_Lines On
        Buffer_Chunk_Size 64k
        Buffer_Max_Size 64k

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name parser
        Match kube.*
        Key_Name log
        Parser glog_format
        Parser catchall
        Reserve_Data On
        Preserve_Key Off

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log_Key log_processed
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Labels On
        Annotations On
        Buffer_Size 128k

    [FILTER]
        Name nest
        Match *
        Operation lift
        Nested_under log_processed

    [FILTER]
        Name record_modifier
        Match *
        # <tag_cluster> para o graylog identificar qual cluster está enviado os eventos
        Record kubernetes_environment <tag_cluster>

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name gelf
        Match kube.*
        Host <GRAYLOG_URL>
        Port <GRAYLOG_PORT>
        Mode udp
        Gelf_Short_Message_Key message

    # [OUTPUT]
    #     Name  stdout
    #     Match *
  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name glog_format
        Format regex
        Regex ^(?<severity>[IWECF])(?<timestamp>\d{4} \d{2}:\d{2}:\d{2}\.\d{6}) +(?<thread_id>\d+) (?<src_file>[^:]+):(?<src_line>\d+)\] (?<message>(?:.|\n)*)
        Time_Key timestamp
        Time_Format %m%d %H:%M:%S.%L
        Time_Keep Off
        Types thread_id:integer src_line:integer

    [PARSER]
        Name   json_timestamp
        Format json
        Time_Key timestamp
        Time_Keep Off
        Time_Format %Y-%m-%dT%H:%M:%S.%s%z

    [PARSER]
        Name catchall
        Format regex
        Regex ^(?<message>.*)$

logLevel: info
Enter fullscreen mode Exit fullscreen mode

Nginx Ingress Controller

Na instalação do nginx-ingress por helm chart, utilizei esses parâmetros no custom-values.yaml

...
controller:
  # Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
  config:
    log-format-escape-json: "true"
    log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x_forward_for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status": $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent" }'
...

Enter fullscreen mode Exit fullscreen mode

Image of Docusign

Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs