Fluent-bit as a sidecar in Pod

Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams. However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution.

Logging architectures require a separate backend to store, analyze, and query logs. Kubernetes does not provide a native storage solution for log data. Instead, many logging solutions integrate with Kubernetes.

In this article, the goal is to collect standard output logs and ship them to a centralized store. The log collector can be configured as a sidecar in the pod or as a daemonset in the cluster.

The Architecture:

Fluent Bit is used to collect and ship the standard output logs from the pod. It is an open-source Log Processor and Shipper, designed with performance in mind: high throughput with low CPU and Memory usage. Influx DB, an open-source time-series database, is used as the centralized store for the logs.

The setup:

Wait! How to access the stdout logs from a pod?

The standard output logs are written to /var/log/pods/{NAMESPACE}_{POD_NAME}_{POD_ID}/{CONTAINER_NAME}/*.log files . We just need to tail all the log files present in the above location.

Fluent Bit Configuration

A configmap is created to hold the fluent bit config template.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config-template
data:
  fluent-bit-template.conf: |
    [INPUT]
      Name tail
      Path /var/log/pods/${NAMESPACE}_${POD_NAME}_${POD_ID}/${CONTAINER_NAME}/*.log
    [OUTPUT]
      Name          influxdb
      Match         *
      Host          ${INFLUX_HOST}
      Port          ${INFLUX_PORT}
      Database      ${INFLUX_DB}
      Sequence_Tag  _seq
      http_token    ${INFLUX_TOKEN}
      Bucket        ${INFLUX_BUCKET}
      Org           ${INFLUX_ORG}

Fluent Bit connection to InfluxDB

A secret object is created containing access configuration and credentials for InfluxDB.

apiVersion: v1
kind: Secret
metadata:
  name: influx-db-cred
data:
  INFLUX_HOST: <<TO_BE_UPDATED>>
  INFLUX_PORT: <<TO_BE_UPDATED>>
  INFLUX_DB: <<TO_BE_UPDATED>>
  INFLUX_TOKEN: <<TO_BE_UPDATED>>
  INFLUX_BUCKET: <<TO_BE_UPDATED>>
  INFLUX_ORG: <<TO_BE_UPDATED>>

Pod configuration for running Fluent Bit as a sidecar

Volumes are used to hold the fluent-bit config file template, the actual template file. Pod logs present in the host are mounted to the sidecar.

  volumes:
  - name: fluent-bit-config-template
    configMap:
      name: fluent-bit-config-template
      defaultMode: 0777
  - name: app-log
    hostPath:
      path: /var/log/pods
      type: Directory
  - name: fluent-bit-config
    emptyDir: {}

A valid fluent-bit config file is generated from the defined template. All the required details and credentials are injected into the init-container, which will prepare the fluent-bit config file from the template. Using K8s Downward API, Pod information are exposed.

  initContainers:
  - name: fluent-bit-config-manager
    image: busybox:latest
    volumeMounts:
      - mountPath: /fluent-bit-template.conf
        name: fluent-bit-config-template
        subPath: fluent-bit-template.conf
      - mountPath: /conf
        name: fluent-bit-config
    envFrom:
    - secretRef:
        name: influx-db-cred
    env:
    - name: POD_ID
      valueFrom:
        fieldRef:
          fieldPath: metadata.uid
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: CONTAINER_NAME
      value: *container1
    command: ["/bin/sh"]
    args:
    - -c
    - eval "echo \"$(cat /fluent-bit-template.conf)\"" > /conf/fluent-bit.conf

The fluent-bit sidecar container requires the access to the log files created by pods and the generated configuration file. Both of which are mounted to the sidecar using volumes.

  - name: fluent
    image: fluent/fluent-bit:1.8
    volumeMounts:
      - mountPath: /fluent-bit/etc/fluent-bit.conf
        name: fluent-bit-config
        subPath: fluent-bit.conf
      - mountPath: /var/log/pods
        name: app-log

Visualizing the logs in InfluxDB dashboard

Running a query on the configured bucket will return a result set containing all the logs shipped by the sidecar.

What's next?

• Explore different input and output plugins supported by Fluent Bit.
• Use parser and filter to format the logs
• Build your own fluent-bit plugin
• Run Fluent Bit as a deamonset in the cluster.

Here are all the manifests used for the setup.