Logz.io: collection logs from Kubernetes — fluentd vs filebeat
We are using Logz.io to collect our Kubernetes cluster logs (also, there is a local Loki instance).
Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here — logzio-k8s.
The problem we faced is that those pods are consuming too much CPU — up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.
So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.
The found solution w a Filebeat collector, see Ship k8s logs with Helm via Filebeat — let’s try it.
At first, we will deploy it manually do a Dev cluster, and then we’ll add an Aisible task to deploy it on our Production.
Logz.io Filebeat Helm chart
Add a repository:
$ helm repo add logzio-helm [https://logzio.github.io/logzio-helm/filebeat](https://logzio.github.io/logzio-helm/filebeat)
“logzio-helm” has been added to your repositories
Next, we need to fin our token and region — go to the account’s General Settings:
As we have the default region — us-east-1, see How to look up your account region, then we can remove the secrets.logzioRegion
from the Helm chart's arguments.
Also, by default Logz.io client will be installed to the kube-system
namespace, but I'd like to place it to a dedicated NS to make it easier to monitor its used resources.
The Namespace is configured in the values file so let’s override it with --set
, plus let's add the --create-namespace
and --debug
options:
$ helm install — namespace=dev-1–18-devops-logzio-ns \
--create-namespace — debug \
--set secrets.logzioShippingToken='AVG***Onq' \
--set secrets.clusterName='bttrm-eks-dev-1–18' \
--set namespace=dev-1–18-devops-logzio-ns \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
Check pods:
$ kubectl -n dev-1–18-devops-logzio-ns get pod
NAME READY STATUS RESTARTS AGE
filebeat-2qt5s 1/1 Running 0 96s
filebeat-4xb44 1/1 Running 0 96s
filebeat-9prr9 1/1 Running 0 96s
filebeat-cth47 1/1 Running 0 96s
filebeat-fgmgx 1/1 Running 0 96s
filebeat-gb5ts 1/1 Running 0 96s
filebeat-hs9tr 1/1 Running 0 96s
filebeat-nskvg 1/1 Running 0 96s
filebeat-wfgbg 1/1 Running 0 96s
And logs in the Kibana on the app.logs.io page:
Ansible
There is a post about deploying a Helm chart with Ansible — Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS (Rus), at this time just a quick example.
Add a when condition as we will deploy Logz.io to only the Production cluster:
- name: "Add Logzio chart repo"
when: "eks_env.startswith('prod')"
community.kubernetes.helm_repository:
name: "logzio-helm"
repo_url: "https://logzio.github.io/logzio-helm/filebeat"
- name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace"
when: "eks_env.startswith('prod')"
community.kubernetes.helm:
kubeconfig: "{{ kube_config_path }}"
name: "logzio-k8s-logs"
chart_ref: "logzio-helm/logzio-k8s-logs"
release_namespace: "{{ eks_env }}-devops-logzio-ns"
create_namespace: true
values:
secrets:
logzioShippingToken: "{{ logzio_token }}"
clusterName: "{{ eks_cluster_name }}"
namespace: "{{ eks_env }}-devops-logzio-ns"
Specify the logzio_token
in the Ansible variables and encrypt it with the ansible-vault
, as we have our Ansible roles stored in our Github repository.
Fluentd vs Filebeat — CPU and performance
And finally CPU usage: old fluentd
(Ruby + C) on the left side vs new filebeat
(Golang) at the right side:
Originally published at RTFM: Linux, DevOps and system administration.
Top comments (0)