Rancher Logging Operator with in-cluster Syslog

The Rancher Logging Operator supports a number of backends, but a very common one is logging to a syslog endpoint. However, rather than maintaining separate logging infrastructure if this is a new piece of your environment, you can deploy it to your Kubernetes clusters directly, either using the method in the UI above, or via the Helm chart.

If you plan to deploy charts via Fleet (which I've written about before, or some other GitOps flow, you would pull the following charts from the Rancher chart repo:

helm repo add rancher-charts https://charts.rancher.io
helm repo update
helm fetch rancher-charts/rancher-logging-crd
helm fetch rancher-charts/rancher-logging

but before we apply them, we want to setup the syslog endpoint in the cluster for the logging operator to log to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsyslog-deployment
  labels:
    app: rsyslog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rsyslog
  template:
    metadata:
      labels:
        app: rsyslog
    spec:
      containers:
      - name: rsyslog
        image: sudheerc1190/rsyslog:latest
        ports:
        - containerPort: 514
        resources:
          requests:
            cpu: 250m
            memory: 524Mi
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

here we're just creating a syslog Deployment; for production, I would recommend creating a PersistentVolume and mount it to /var/log/remote so in subsequent recreations of this Pod, you can restore past logging data for this or other clusters.

Because we want this just available internal to the cluster in this example, we'll just create the following Service objects to create cluster DNS entires for this endpoint:

---
apiVersion: v1
kind: Service
metadata:
  name: "syslog-service-tcp"
spec:
  ports:
    - port: 514
      targetPort: 514
      protocol: TCP
  type: LoadBalancer
  selector:
    app: "rsyslog"
---
apiVersion: v1
kind: Service
metadata:
  name: "syslog-service-udp"
spec:
  ports:
    - port: 514
      targetPort: 514
      protocol: UDP
  type: LoadBalancer
  selector:
    app: "rsyslog"

so depending upon which protocol you'd like to use for logging, you can use a hostname like syslog-service-udp.default.svc.cluster.local.

Now, you can create the CRDs and Logging Operator:

helm install rancher-logging-crd ./rancher-logging-crd-100.1.2+up3.17.4.tgz -n cattle-logging-system --create-namespace

helm install rancher-logging ./rancher-logging-100.1.2+up3.17.4.tgz -n cattle-logging-system

Note that the version on these charts should match, and that they should be deployed to the cattle-logging-system namespace.

In the logging operator, there are two pieces Flow (and cluster-wide ClusterFlow) resources and Output (and ClusterOutput) resources. This example will just use cluster-scoped examples that span namespaces, but the matching rules and output formatting will work similarly in both cases.

Our ClusterOutput will be what connects to our Syslog service, using host: syslog-service-udp.default.svc.cluster.local on port 514, in this case using transport: udp:

apiVersion: v1
items:
- apiVersion: logging.banzaicloud.io/v1beta1
  kind: ClusterOutput
  metadata:
    name: testing
    namespace: cattle-logging-system
  spec:
    syslog:
      buffer:
        total_limit_size: 2GB
        flush_thread_count: 8
        timekey: 10m
        timekey_use_utc: true
        timekey_wait: 1m
      format:
        app_name_field: kubernetes.pod_name
        hostname_field: custom-cluster-name
        log_field: message
        rfc6587_message_size: false
      host: syslog-service-udp.default.svc.cluster.local
      insecure: true
      port: 514
      transport: udp
kind: List

and you can verify the status of the output (deployment status, problems, etc.) with kubectl get clusteroutput/testing -n cattle-logging-system. The ClusterOutput should be namespaces with the operator deployment. In the above spec, under format, you can refer to the RFC format for other options for logging fields.

In our ClusterFlow, we're going to use just a basic rule, ask to exclude logs from the kube-system namespace, and then select everything from the applications namespace:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: global-ns-clusterflow
spec:
  match:
    - exclude:
        namespaces:
        - kube-system
    - select:
        namespaces:
        - applications
  globalOutputRefs:
    - testing

and you'll see we use the testing output as the global endpoint for sending logs. You can check the status of the Flow using kubectl get clusterflow/global-ns-clusterflow.

I usually test after this point using a deployment like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

and then deploy it once to an excluded namespace and then an included one, then make a few requests (curl -s http://{LB_IP}, if the output and flow report no issues, you'll see the logs start to populate after a few minutes). You can then check the logs manually (if nothing is scraping syslog externally) with something formatted like kubectl exec -it {rsyslog_pod} -- tail -f /var/log/remote/{date hierarchy}/{hour}.log.

More about the BanzaiCloud Operator can be found here, and more on the Logging features in Rancher from this operator can be found here.