Use CloudWatch to monitor EKS cluster and gather logs from Rails app on Kubernetes

This article was wrote 2019.
Be careful there may be deprecated contents.

Overview

The purpose is monitor EKS, an AWS managed Kubernetes cluster, and manage logs emitted by Ruby on Rails apps running on Kubernetes with AWS CloudWatch.

For cluster monitoring, we can use Container Insights. (Now Container Insights got GA https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html)

And fluentd-kubernetes-daemonset is used to gather application logs.

Kubernetes version 1.12 or higher is required because ConfigMap is necessary.

But even if you are using an older version, you can easily update it by just clicking the button if you are using EKS.

Steps

1. Policy settings for EC2 instances of Kubernetes worker nodes

First, configure IAM so that Kubernetes workers can send logs to CloudWatch.

Simply attach CloudWatchAgentServerPolicy to the worker’s role.

2. Introduction of Container Insights

First, install Container Insights to monitor the cluster.

Create a namespace, service account, and ConfigMap for CloudWatch, and deploy the Container Insights daemon set.

https://docs.amazonaws.cn/en_us/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html

Then, run this.

$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cloudwatch-namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-serviceaccount.yaml 
$ curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-configmap.yaml 
$ micro cwagent-configmap.yaml # replace {{cluster_name}} by your cluster name.
$ kubectl apply -f cwagent-configmap.yaml

With this alone, logs can be sent to CloudWatch.

Next, let’s enable to send the Rails application log to CloudWatch.

3. Logging settings of Rails app

fluentd-kubernetes-daemonset makes it easy to send the standard output of each pod to CloudWatch.

So, change all the Rails application logs to standard output. Edit config/{{environment}}.rb and config/puma.rb.

Rails5 has the following description in production.log, so if you set the environment variable, that’s it.

if ENV["RAILS_LOG_TO_STDOUT"].present?
  stdout_logger = ActiveSupport::Logger.new(STDOUT)
  stdout_logger.formatter = config.log_formatter
  multiple_loggers = ActiveSupport::Logger.broadcast(stdout_logger)
  logger.extend(multiple_loggers)   
end

For puma, if stdout_redirect is set, just delete that line.

4. Deploy fluentd-kubernetes-daemonset

Since you have already added a namespace for CloudWatch, just run the following command:

$ curl -O https://s3.amazonaws.com/cloudwatch-agent-k8s-yamls/fluentd/fluentd.yml
// fill cluster_name and region_name by your environment
$ kubectl create configmap cluster-info --from-literal=cluster.name=cluster_name --from-literal=logs.region=region_name -n amazon-cloudwatch

This will transfer the application logs to CloudWatch.

If this is all you need, it’s OK.

5. Formatting logs

If the log output by Rails is left as default, it will be output in multiple lines, so each line will be treated as a separate log on CloudWatch.

Therefore, instead of outputting in multiple lines, output it as a single line log in JSON format.

If it is in JSON format, you can search on CloudWatch.

Use lograge to change the request log in json format. Add lograge to Gemfile and run bundle install.

Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.formatter = Lograge::Formatters::Json.new   
  config.lograge.custom_options = Proc.new do |event|
    exceptions = %w(controller action format id)
    {
      time: event.time,
      host: event.payload[:host],
      remote_ip: event.payload[:remote_ip],
      params: event.payload[:params].except(*exceptions),
      exception_object: event.payload[:exception_object],
      exception: event.payload[:exception],
      backtrace: event.payload[:exception_object].try(:backtrace),
    }.compact
  end      
  ...
end

class ApplicationController < ActionController::Base
  def append_info_to_payload(payload)
    super
    payload[:user_agent] ||= request.user_agent
    payload[:request_id] ||= request.request_id
    if @exception.present?
      payload[:exception_object] ||= @exception
      payload[:exception] ||= [@exception.class, @exception.message]
    end
  end
end

Also, if only this, outside the ActionController, it can not respond to RoutingError that occurs at the ActionDispatch level, so add the following monkey patch

if defined? Lograge
  class ActionDispatch::DebugExceptions
    alias_method :org_log_error, :log_error
    def log_error(request, wrapper)
      msg = {
        exception: wrapper.exception,
        backtrace: wrapper.exception.try(:backtrace),
        raw_request: request,
        raw_wrapper: wrapper
      }      Rails.logger.fatal(msg.to_json)
    end
  end
end

However, the Rails log output in JSON format is just an escaped string.

Therefore, change the setting of fluentd.

In the fluend.yml downloaded in the previous article, add the following to parse the json log. This will cause the log parsed with the key parsed_logto be recorded.

<filter **>
  @type parser
  format json
  key_name log
  reserve_time true
  reserve_data true
  emit_invalid_record_to_error false
  hash_value_field parsed_log
 </filter>

https://s3.amazonaws.com/cloudwatch-agent-k8s-yamls/fluentd/fluentd.yml

Around line 125. Add the above settings and apply them.

Also, if the log stream name is defalut, it becomes the pod name, and if the pod changes, it creates a different log stream.

Personally, I want the log stream to appear in the same log stream even if the pod changes, so take it from the label.

<filter **>
  @type record_transformer
  @id filter_containers_stream_transformer
  enable_ruby # 追加
  <record>
    stream_name ${record["kubernetes"]["labels"]["app"]}
  </record>
</filter>

Summary

You can now monitor the cluster and aggregate application logs with CloudWatch.