DEV Community: Keigo Yamamoto

Use CloudWatch to monitor EKS cluster and gather logs from Rails app on Kubernetes

Keigo Yamamoto — Mon, 30 Jan 2023 02:53:34 +0000

This article was wrote 2019.
Be careful there may be deprecated contents.

Overview

The purpose is monitor EKS, an AWS managed Kubernetes cluster, and manage logs emitted by Ruby on Rails apps running on Kubernetes with AWS CloudWatch.

For cluster monitoring, we can use Container Insights. (Now Container Insights got GA https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html)

And fluentd-kubernetes-daemonset is used to gather application logs.

Kubernetes version 1.12 or higher is required because ConfigMap is necessary.

But even if you are using an older version, you can easily update it by just clicking the button if you are using EKS.

Steps

1. Policy settings for EC2 instances of Kubernetes worker nodes

First, configure IAM so that Kubernetes workers can send logs to CloudWatch.

Simply attach CloudWatchAgentServerPolicy to the worker’s role.

2. Introduction of Container Insights

First, install Container Insights to monitor the cluster.

Create a namespace, service account, and ConfigMap for CloudWatch, and deploy the Container Insights daemon set.

https://docs.amazonaws.cn/en_us/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html

Then, run this.

$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cloudwatch-namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-serviceaccount.yaml 
$ curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-configmap.yaml 
$ micro cwagent-configmap.yaml # replace {{cluster_name}} by your cluster name.
$ kubectl apply -f cwagent-configmap.yaml

With this alone, logs can be sent to CloudWatch.

Next, let’s enable to send the Rails application log to CloudWatch.

3. Logging settings of Rails app

fluentd-kubernetes-daemonset makes it easy to send the standard output of each pod to CloudWatch.

So, change all the Rails application logs to standard output. Edit config/{{environment}}.rb and config/puma.rb.

Rails5 has the following description in production.log, so if you set the environment variable, that’s it.

if ENV["RAILS_LOG_TO_STDOUT"].present?
  stdout_logger = ActiveSupport::Logger.new(STDOUT)
  stdout_logger.formatter = config.log_formatter
  multiple_loggers = ActiveSupport::Logger.broadcast(stdout_logger)
  logger.extend(multiple_loggers)   
end

For puma, if stdout_redirect is set, just delete that line.

4. Deploy fluentd-kubernetes-daemonset

Since you have already added a namespace for CloudWatch, just run the following command:

$ curl -O https://s3.amazonaws.com/cloudwatch-agent-k8s-yamls/fluentd/fluentd.yml
// fill cluster_name and region_name by your environment
$ kubectl create configmap cluster-info --from-literal=cluster.name=cluster_name --from-literal=logs.region=region_name -n amazon-cloudwatch

This will transfer the application logs to CloudWatch.

If this is all you need, it’s OK.

5. Formatting logs

If the log output by Rails is left as default, it will be output in multiple lines, so each line will be treated as a separate log on CloudWatch.

Therefore, instead of outputting in multiple lines, output it as a single line log in JSON format.

If it is in JSON format, you can search on CloudWatch.

Use lograge to change the request log in json format. Add lograge to Gemfile and run bundle install.

Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.formatter = Lograge::Formatters::Json.new   
  config.lograge.custom_options = Proc.new do |event|
    exceptions = %w(controller action format id)
    {
      time: event.time,
      host: event.payload[:host],
      remote_ip: event.payload[:remote_ip],
      params: event.payload[:params].except(*exceptions),
      exception_object: event.payload[:exception_object],
      exception: event.payload[:exception],
      backtrace: event.payload[:exception_object].try(:backtrace),
    }.compact
  end      
  ...
end

class ApplicationController < ActionController::Base
  def append_info_to_payload(payload)
    super
    payload[:user_agent] ||= request.user_agent
    payload[:request_id] ||= request.request_id
    if @exception.present?
      payload[:exception_object] ||= @exception
      payload[:exception] ||= [@exception.class, @exception.message]
    end
  end
end

Also, if only this, outside the ActionController, it can not respond to RoutingError that occurs at the ActionDispatch level, so add the following monkey patch

if defined? Lograge
  class ActionDispatch::DebugExceptions
    alias_method :org_log_error, :log_error
    def log_error(request, wrapper)
      msg = {
        exception: wrapper.exception,
        backtrace: wrapper.exception.try(:backtrace),
        raw_request: request,
        raw_wrapper: wrapper
      }      Rails.logger.fatal(msg.to_json)
    end
  end
end

However, the Rails log output in JSON format is just an escaped string.

Therefore, change the setting of fluentd.

In the fluend.yml downloaded in the previous article, add the following to parse the json log. This will cause the log parsed with the key parsed_logto be recorded.

<filter **>
  @type parser
  format json
  key_name log
  reserve_time true
  reserve_data true
  emit_invalid_record_to_error false
  hash_value_field parsed_log
 </filter>

https://s3.amazonaws.com/cloudwatch-agent-k8s-yamls/fluentd/fluentd.yml

Around line 125. Add the above settings and apply them.

Also, if the log stream name is defalut, it becomes the pod name, and if the pod changes, it creates a different log stream.

Personally, I want the log stream to appear in the same log stream even if the pod changes, so take it from the label.

<filter **>
  @type record_transformer
  @id filter_containers_stream_transformer
  enable_ruby # 追加
  <record>
    stream_name ${record["kubernetes"]["labels"]["app"]}
  </record>
</filter>

Summary

You can now monitor the cluster and aggregate application logs with CloudWatch.

Making Nuxt.js web apps compatible with IE11 (Only about Javascript)

Keigo Yamamoto — Fri, 23 Sep 2022 15:56:53 +0000

This post was moved from Medium (https://k5trismegistus.medium.com/making-nuxt-js-web-apps-compatible-with-ie11-only-about-javascript-83f5bbcbe3c8)

Overview

Nuxt.js will generate code which can run on IE9, but this is only for Nuxt.js itself and the code you write, but not for external libraries.

So you need to configure them yourself as long as you use external libraries.

I’ve seen a lot of how-to articles that say that external libraries need to be transpiled, so let’s set them up!

But I couldn’t find anything that answered the question of which one I should do, so I tried to do it myself.

I’m going to add libraries that use the new notation to the transpile list, and load the script for Polyfill.

(What is transpiling and Polyfill is attached at the end of this article.)

Let’s start with the following.

Introducing Polyfill

At polyfill.io, there is a useful site that returns the required Polyfill code for the accessing browser.

You can select features to be polyfilled.

https://polyfill.io/v3/url-builder/

But it’s hard to look up which features need to be polyfilled one by one, so you should load them all together.

If user isusing a modern browser like Firefox or Chrome, and user has updated it properly, user will get empty files back

User have to wait a little more to download that file, but user shouldn’t have any problems with Polyfill overwriting browser-supported features.

export default {  ...  head: { ...    script: [...      { src: 'https://polyfill.io/v3/polyfill.min.js' } // add this    ],  },  ...}

I think I don’t have to do this because Nuxt.js must use polyfill in nuxt build to generate code that is compatible with IE9, but I couldn’t find the way to apply polyfill to external libraries…

Transpiling external libraries

Unlike Polyfill, this one is a bit more difficult.

To include a library in the transpile target, simply write the library name in nuxt.config.js as follows. You can also use a regular expression to specify a group of libraries with a specific pattern of names.

export default {  ...  build: { ...    transpile: ['vuetify'], { ...  },  ...}

At first I thought I’d take the easy way out and put

transpile: [/. +/]  // You can use regex

but sadly, the current project broke even in Chrome. Also, the build time got quite long. You shouldn’t do that…

So, you should detect which libraries must be transpiled one by one, but it was hard way.

Since I didn’t have a Windows testing machine, I could only borrow a Windows PC from my colleague and have access to production build from it.

This is why I had to do something a bit complicated.

Detect which libraries must be transpiled

1. Find the syntax error in IE

Open the page using IE in the production environment.

Open the console of the developer tools and look at the line that is causing the error to see what is happening.

If the error is SCRIPT1002, see if there are any arrow functions or class being used.

The following article was very helpful in identifying the error. (Written in Japanese, you can use translator)

https://qiita.com/siruku6/items/3465fd6e0588ee35cc78

2. Find what library cause syntax error

Start up the Nuxt server in dev mode and access it with Firefox or Chrome.

Analyze the file _nuxt/vendors/app.js, which is a collection of bundled external libraries.

For example, if the arrow function => is causing problems in IE, find the place where the arrow function is used, and look a few lines above it to find the name of the original library.

In the example below, the library named ansi-regex looks suspicious.

Once you know the name, add this library to the target of the nuxt.config.js transpile as before.

Reload and check that the arrow function that was there earlier is gone.

3. repeat steps 1 and 2

It would be nice if all the errors came out in step 1, but we can only find the first non-IE syntax.

I dealt with each one one by crushing the arrow function, then crushing the class, then the next, and so on.
This is what I did, I hope this will help anyone!

Prerequisite knowledge: Transpile and polyfill

To make new Javascript compatible with older browsers, there are two things to do.

One is “transpile”. Transpiling is to rewrite code with the NEW Javascript syntax that would cause syntax errors (ex. Arrow functions)to the old Javascript for the purpose browser’s Javascript interpreter.

For example,

() => {  [1, 2, 3].map(e => console.log(e * 2))} // Arrow function

will be transpiled into

(function () {  [1, 2, 3].map(function (e) {    return console.log(e * 2);  });});

The other is Polyfill. Polyfill will give support of features that is syntactically correct but does not work.

For example, Array.prototype.includes.

[1, 2, 3].includes(3)

Above code is syntactically correct, but don’t work in old browser.

Polyfill code will inject new methods in Array.prototype and make above code work.

I was going to post an example, but it was too long.

https://polyfill.io/v3/polyfill.js?features=Array.prototype.includes

You can see it by accessing with IE or wget.

Using Amazon EFS file system as a permanent volume for EKS cluster

Keigo Yamamoto — Fri, 23 Sep 2022 15:50:17 +0000

This post was moved from Medium. I wrote this in 2019, so content may be out of date.

Introduction

I have been struggling with EKS / Kubernetes in this several months.

As long as the application is containerized, it is best practice to keep the container as little as possible and save the state in the database or object storage, but there may be cases where you want to save the file in the file system.

In such cases, you can mount a non-volatile file system on a pod using Kubernetes’ Persistent Volumes feature.
However, you must prepare the file system to be mounted by yourself.

You can choose several mount targets, but it seems better to mount an NFS file system to be able to read from multiple pods at the same time and share their state.

Therefore, I tried to use Amazon EFS file system, which is a managed NFS file system for peristent volume for EKS cluster.
I thought that EKS and EFS were both AWS managed services, and it would be easy. However I had to spend whole of a day to achieve this. That’s why I wrote this article, I hope that this article will help for you.

Steps

Create an EFS file system

Create an EFS file system from the AWS console.

At this time, add all subnets where EKS worker nodes exist to the mount target, and set up a security groups for EKS worker nodes.

https://aws.amazon.com/jp/about-aws/whats-new/2019/02/deploy-a-kubernetes-cluster-using-amazon-eks-with-new-quick-start/

If you used this quick start, you’ll have xxx-NodeSecurityGroup-yyysecurity groups.
Sorry for not in English

Worker node IAM role

Add the policy for EFS volume created in 1 to the role of EKS worker node. It ’s also called “Quick Start” xxx-NodeInstanceRole-yyy.

If this role is granted “list, read, write” permission for the EFS file system created earlier, it is OK.

efs-provisioner settings

To use EFS as a Kubernetes persistent volume, efs-provisioneryou need to use.

https://github.com/kubernetes-retired/external-storage/tree/master/aws/efs

However, there is a trap that this does not work if you follow the README.
This is because the sample manifests in the repository areincorrect. . 😡

I fount a solution in this issue.

https://github.com/kubernetes-incubator/external-storage/issues/1209

If you just want to mount EFS file system for Pods, you have to apply 2 yaml files,

https://github.com/kubernetes-incubator/external-storage/blob/master/aws/efs/deploy/rbac.yaml

and
https://github.com/kubernetes-incubator/external-storage/blob/master/aws/efs/deploy/manifest.yaml

Both need to be fixed.

First of all, rbac.yaml, there are definitions of Role and ClusterRole, but I’m going to unify them into ClusterRole. You must also add a description of the service role.

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: efs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-efs-provisioner
subjects:
  - kind: ServiceAccount
    name: efs-provisioner
    namespace: development # Set namespace
roleRef:
  kind: ClusterRole
  name: efs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: development # Set namespace
  name: efs-provisioner

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: development # Set namespace
  name: efs-provisioner
  spec:
    serviceAccount: efs-provisioner
    containers:
      - name: efs-provisioner
        image: quay.io/external_storage/efs-provisioner:latest

Mount EFS filesystem

Then, you can use EFS filesystem from your Pods.

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: gcr.io/google_containers/busybox:1.24
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: efs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: efs-pvc
      persistentVolumeClaim:
        claimName: efs

Directory splitting

As you can see if you create an appropriate EC2 instance and mount the EFS file system, the directory is cut for each PersistentVolumeClaim, and each pod can only see inside it.
So, if you want to use one EFS file system for multiple pods, you can define multiple PersistentVolumeClaim sharing one StorageClass .

Conclusion

Now the EFS file system can now be used as a Pod’s persistent volume running on an EKS cluster.
I thought that EKS and EFS are both AWS products, so it is common use case that combine them. But unexpectedly I could not find good information, so I hope thiswould be helpful.

Nexxtrack

Keigo Yamamoto — Thu, 09 Sep 2021 11:37:00 +0000

Hello, I'm Keigo, new to dev.to.

Let me introduce my works. The first one is Nexxtrack.
I do DJing for hobby and wanted to improve my skill with hacking.
So I analyzed popular mix created by pro DJs and created recommend app.

This app show what song tend to be mixed with certain song.

Using Nuxt.js for frontend, flask for api.

If you have interest, please try and give me feedbacks.

Thanks.

https://nexxtrack.club/

I wrote an introduction on medium too.
https://k5trismegistus.medium.com/for-dj-beginners-having-trouble-selecting-songs-use-nexxtrack-8a3cfdd1d5a0