Serverless error-handling pipeline using Redis

Rayan Dasoriya — Sat, 05 Sep 2020 20:39:44 +0000

Errors are an essential part of an application. These are the thing that you don't want to occur, but if they do, then there should be some mechanism to detect them. These errors can be due to some service unavailability, bugs, or any other unforeseen issue. In this article, I am going to highlight how we used IBM Cloud Functions, a serverless offering by IBM, and Redis to log the errors in our Slack channel.

Background

We have a Node.js application that interacts with Salesforce CRM offerings and performs certain operations. There are times when the application gets an error message. Whenever we get these messages, we want to detect and log these errors into our Slack channel so that appropriate actions can be taken.

Technology used

We used IBM Cloud Functions, which is a FaaS(Functions as a Service) platform based on Apache Openwhish. It allows us to run the application without worrying about the configurations. Redis cache was used to store the error logs within our system. It is an in-memory key-value data structure with in-build replication and LRU cache features.

Working

Our Node.js application runs within our IBM Cloud Kubernetes cluster which interacts with Salesforce CRM systems. Whenever it sees an error, we trigger IBM Cloud Functions. At this point, the Node.js application continues its work in the Kubernetes cluster without worrying about the errors. The control for error handling is being transferred to IBM Cloud functions.

IBM Cloud functions try to establish a connection with the Redis cluster and stores the log over there. We categorize the type of error and then check if the error is eligible to be sent over to the Slack channel. If we have to send the message on Slack, we use Slack Webhooks. Once this message is sent, we remove the log from Redis.

We are using Redis to make sure that our logs have been processed. If we get an error next time, we will check if our Redis cluster contains any error messages that need to be processed. It will then retry sending these messages to Slack, thus helping us to ensure we don't miss any errors.

Kubernetes monitoring with Prometheus

Rayan Dasoriya — Thu, 25 Jul 2019 16:49:11 +0000

Microservices architecture is going to be one of the essential features in software development in the coming years. Packing a large monolithic application into small containers poses various advantages. One of the key advantages I can think of is that if certain things fail, then a part of the application will be down and it can auto-heal rather than crashing the whole application. This is one of the reasons that when Instagram, Facebook, and Whatsapp crashes, only certain functionalities stop working. The application, as a whole, is still working. This serves a great advantage. Also, when the demand for a particular service or component increases, the number of components can be increased or decreased easily. The small containers which I am referring here are the Docker containers and the application is deployed on Kubernetes. Kubernetes is a container orchestration software which manages the containers at various levels and enables the management of connections and endpoints of these containers.

Kubernetes or K8s is an open-source self-healing application which manages the deployment, scaling, and operation of these containers. It was originally developed by Google but later on, donated to CNCF. Since Kubernetes provides tons of services, there needs to be an easier way to monitor the activities in the Kubernetes cluster. This is possible by Prometheus. It is an open-source solution to monitor the metrics and manage the alerts in the system. It was developed by SoundCloud, but later joined CNCF as the second hosted project after Kubernetes. Prometheus provided a rich set of monitoring metrics and alert management system which helps the developers to monitor and get notified about any unusual activity or consumption.

Prometheus Operator

CoreOs launched Prometheus operator to ease the process of integrating K8s with Prometheus. It preserves the configuration of both the K8s and Prometheus while installing and configuring the cluster. It provides easy monitoring for K8s services and deployments, along with managing Prometheus, Grafana and Alertmanager configuration.
When a new version of the application is deployed, K8s manages the creation of new pod and deletes the older one. Prometheus, on the other hand, constantly watches the K8s API and creates a new Prometheus configuration whenever it detects a change, based on the services/pods changes. It uses a ServiceMonitor, a CRD(Custom Resource Definition), to abstract the configuration to target.

Installation

Prerequisites

Kubernetes
Helm (Package installer for K8s)

Steps

Install Prometheus operator in a different namespace. It is preferable to keep your monitoring containers in a separate namespace.

$ helm install stable/prometheus-operator --name prometheus-operator --namespace monitor

If everything got installed perfectly, you can see these pods available:

$ kubectl get pods -n monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0          2/2     Running   0          13d
prometheus-operator-grafana-749b598b6c-t4r48             2/2     Running   0          13d
prometheus-operator-kube-state-metrics-d7b8b7666-zfqg5   1/1     Running   0          13d
prometheus-operator-operator-667dd7cbb7-hjbl6            1/1     Running   0          13d
prometheus-operator-prometheus-node-exporter-mgsqb       1/1     Running   0          13d
prometheus-prometheus-operator-prometheus-0              3/3     Running   1          13d

To run the dashboard, enter the following command and go to http://localhost:9000.

$ kubectl port-forward -n monitor prometheus-prometheus-operator-prometheus-0 9090

Prometheus Dashboard

You can enter your query to get the results about any particular instance or even a graph of it as shown in the figure above.
To see the visual representation at each level, we use Grafana. It provides some great visual insights regarding the usage, health and other metrics. We can also add more custom metrics. We will get real-time analysis of the data.

$ kubectl port-forward $(kubectl get  pods --selector=app=grafana -n  monitor --output=jsonpath="{.items..metadata.name}") -n monitor  3000

Go to http://localhost:3000 and enter ‘admin’ as username and ‘prom-operator’ as password. These are the available options:

You can get visual graphs by selecting any one of the options. Node-level metrics are shown here:

We can configure alerts in many ways. We can access the dashboard to configure the AlertManager by going to http://localhost:9093 after executing this command:

$ kubectl port-forward -n monitor alertmanager-prometheus-operator-alertmanager-0 9093

It’ll look like this. Here you can add more alerts and can see the Slack API URL under the status tab. We can set the notifications on Slack, HipChat or even email. Some of the templates are available here.

Conclusion

The use of microservices is going to increase soon and monitoring the metrics and alert notifications are going to be an essential part of it. Prometheus provides optimal monitoring services with easy installation services.

DEV Community: Rayan Dasoriya