DEV Community: Ladies In DevOps

Load Balancing

Nancy Chauhan — Tue, 28 Sep 2021 05:58:15 +0000

We encounter load balancers every day. Even when you are reading this article, your requests flow through multiple load balancers, before this content reaches your browser.

Load balancing is one of the most important and basic concepts we encounter every single day. It is the process of distributing incoming requests across multiple servers/processes/machines at the backend.

Why do we need load balancing?

Usually, when we make an application, clients will route their request to one of the backend servers, but soon as traffic grows, that server will reach its limits. To overcome this, we can spin up another server to share the traffic. But how to let the clients know to connect to the new machine?
Load balancing is the technique used for discovery and decision-making for this routing. There are two ways of achieving this — server-side load balancing or client-side load balancing.

Single application server gets overloaded with request

Server-side Load Balancing:

There is a middle layer, a load balancer that forwards the incoming requests to different servers to remove that complexity. All backend servers get registered with a load balancer which then routes to one of the server instances using various algorithms. AWS ELB, Nginx, Envoy are some examples of server-side load balancers.

Server-side load balancing

Advantages

No need for client-side changes.
Easy to make changes to load balancing algorithms and backend servers.

Client-Side Load Balancing:

In client-side load balancing, the client handles the load balancing. Let’s take an abstract look at how this can be achieved. To perform load balancing on the client-side -

The client should be aware of all available web servers
A library on the client-side to implement a load balancing algorithm

The client routes the requests to one of the servers using client-side load balancing libraries like Ribbon. Client-side load balancing is also used for service discovery.

Suppose Service A (client-side) wants to access Service B (server-side). Service B has three instances and register all at the discovery server (X). Service A has enabled the Ribbon client which allows doing the client-side load balancing. It fetches the available Service B instances from the discovery server and redirects the traffic from the client-side and constantly listens for any changes.

Here I have implemented client-side load balancing using consul service discovery: https://github.com/Nancy-Chauhan/consul-service-discovery

Server-side load balancing

Advantages

No need for additional infrastructure.

Benefits of Loadbalancing

Reference: https://www.nginx.com/resources/glossary/load-balancing/

Load balancers are the foundation of modern cloud-native applications. The concept of load balancing and the ability to be dynamically configured has created innovations such as service mesh.

Branch Protection in GitHub

Mbaoma — Sat, 12 Jun 2021 08:53:18 +0000

Ever been in a position where you wish you could prevent your teammates from merging unapproved code from a development branch to the main branch?

Do you want to prevent merging code which you are not sure of its build status to your main branch?

Recently, I found myself in this situation and I plan to share a concept which helped me out - 'Branch Protection in GitHub'.

What is Branch Protection?

Branch protection is the act of setting rules to prevent certain actions from occurring on your branch(es) without your approval.

This article focuses on, preventing branches (development etc) from being merged to the main branch; such that before any merge can occur, a pull request would require a selected reviewer to review the request and then merge the commit.

Prerequisites

It is expected that you have prior knowledge of:

Github
CI/CD tools (in this article, Travis CI)

Check out this guides for an introduction to Github and creating a simple .travis.yml file

Setting up branch protection rules

We take the following steps:

Click on the Settings option in your repository and then Branches (located on the left hand side of the page) - Click on Add Rule to create the rule(s) for your branch of choice

Next, under Branch name pattern type in the name of the branch you want to protect
For this article, we choose the following rules:
- 'Require pull request reviews before merging': we limit the number of required reviews to 1 (you can choose to increase the required reviews).
- Then, we select Include administrators , to ensure that as owners of the branch, our pull requests will have to be reviewed before a merge can occur (I mean, nobody is above mistakes 🥴)
Finally, we click on the 'Save changes' button to save our settings.

Setting up our Travis CI script

According to the Travis CI documentation, 'Travis CI supports your development process by automatically building and testing code changes, providing immediate feedback on the success of the change. Travis CI can also automate other parts of your development process by managing deployments and notifications.'

It is a Continuous Integration/Continuous Deployment tool which automatically runs the test(s) you specify in a .travis.yml file and sends you a report stating the build status of your commit, in this way, broken code is prevented from being pushed to production.

A simple Travis script can be written as follows:

language: python
python:
  - "3.6"      # current default Python on Travis CI

# command to install dependencies
install:
  - pip install -r requirements.txt

# command to run tests
script:
  - python -m unittest test

# safelist
branches:
  only:
  - main
  - dev

From the above script, and in other Travis scripts, commands are used to perform different operations. The ones used here are:

language: This is used to specify the programming language in which our code is written (in this case Python).
python: We can specify the language version to run our tests against.
install: This is used to specify the language specific command to install dependencies upon which our code is dependent.
script: This is used to specify the language specific command to run our pre-defined tests.
branches: the 'only' option shows the branches we want to build using a safelist (in this case 'main' and 'dev')

Demo Time

Now, to check out if all our branch protection and CI/CD rules work, we push some code to our secondary branch and open up a pull request.

The pull request will fail.

voila, we are unable to merge our pull request to the main branch (it's the audacity for me😁).
We are told that our pull request needs to be reviewed, so we add a reviewer by clicking on the icon next to 'Reviewers'.
Also, our builds passed (yay!), so our reviewer will be more confident in merging our pull request.

More information can be found in the GitHub Docs.

Feel free to check out my repository on which this article was built

I hope we protect our branches better from now onwards.

Feel free to reach out to me via Linkedin

Selah!!

Enforcing Coding Best Practices using CI

Nancy Chauhan — Sun, 30 May 2021 11:37:02 +0000

High-performing teams usually ship faster, better, and often! Organizations irresepctive of their level, focusing on stability and continuous delivery, will deploy frequently. Hundreds of continuous integration build run for every organization on a typical day. It indicates how CI has become an integral part of our development process. Hence to ensure that we are shipping quality code, we should integrate code quality checking in our CI.

// Detect dark theme var iframe = document.getElementById('tweet-1368692809436303360-796'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1368692809436303360&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1281102326929874944-545'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1281102326929874944&theme=dark" }

Continuous integration ensures easier bug fixes, improves software quality, and reduces project risk. This blog will show what steps we should integrate into our CI pipelines to ensure that we ship better code.

CI Pipeline integrating code quality checks

Traditionally code reviews use to enforce code quality, However checking for things like missing spaces, missing parameters becomes a burden for code reviewers. It would be great if you some tools to automate these checks. We can set some mandatory steps in our CI to run static analysis on the code for every code push. It creates a better development lifecycle by providing early feedback without human intervention.

Source: https://xkcd.com/1285/

Unit Testing

Unit testing is the process of testing discrete functions at the source code. It is ubiquitous that a CI pipeline contains a test job that verifies your code. If the tests fail, the pipeline fails, and users get notified. It allows fixing the code earlier. Unit tests should be fast and should aim to cover 100% of the codebase. It can give enough confidence that the application is functioning correctly at this point. If unit tests are not automated, the feedback cycle would be slow.

Code coverage

Code coverage is a metric that can help you understand how comprehensive are your unit tests. It’s a handy metric that can help you assess the quality of your test suite. Code Coverage Reporting is how we know that all lines of code written have been exercised through testing.

Static code analysis

Static code analysis parses and checks the source code and gives feedback about potential issues in code. It acts as a powerful tool to detect common security vulnerabilities, possible runtime errors, and other general coding errors. It can also enforce your coding guidelines or naming conventions along with your maintainability requirements.

Static code analysis accelerates the feedback cycle in the development process. It gives feedback on new coding issues specific to the branch or commits containing them. It quickly exposes the block of code that we can optimize in terms of quality. By integrating these checks into the CI workflow, we can tackle these code quality issues in the early stages of the delivery.

Linting

Linter is a tool that analyzes source code to flag programming errors, bugs, stylistic errors, and suspicious constructs. It helps in enforcing a standard code style.

We can introduce linter checks in our CI pipelines according to our project setup. There are a vast number of linters out there. Depending on the programming language, there are even more than one linters for each job.

Source: https://xkcd.com/1513/

Linters for Static Analysis

pep8 for Python
PMD for Java
ESLint for Javascript

Linters focused on Security

Bandit for Python
Node Security for JavaScript
SpotBugs with Find sec bugs for Java

Docker lint check

Considering that dockerizing applications are the norm, it is evident how important it is to introduce docker lint checks in our CI pipelines. We should make sure that our docker image generated for our application is optimized and secure.

There are many open source docker linters available :

Secrets checks

Sometimes developers leak GitHub tokens and various secrets in codebases which should be avoided. We should prevent the leaking of secrets when committing code. We can integrate Yelp’s detect-secret in our workflow, which we can use to scan files for secrets and whitelist false positives to reduce the noise.

Dependency Checks

Our code often uses many open source dependencies from public repositories such as maven, PyPI, or npm. These dependencies are maintained by third-party developers who often discover security vulnerabilities in their code. Such vulnerabilities are usually assigned a CVE number and are disclosed publicly to sensitize other developers utilizing their code to update the packages.

Dependency checkers use information from CVE to check for vulnerable dependencies used in our codebase. There are different tools for this:

Snyk for many languages
OWASP Dependency-Check for Java and Python
npm comes with inbuilt dependency checks

All-in-one tools

Some tools aggregate these different static code analysis tools into a single easy to use a package such as:

Sonarqube: Broad analysis tool
Semgrep tool: Used for go, java, python

These tools provide an easy-to-use GUI to find, track and assign issues to developers.

Conclusion

Having the above-discussed steps as a part of your CI/CD pipeline will allow you to monitor, quickly rectify, and grow your code with much higher code quality.

Originally Posted at Medium

Designing Idempotent APIs

Nancy Chauhan — Fri, 14 May 2021 07:09:11 +0000

Networks fail! Timeouts, outages, and routing problems are bound to happen at any time. It challenges us to design our APIs and clients that will be robust in handling failures and ensuring consistency.

We can design our APIs and systems to be idempotent, which means that they can be called any number of times while guaranteeing that side effects only occur once. Let’s get a deeper dive into why incorporating idempotency is essential, how it works, and how to implement it.

Why Idempotency is critical in backend applications?

Consider the design of a social networking site like Instagram where a user can share a post with all their followers. Let’s assume that we are hosting the app-server and database-server on two different machines for better performance and scalability. And also, we are using PostgreSQL to store the data. A post and creating a post will have the following model:

CREATE TABLE public.posts (
   id int(11) PRIMARY KEY,
   user_id int(11) REFERENCES users,
   image_id int(11) REFERENCES images NULL,
   content character varying(2048) COLLATE pg_catalog."default",
   create_timestamp timestamp with time zone NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Failures and Retries

If we have our database on a separate server from our application server, sometimes posts will fail because of network issues. There could be the following issues:

The initial connection could fail as the application server tries to connect to a database server.
The call could fail midway while the app server is fulfilling the operation, leaving the work in limbo.
The call could succeed, but the connection breaks before the database server can tell the application server about it.

We can fix this with retry logic, but it is very tough to suspect the real cause of network failure. Hence it could lead to a scenario where the entry of the post has been made in the database but it could not send the ACK to the app server. Here app server unknowingly keeps retrying and creating duplicate posts. This would eventually lead to business loss. There are many other critical systems like payments, shopping sites, where idempotent systems are quite important.

Solution

The solution to this is to retry, but make the operation idempotent. If an operation is idempotent, the app server can make that same call repeatedly while producing the same result.

In our design, we can use universally unique identifiers. Each post will be given its own UUID by our application server. We can change our models to have a unique key constraint.


CREATE TABLE public.posts (
   id uuid PRIMARY KEY,
   user_id uuid REFERENCES users,
   image_id uuid REFERENCES images NULL,
   content character varying(2048) COLLATE pg_catalog."default",
   create_timestamp timestamp with time zone NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO posts (id, user_id, image_id, content)
VALUES ("DC2FB40E-058F-4208-B9A3-EB1790C532C8", "20C5ADC5-D1A5-4A1F-800F-1AADD1E4E954", "3CC32CAE-B6AC-4C53-97EC-25EB49F2E7F3", "Hello-world") RETURNING id ON CONFLICT DO NOTHING;

Our application server will generate the UUID when it wants to create a post and retry the Insert statement until it gets a successful response from a database server. We need to change our system to handle constraint violations and return the existing post. Hence, there will always be exactly one post created.

Idempotency in HTTP

One of the important aspects of HTTP is the concept that some methods are idempotent. Take GET for an example, how many times you may call the GET method it results in the same outcome. On the other hand, POST is not expected to be an idempotent method, calling it multiple times may result in incorrect updates.

Safe methods don’t change the representation of the resource in the server e.g. GET method should not change the content of the page your accessing. They are read-only methods while the PUT method will update the page but will be idempotent in nature. To be idempotent, only the actual back-end state of the server is considered, the status code returned by each request may differ: the first call of a DELETE will likely return a 200, while successive ones will likely return a 404.

DELETE /idX/delete HTTP/1.1   -> Returns 200 if idX exists
DELETE /idX/delete HTTP/1.1   -> Returns 404 as it just got deleted
DELETE /idX/delete HTTP/1.1   -> Returns 404

GET is both safe and idempotent.
HEAD is also both safe and idempotent.
OPTIONS is also safe and idempotent.
PUT is not safe but idempotent.
DELETE is not safe but idempotent.
POST is neither safe nor idempotent.
PATCH is also neither safe nor idempotent.

The HTTP specification defines certain methods to be idempotent but it is up to the server to actually implement it. For example, send a request-id header with a UUID which the server uses to deduplicate PUT request. If you are serving a GET request, we should not change the server-side data.

Conclusion

Designing idempotent systems is important for building a resilient microservice-based architecture. This helps in solving a lot of problems caused due to the network which is inherently lossy. By leveraging an idempotent queue such as Kafka, it makes sure your operations can be retried in case of a long outage. This helps you to design systems that never lose data and any missing data can be adjusted by replaying the message queue. If all operations are idempotent it will result in the same state regardless of how many times messages are processed.

Originally published at https://medium.com/@_nancychauhan/idempotency-in-api-design-bc4ea812a881

Building a Prometheus Exporter

Nancy Chauhan — Mon, 03 May 2021 11:34:39 +0000

Prometheus is an open-source monitoring tool for collecting metrics from your application and infrastructure. As one of the foundations of the cloud-native environment, Prometheus has become the de-facto standard for visibility in the cloud-native landscape.

How Prometheus Works?

Prometheus is a time-series database and a pull-based monitoring system. It periodically scrapes HTTP endpoints (targets) to retrieve metrics. It can monitor targets such as servers, databases, standalone virtual machines, etc.
Prometheus read metrics exposed by target using a simple text-based exposition format. There are client libraries that help your application to expose metrics in Prometheus format.

Prometheus Metrics

While working with Prometheus it is important to know about Prometheus metrics. These are the four types of metrics that will help in instrumenting your application:

Counter (the only way is up): Use counters for counting events, jobs, money, HTTP request, etc. where a cumulative value is useful.
Gauges (the current picture): Use where the current value is important — CPU, RAM, JVM memory usage, queue levels, etc.
Histograms (Sampling Observations): Generally use with timings, where an overall picture over a time frame is required — query times, HTTP response times.
Summaries (client-side quantiles): Similar in spirit to the Histogram, with the difference being that quantiles are calculated on the client-side as well. Use when you start using quantile values frequently with one or more histogram metrics.

Using Prometheus

Prometheus provides client libraries that you can use to add instrumentation to your applications.
The client library exposes your metrics at URLs such as http://localhost:8000/metrics
Configure the URL as one of the targets in Prometheus. Prometheus will now scrape metrics in periodic intervals. You can use visualization tools such as Grafana to view your metrics or configure alerts using Alertmanager via custom rules defined in configuration files.

Prometheus Exporters

Prometheus has a huge ecosystem of exporters. Prometheus exporters bridge the gap between Prometheus and applications that don’t export metrics in the Prometheus format. For example, Linux does not expose Prometheus-formatted metrics. That’s why Prometheus exporters, like the node exporter, exist.

Some applications like Spring Boot, Kubernetes, etc. expose Prometheus metrics out of the box. On the other hand, exporters consume metrics from an existing source and utilize the Prometheus client library to export metrics to Prometheus.

Prometheus exporters can be stateful or stateless. A stateful exporter is responsible for gathering data and exports them using the general metrics format such as counter, gauge, etc. Stateless exporters are exporters that translate metrics from one format to Prometheus metrics format using counter metric family, gauge metric family, etc. They do not maintain any local state instead they show a view derived from another metric source such as JMX. For example, Jenkins Jobmon is a Prometheus exporter for Jenkins which calls Jenkins API to fetch the metrics on every scrape.

grofers / jenkins-jobmon

Prometheus exporter to monitor Jenkins jobs

Jenkins Jobmon

Jenkins exporter for Prometheus in python.

It uses Prometheus custom collector API, which allows making custom collectors by proxying metrics from other systems.

Currently we fetch following metrics:

Metric	Type	Description	Labels
`jenkins_job_monitor_total_duration_seconds_sum`	Gauge	Jenkins build total duration in millis	`jobname`, `group`, `repository`
`jenkins_job_monitor_fail_count`	Gauge	Jenkins build fail counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_total_count`	Gauge	Jenkins build total counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_pass_count`	Gauge	Jenkins build pass counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_pending_count`	Gauge	Jenkins build pending counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_stage_duration`	Gauge	Jenkins build stage duration in ms	`jobname`, `group`, `repository`, `stagename`, `build`
`jenkins_job_monitor_stage_pass_count`	Counter	Jenkins build stage pass count	`jobname`, `group`, `repository`, `stagename`
`jenkins_job_monitor_stage_fail_count`	Counter	Jenkins build stage fail count	`jobname`, `group`, `repository`, `stagename`

Usage

Configuration

Create a file config.yml using this template:

jobs
  example:

…

View on GitHub

Let’s build a generic HTTP server metrics exporter!

We will build a Prometheus exporter for monitoring HTTP servers from logs. It extracts data from HTTP logs and exports it to Prometheus. We will be using a python client library, prometheus_client, to define and expose metrics via an HTTP endpoint.

Our HTTP exporter will repeatedly follow server logs to extract useful information such as HTTP requests, status codes, bytes transferred, and requests timing information. HTTP logs are structured and standardized across different servers such as Apache, Nginx, etc. You can read more about it from here.

127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

We will use a counter metric to store the HTTP requests using status code as a label.
We will use a counter metric to store bytes transferred.

Here is the script which collects data from apache logs indefinitely and exposes metrics to Prometheus :

The follow_log function tails apache logs stored var/log/apache in your system infinitely. gather_metrics() uses a regular expression to fetch the useful information from logs like status_code and total_bytes_sent and accordingly increments the counters.

If you run the script, it will start the server at http://localhost:8000 The collected metrics will show up there. Setup Prometheus to scrape the endpoint. Over time, Prometheus will build the time-series for the metrics collected. Setup Grafana to visualize the data within Prometheus.

You can find the code here and run the exporter:

Nancy-Chauhan / httpd_exporter

Prometheus exporter for monitoring apache

httpd_exporter

Prometheus exporter for monitoring http servers from logs.

It extracts data from http logs and export to prometheus.

Requirements

python 3.6 +

Usage

Clone the repo
Run docker-compose up

Grafana Dashboard

View on GitHub

Originally Posted at https://medium.com/@_nancychauhan/building-a-prometheus-exporter-8a4bbc3825f5