DEV Community: Nancy Chauhan

Load Balancing

Nancy Chauhan — Tue, 28 Sep 2021 05:58:15 +0000

We encounter load balancers every day. Even when you are reading this article, your requests flow through multiple load balancers, before this content reaches your browser.

Load balancing is one of the most important and basic concepts we encounter every single day. It is the process of distributing incoming requests across multiple servers/processes/machines at the backend.

Why do we need load balancing?

Usually, when we make an application, clients will route their request to one of the backend servers, but soon as traffic grows, that server will reach its limits. To overcome this, we can spin up another server to share the traffic. But how to let the clients know to connect to the new machine?
Load balancing is the technique used for discovery and decision-making for this routing. There are two ways of achieving this — server-side load balancing or client-side load balancing.

Single application server gets overloaded with request

Server-side Load Balancing:

There is a middle layer, a load balancer that forwards the incoming requests to different servers to remove that complexity. All backend servers get registered with a load balancer which then routes to one of the server instances using various algorithms. AWS ELB, Nginx, Envoy are some examples of server-side load balancers.

Server-side load balancing

Advantages

No need for client-side changes.
Easy to make changes to load balancing algorithms and backend servers.

Client-Side Load Balancing:

In client-side load balancing, the client handles the load balancing. Let’s take an abstract look at how this can be achieved. To perform load balancing on the client-side -

The client should be aware of all available web servers
A library on the client-side to implement a load balancing algorithm

The client routes the requests to one of the servers using client-side load balancing libraries like Ribbon. Client-side load balancing is also used for service discovery.

Suppose Service A (client-side) wants to access Service B (server-side). Service B has three instances and register all at the discovery server (X). Service A has enabled the Ribbon client which allows doing the client-side load balancing. It fetches the available Service B instances from the discovery server and redirects the traffic from the client-side and constantly listens for any changes.

Here I have implemented client-side load balancing using consul service discovery: https://github.com/Nancy-Chauhan/consul-service-discovery

Server-side load balancing

Advantages

No need for additional infrastructure.

Benefits of Loadbalancing

Reference: https://www.nginx.com/resources/glossary/load-balancing/

Load balancers are the foundation of modern cloud-native applications. The concept of load balancing and the ability to be dynamically configured has created innovations such as service mesh.

Enforcing Coding Best Practices using CI

Nancy Chauhan — Sun, 30 May 2021 11:37:02 +0000

High-performing teams usually ship faster, better, and often! Organizations irresepctive of their level, focusing on stability and continuous delivery, will deploy frequently. Hundreds of continuous integration build run for every organization on a typical day. It indicates how CI has become an integral part of our development process. Hence to ensure that we are shipping quality code, we should integrate code quality checking in our CI.

// Detect dark theme var iframe = document.getElementById('tweet-1368692809436303360-796'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1368692809436303360&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1281102326929874944-545'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1281102326929874944&theme=dark" }

Continuous integration ensures easier bug fixes, improves software quality, and reduces project risk. This blog will show what steps we should integrate into our CI pipelines to ensure that we ship better code.

CI Pipeline integrating code quality checks

Traditionally code reviews use to enforce code quality, However checking for things like missing spaces, missing parameters becomes a burden for code reviewers. It would be great if you some tools to automate these checks. We can set some mandatory steps in our CI to run static analysis on the code for every code push. It creates a better development lifecycle by providing early feedback without human intervention.

Source: https://xkcd.com/1285/

Unit Testing

Unit testing is the process of testing discrete functions at the source code. It is ubiquitous that a CI pipeline contains a test job that verifies your code. If the tests fail, the pipeline fails, and users get notified. It allows fixing the code earlier. Unit tests should be fast and should aim to cover 100% of the codebase. It can give enough confidence that the application is functioning correctly at this point. If unit tests are not automated, the feedback cycle would be slow.

Code coverage

Code coverage is a metric that can help you understand how comprehensive are your unit tests. It’s a handy metric that can help you assess the quality of your test suite. Code Coverage Reporting is how we know that all lines of code written have been exercised through testing.

Static code analysis

Static code analysis parses and checks the source code and gives feedback about potential issues in code. It acts as a powerful tool to detect common security vulnerabilities, possible runtime errors, and other general coding errors. It can also enforce your coding guidelines or naming conventions along with your maintainability requirements.

Static code analysis accelerates the feedback cycle in the development process. It gives feedback on new coding issues specific to the branch or commits containing them. It quickly exposes the block of code that we can optimize in terms of quality. By integrating these checks into the CI workflow, we can tackle these code quality issues in the early stages of the delivery.

Linting

Linter is a tool that analyzes source code to flag programming errors, bugs, stylistic errors, and suspicious constructs. It helps in enforcing a standard code style.

We can introduce linter checks in our CI pipelines according to our project setup. There are a vast number of linters out there. Depending on the programming language, there are even more than one linters for each job.

Source: https://xkcd.com/1513/

Linters for Static Analysis

pep8 for Python
PMD for Java
ESLint for Javascript

Linters focused on Security

Bandit for Python
Node Security for JavaScript
SpotBugs with Find sec bugs for Java

Docker lint check

Considering that dockerizing applications are the norm, it is evident how important it is to introduce docker lint checks in our CI pipelines. We should make sure that our docker image generated for our application is optimized and secure.

There are many open source docker linters available :

Secrets checks

Sometimes developers leak GitHub tokens and various secrets in codebases which should be avoided. We should prevent the leaking of secrets when committing code. We can integrate Yelp’s detect-secret in our workflow, which we can use to scan files for secrets and whitelist false positives to reduce the noise.

Dependency Checks

Our code often uses many open source dependencies from public repositories such as maven, PyPI, or npm. These dependencies are maintained by third-party developers who often discover security vulnerabilities in their code. Such vulnerabilities are usually assigned a CVE number and are disclosed publicly to sensitize other developers utilizing their code to update the packages.

Dependency checkers use information from CVE to check for vulnerable dependencies used in our codebase. There are different tools for this:

Snyk for many languages
OWASP Dependency-Check for Java and Python
npm comes with inbuilt dependency checks

All-in-one tools

Some tools aggregate these different static code analysis tools into a single easy to use a package such as:

Sonarqube: Broad analysis tool
Semgrep tool: Used for go, java, python

These tools provide an easy-to-use GUI to find, track and assign issues to developers.

Conclusion

Having the above-discussed steps as a part of your CI/CD pipeline will allow you to monitor, quickly rectify, and grow your code with much higher code quality.

Originally Posted at Medium

Designing Idempotent APIs

Nancy Chauhan — Fri, 14 May 2021 07:09:11 +0000

Networks fail! Timeouts, outages, and routing problems are bound to happen at any time. It challenges us to design our APIs and clients that will be robust in handling failures and ensuring consistency.

We can design our APIs and systems to be idempotent, which means that they can be called any number of times while guaranteeing that side effects only occur once. Let’s get a deeper dive into why incorporating idempotency is essential, how it works, and how to implement it.

Why Idempotency is critical in backend applications?

Consider the design of a social networking site like Instagram where a user can share a post with all their followers. Let’s assume that we are hosting the app-server and database-server on two different machines for better performance and scalability. And also, we are using PostgreSQL to store the data. A post and creating a post will have the following model:

CREATE TABLE public.posts (
   id int(11) PRIMARY KEY,
   user_id int(11) REFERENCES users,
   image_id int(11) REFERENCES images NULL,
   content character varying(2048) COLLATE pg_catalog."default",
   create_timestamp timestamp with time zone NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Failures and Retries

If we have our database on a separate server from our application server, sometimes posts will fail because of network issues. There could be the following issues:

The initial connection could fail as the application server tries to connect to a database server.
The call could fail midway while the app server is fulfilling the operation, leaving the work in limbo.
The call could succeed, but the connection breaks before the database server can tell the application server about it.

We can fix this with retry logic, but it is very tough to suspect the real cause of network failure. Hence it could lead to a scenario where the entry of the post has been made in the database but it could not send the ACK to the app server. Here app server unknowingly keeps retrying and creating duplicate posts. This would eventually lead to business loss. There are many other critical systems like payments, shopping sites, where idempotent systems are quite important.

Solution

The solution to this is to retry, but make the operation idempotent. If an operation is idempotent, the app server can make that same call repeatedly while producing the same result.

In our design, we can use universally unique identifiers. Each post will be given its own UUID by our application server. We can change our models to have a unique key constraint.


CREATE TABLE public.posts (
   id uuid PRIMARY KEY,
   user_id uuid REFERENCES users,
   image_id uuid REFERENCES images NULL,
   content character varying(2048) COLLATE pg_catalog."default",
   create_timestamp timestamp with time zone NOT NULL DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO posts (id, user_id, image_id, content)
VALUES ("DC2FB40E-058F-4208-B9A3-EB1790C532C8", "20C5ADC5-D1A5-4A1F-800F-1AADD1E4E954", "3CC32CAE-B6AC-4C53-97EC-25EB49F2E7F3", "Hello-world") RETURNING id ON CONFLICT DO NOTHING;

Our application server will generate the UUID when it wants to create a post and retry the Insert statement until it gets a successful response from a database server. We need to change our system to handle constraint violations and return the existing post. Hence, there will always be exactly one post created.

Idempotency in HTTP

One of the important aspects of HTTP is the concept that some methods are idempotent. Take GET for an example, how many times you may call the GET method it results in the same outcome. On the other hand, POST is not expected to be an idempotent method, calling it multiple times may result in incorrect updates.

Safe methods don’t change the representation of the resource in the server e.g. GET method should not change the content of the page your accessing. They are read-only methods while the PUT method will update the page but will be idempotent in nature. To be idempotent, only the actual back-end state of the server is considered, the status code returned by each request may differ: the first call of a DELETE will likely return a 200, while successive ones will likely return a 404.

DELETE /idX/delete HTTP/1.1   -> Returns 200 if idX exists
DELETE /idX/delete HTTP/1.1   -> Returns 404 as it just got deleted
DELETE /idX/delete HTTP/1.1   -> Returns 404

GET is both safe and idempotent.
HEAD is also both safe and idempotent.
OPTIONS is also safe and idempotent.
PUT is not safe but idempotent.
DELETE is not safe but idempotent.
POST is neither safe nor idempotent.
PATCH is also neither safe nor idempotent.

The HTTP specification defines certain methods to be idempotent but it is up to the server to actually implement it. For example, send a request-id header with a UUID which the server uses to deduplicate PUT request. If you are serving a GET request, we should not change the server-side data.

Conclusion

Designing idempotent systems is important for building a resilient microservice-based architecture. This helps in solving a lot of problems caused due to the network which is inherently lossy. By leveraging an idempotent queue such as Kafka, it makes sure your operations can be retried in case of a long outage. This helps you to design systems that never lose data and any missing data can be adjusted by replaying the message queue. If all operations are idempotent it will result in the same state regardless of how many times messages are processed.

Originally published at https://medium.com/@_nancychauhan/idempotency-in-api-design-bc4ea812a881

Top 10 Productive Hacks for Software Developers

Nancy Chauhan — Tue, 04 May 2021 13:44:03 +0000

Recently, I posted a tweet asking all the amazing software developers in my network to tell about the hacks they use to keep themselves productive. I have compiled the wonderful and helpful solutions I received. Hope you find them useful as well.

Nancy Chauhan

@_nancychauhan

What hacks do you follow as a software developer that enhance your productivity or make your life easier? For instance, Bash aliases for frequently used commands works wonders for me.

12:18 PM - 14 Apr 2021

When I started my tech career, I found that I often had to do a lot of repetitive and manual work. I wanted to improve my skills and boost my productivity.
When I met other developers, I discovered various hacks. I was always amazed that how easy things can become.

Automation

We usually repeat a set of tasks every day which are time-consuming. You can automate them. Tasks such as compiling code after minute changes or migrating data into databases after minor modifications can be automated. Following are the few takeaways :

Jakub Pomykała 👋

@jakub_pomykala

@_nancychauhan I use a bash script to reload test data on the local SQL database or stop all containers and set up a new stack. But the thing that really makes a trick is that I’m using the touch bar on my MacBook to run those scripts.

01:57 AM - 15 Apr 2021

Bespoke python or bash scripts for automating jobs at work
Use cookie-cutter templates to automate project creation. Example: https://github.com/cookiecutter-flask/cookiecutter-flask
This blog is an amazing piece about managing the scripts and binaries downloaded randomly from the internet.
Tools like espanso allow substituting strings like date to present date, etc.

Use Fish or Zsh with Multiplexing terminals

A terminal multiplexer is a program that allows its user to multiplex one or more virtual sessions, so the user can have several sessions inside one single terminal like tmux, iterm, terminator etc. It is a must-have in your toolkit to gain greater control over the shells by working with a terminal multiplexer.
There are a lot of widely available shells such as Bash, Zsh, and fish shells. Switching to fish or even Zsh is one of the best things you could do to make your programming experience more pleasant. It’s faster and much more customizable than Bash.

Zsh & Fish

Zsh has many useful features, including spelling correction, sharing your command history across multiple terminals, naming directory shortcuts, etc.
For productivity, I use iTerm + ZSH + oh-my-zsh. You can enrich ZSH by using the Oh My ZSH framework, which provides some functionality that will boost your efficiency. A few of my favorite plugins are:-

zsh-autosuggestions: suggests commands as you type based on history and completions
zsh-syntax-highlighting: provides syntax highlighting for the shell zsh, red for invalid, and green for valid commands:
ZSH-z: It is a command-line tool that allows you to jump quickly to directories that you have visited frequently in the past or recently.

Varun

@asdeyquote

@_nancychauhan fish shell. startup time for the prompt to show up in zshell is enough to lose my chain of thoughts. plus most of the plugins that you have to install with zsh comes out of the box with fish.

15:04 PM - 15 Apr 2021

On the other hand, Fish is full of awesome features that will skyrocket your productivity to a different level altogether. It is extremely well documented and it’s easy to install too. I want to use it as well.

Use and know your IDE very well

Kushal Das

@kushaldas

@_nancychauhan [0] Touch typing. [1] Learn any proper text editor.

07:19 AM - 15 Apr 2021

You should learn to use your favourite IDE efficiently. If you get to know the features and capabilities of IDE very well, you can significantly improve your productivity. For me, that mainly involves becoming familiar with the most commonly used commands and learning their key shortcuts.

Clipboard manager

Arnav Gupta

@championswimmer

@_nancychauhan I use this tool called Clipy (there's a similar called Clipmenu too) which keeps your last 30 cut/copies items. Not just text but images too.

When copying 4-5 items from one place to another I can first copy them all, then paste in one go.

07:26 AM - 15 Apr 2021

Clipboard managers can be extremely useful and increase your productivity. Clipboard manager can be used to recall text that have been copied such as webservice endpoints, test user names and passwords, and code snippets. We can use different tools like Clipy, Klipper, Clipman.

Gmail labels, and filters to keep annoying mails out of life

Amitosh Swain Mahapatra

@recrsn

@_nancychauhan fzf
bespoke python scripts for automating jobs at my work
dig to internal DNS server to find a VM by IP/name
cookie-cutter templates for some projects (shared with team as well)
Clipboard manager
Gmail labels, and filters to keep annoying mails out of life

16:29 PM - 14 Apr 2021

A filter can save your time and space, rid your inbox of unwanted emails, and turn your Gmail into a multi-functional tool with simple filters.
This way, you can filter out spam and emails of less priority and save emails that you wouldn’t want to miss. For instance, you can have Gmail filter out and label all emails from a specific email address. You will then have them conveniently organized under one label. Gmail shares how to create, edit and delete labels here and how to use filters here.

Nap and Breaks

Arihant Verma

@gdadsriver

@_nancychauhan Taking a nap

12:24 PM - 14 Apr 2021

While it’s important to focus on work, it’s as important to focus on your rest. You should take frequent breaks. The Pomodoro technique is something that I want to try for a long time. Every focus session should have at least a 5 to 10-minute break. Also, I believe exercise is quite important as well. It keeps your day productive.

A place to put all tasks, Github issues, Jira tickets, private stuff

Maciej Walkowiak 🍃

@maciejwalkowiak

@_nancychauhan I put all my tasks, github issues, jira tickets, private stuff to @MicrosoftToDo so that i can quickly pick things to do without thinking and searching for them. Works well when waiting for build etc

15:39 PM - 14 Apr 2021

You can use various tools or ways to accumulate and organize all your tasks and learnings in one place. You can jump to another task when blocked on some task due to some reason. Also, you can maintain short notes and learnings which you might need for future reference.
Some of the tools which you can use are:

Taskwarrior
Notion
Microsoft to do
Short notes and learnings (I maintain short notes of my learnings on Github)

Nancy-Chauhan / Today-I-Learnt

TIL: 26-02-2021

DHCP Protocol is used to assign IP addresses It is present in router and it is also used in Docker, Kubernetes, AWS and etc.

Docker: By default, the container is assigned an IP address for every Docker network it connects to. The IP address is assigned from the pool assigned to the network, so the Docker daemon effectively acts as a DHCP server for each container.

TIL: 07-03-2021

Idempotency: Idempotence is talked about a lot in the context of "RESTful" web services. Basically, if making multiple identical requests has the same effect as making a single request it is idempotency.

TIL: 08-03-2021

Interface vs abstract class in Java

When to use what?
Interfaces can be used if we want a full implementation and use abstract classes when you want partial pieces for your design.

TIL: 10-03-2021

Problem : You are calling resource even when it is not needed…

View on GitHub

Touch typing

It provides the ability to work more ergonomically. It helps in syncing code/any work with thoughts. Also, as a developer, we are not only focused on writing actual code. There are many more things like code reviews, documentation, and slack messages that depend on typing to contribute to a product, team or discussion. For programmers, the main benefit of touch-typing comes from being able to spot typing errors as soon as they occur.

Linux Commands and Shortcuts

Learning Linux commands and shortcuts helps a lot. Linux command tricks will save you a lot of time and, in some cases, from plenty of frustration. Some of my favourite tricks are:

Using grep when dealing with large chunks of data
Using alias to fix typos or setup various keys
Using tab for autocompletion
Easily search and use the command you have used in the past with Ctrl + r search_term

Focus

Avinash Jain

@logicbomb_1

@_nancychauhan - using google addon to block social networking sites during office hours.
- using github.com/httpie/httpie instead of curl.
- play science games testtubegames.com when I get bored working.
- using Gmail multiple inbox feature to keep mails which I need to respond separate.

17:13 PM - 14 Apr 2021

We can use various addons to block the usage of social networking sites and keep the distractions away.

In the end, I am thankful to all the developers for their valuable inputs.

Originally Published on Medium

Building a Prometheus Exporter

Nancy Chauhan — Mon, 03 May 2021 11:34:39 +0000

Prometheus is an open-source monitoring tool for collecting metrics from your application and infrastructure. As one of the foundations of the cloud-native environment, Prometheus has become the de-facto standard for visibility in the cloud-native landscape.

How Prometheus Works?

Prometheus is a time-series database and a pull-based monitoring system. It periodically scrapes HTTP endpoints (targets) to retrieve metrics. It can monitor targets such as servers, databases, standalone virtual machines, etc.
Prometheus read metrics exposed by target using a simple text-based exposition format. There are client libraries that help your application to expose metrics in Prometheus format.

Prometheus Metrics

While working with Prometheus it is important to know about Prometheus metrics. These are the four types of metrics that will help in instrumenting your application:

Counter (the only way is up): Use counters for counting events, jobs, money, HTTP request, etc. where a cumulative value is useful.
Gauges (the current picture): Use where the current value is important — CPU, RAM, JVM memory usage, queue levels, etc.
Histograms (Sampling Observations): Generally use with timings, where an overall picture over a time frame is required — query times, HTTP response times.
Summaries (client-side quantiles): Similar in spirit to the Histogram, with the difference being that quantiles are calculated on the client-side as well. Use when you start using quantile values frequently with one or more histogram metrics.

Using Prometheus

Prometheus provides client libraries that you can use to add instrumentation to your applications.
The client library exposes your metrics at URLs such as http://localhost:8000/metrics
Configure the URL as one of the targets in Prometheus. Prometheus will now scrape metrics in periodic intervals. You can use visualization tools such as Grafana to view your metrics or configure alerts using Alertmanager via custom rules defined in configuration files.

Prometheus Exporters

Prometheus has a huge ecosystem of exporters. Prometheus exporters bridge the gap between Prometheus and applications that don’t export metrics in the Prometheus format. For example, Linux does not expose Prometheus-formatted metrics. That’s why Prometheus exporters, like the node exporter, exist.

Some applications like Spring Boot, Kubernetes, etc. expose Prometheus metrics out of the box. On the other hand, exporters consume metrics from an existing source and utilize the Prometheus client library to export metrics to Prometheus.

Prometheus exporters can be stateful or stateless. A stateful exporter is responsible for gathering data and exports them using the general metrics format such as counter, gauge, etc. Stateless exporters are exporters that translate metrics from one format to Prometheus metrics format using counter metric family, gauge metric family, etc. They do not maintain any local state instead they show a view derived from another metric source such as JMX. For example, Jenkins Jobmon is a Prometheus exporter for Jenkins which calls Jenkins API to fetch the metrics on every scrape.

grofers / jenkins-jobmon

Prometheus exporter to monitor Jenkins jobs

Jenkins Jobmon

Jenkins exporter for Prometheus in python.

It uses Prometheus custom collector API, which allows making custom collectors by proxying metrics from other systems.

Currently we fetch following metrics:

Metric	Type	Description	Labels
`jenkins_job_monitor_total_duration_seconds_sum`	Gauge	Jenkins build total duration in millis	`jobname`, `group`, `repository`
`jenkins_job_monitor_fail_count`	Gauge	Jenkins build fail counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_total_count`	Gauge	Jenkins build total counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_pass_count`	Gauge	Jenkins build pass counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_pending_count`	Gauge	Jenkins build pending counts	`jobname`, `group`, `repository`
`jenkins_job_monitor_stage_duration`	Gauge	Jenkins build stage duration in ms	`jobname`, `group`, `repository`, `stagename`, `build`
`jenkins_job_monitor_stage_pass_count`	Counter	Jenkins build stage pass count	`jobname`, `group`, `repository`, `stagename`
`jenkins_job_monitor_stage_fail_count`	Counter	Jenkins build stage fail count	`jobname`, `group`, `repository`, `stagename`

Usage

Configuration

Create a file config.yml using this template:

jobs
  example:

…

View on GitHub

Let’s build a generic HTTP server metrics exporter!

We will build a Prometheus exporter for monitoring HTTP servers from logs. It extracts data from HTTP logs and exports it to Prometheus. We will be using a python client library, prometheus_client, to define and expose metrics via an HTTP endpoint.

Our HTTP exporter will repeatedly follow server logs to extract useful information such as HTTP requests, status codes, bytes transferred, and requests timing information. HTTP logs are structured and standardized across different servers such as Apache, Nginx, etc. You can read more about it from here.

127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

We will use a counter metric to store the HTTP requests using status code as a label.
We will use a counter metric to store bytes transferred.

Here is the script which collects data from apache logs indefinitely and exposes metrics to Prometheus :

The follow_log function tails apache logs stored var/log/apache in your system infinitely. gather_metrics() uses a regular expression to fetch the useful information from logs like status_code and total_bytes_sent and accordingly increments the counters.

If you run the script, it will start the server at http://localhost:8000 The collected metrics will show up there. Setup Prometheus to scrape the endpoint. Over time, Prometheus will build the time-series for the metrics collected. Setup Grafana to visualize the data within Prometheus.

You can find the code here and run the exporter:

Nancy-Chauhan / httpd_exporter

Prometheus exporter for monitoring apache

httpd_exporter

Prometheus exporter for monitoring http servers from logs.

It extracts data from http logs and export to prometheus.

Requirements

python 3.6 +

Usage

Clone the repo
Run docker-compose up

Grafana Dashboard

View on GitHub

Originally Posted at https://medium.com/@_nancychauhan/building-a-prometheus-exporter-8a4bbc3825f5

Introduction to Message Queue: Build a newsletter app using Django, Celery, and RabbitMQ in 30 min

Nancy Chauhan — Wed, 28 Apr 2021 06:57:19 +0000

Messaging Queues are widely used in asynchronous systems. In a data-intensive application using queues makes sure users have a fast experience while still completing complicated tasks. For instance, you can show a progress bar in your UI while your task is being completed in the background. This allows the user to relieve themselves from waiting for a task to complete and, hence, can do other jobs during that time.

A typical request-response architecture doesn’t cut where response time is unpredictable because you have many long-running requests coming. If you are sure that your systems request will exponentially or polynomially go large, a queue could be very beneficial.

Messaging queues provide useful features such as persistence, routing, and task management. Message queues are typical ‘brokers’ that facilitate message passing by providing an interface that other services can access. This interface connects producers who create messages and the consumers who then process them.

We will build a newsletter app, where users can subscribe to various newsletters and they will receive the issues regularly on their emails. But before we proceed let’s understand the working of workers + message queues.

Workers & Message Queues

Workers are “background task servers”. While your web server is responding to user requests, the worker servers can process tasks in the background. These workers can be used for sending emails, making large changes in the database, processing files, etc.

Workers are assigned tasks via a message queue. For instance, consider a queue storing a lot of messages. It will be processed in a first-in, first-out (FIFO) fashion. When a worker becomes available, it takes the first task from the front of the queue and begins processing. If we have many workers, each one takes a task in order. The queue ensures that each worker only gets one task at a time and that each task is only being processed by one worker.

We will use Celery which is a task queue implementation for Python web applications used to asynchronously execute work outside the HTTP request-response cycle. We will also use RabbitMQ, which is the most widely deployed open-source message broker. It supports multiple messaging protocols.

Build Newsletter App

We will build a newsletter app where a user can subscribe to various newsletters simultaneously and will receive the issues over their emails regularly.

We will have our newsletter app running as a Django app with celery. Whenever authors publish a new issue the Django app will publish a message to email the issue to the subscribers using celery. Celery workers will receive the task from the broker and start sending emails.

Requirements

Python 3+ version
Pipenv

Setup Django

Create a folder newsletter locally and install Django in a virtual environment. Inside folder run:



pipenv shell
pipenv install django

Create an app:

django-admin startproject newsletter_site .

Setup the models:

python manage.py migrate

Make sure it works and visit http://127.0.0.1:8000/ :

python manage.py runserver 8000

Create the newsletter app:

python manage.py startapp newsletter

Installation

Install celery
Install dotenv for reading settings from the environment.
Install psycopg2-binary for connecting with Postgres.



pipenv install celery
pipenv install python-dotenv
pipenv install psycopg2-binary

Setup Postgres and RabbitMQ

Create a docker-compose.yaml to run Postgres and Rabbitmq in the background.



version: '3'
services:
  db:
    image: postgres:13
    env_file:
      - .env
    ports:
      - 5432:5432
  rabbitmq:
    image: rabbitmq
    ports:
      - 5672:5672

Configuring settings.py

To include the app in our project, we need to add a reference to its configuration class in the INSTALLED_APPS setting in newsletter_site/settings.py.



INSTALLED_APPS = [
    'newsletter.apps.NewsletterConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

We need to tell Celery how to find RabbitMQ. So, open settings.py and add this line:
CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL')
We need to configure database settings:



uri = os.getenv('DATABASE_URL')

result = urlparse(uri)

database = result.path[1:]
user = result.username
password = result.password
host = result.hostname
port = result.port

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': database,
        'USER': user,
        'PASSWORD': password,
        'HOST': host,
        'PORT': port,
    }
}

We need to configure the SMTP server in settings.py . SMTP server is the mail server responsible to deliver emails to the users. For development, you may use a Gmail SMTP server, but this has limits and will not work if you have 2 FA. You can refer to this article. For production, you can use commercial services such as sendgrid.



EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = os.getenv('EMAIL_HOST')
EMAIL_USE_TLS = bool(os.getenv('EMAIL_USE_TLS'))
EMAIL_PORT = os.getenv('EMAIL_PORT')
EMAIL_HOST_USER = os.getenv('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD = os.getenv('EMAIL_HOST_PASSWORD')

For your reference, you can see the settings.py here.

Create `.env` file

Create a .env file and assign the secrets. ```

EMAIL_USE_TLS=True
EMAIL_PORT={EMAIL_PORT}
EMAIL_HOST_USER={EMAIL_HOST_USER}
EMAIL_HOST_PASSWORD={EMAIL_HOST_PASSWORD}
CELERY_BROKER_URL="pyamqp://localhost:5672"
SECRET_KEY={SECRET_KEY}
DATABASE_URL=postgres://postgres:password@localhost:5432/postgres
POSTGRES_PASSWORD=password
APP_DOMAIN=*
DEBUG=True


## Celery

We need to set up Celery with some config options. Create a new file called `celery.py` inside `newseletter_site` directory :

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'newsletter_site.settings')

app = Celery('newsletter_site')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()


## Design and Implement Models & Configure Admin
This is the schema we are trying to build. The schema is implemented [here](https://github.com/Nancy-Chauhan/newsletter/blob/main/newsletter/models.py). Create a `newsletter/models.py` with the same content.

![Schema design](https://miro.medium.com/max/1400/1*1SYrd7LgJ-nonMft0TBJ4g.png)

We need a UI to manage the newsletter. We will be using Django Admin for this purpose. Create a `newsletter/admin.py` with the contents of this file.

Register URL for admin in `newsletter_site/urls.py`:

urlpatterns = [
path('admin/', admin.site.urls),
]


# Run the app

Run docker-compose to start the dependencies:
`docker-compose up`

Generate migrations for our models:
`python manage.py makemigrations`

To apply generated migrations to database run:
`python manage.py migrate `

To create a user for login run the following command and provide your details:
`python manage.py createsuperuser`

Run the following command to run the app and open http://127.0.0.1:8000/admin to open Django Admin :
`python manage.py runserver`

![Django Admin](https://miro.medium.com/max/3836/1*H4Vbp888k0K-4S5cImov6Q.png)

Run celery:
`celery -A newsletter_site worker --loglevel=INFO`
Add a newsletter and a subscriber and subscribe them to it. Create an issue and send it. If everything is fine you will see an issue arriving in your email.

#How does it work?

When we click send the following action gets executed:

def send(modeladmin, request, queryset):
for issue in queryset:
tasks.send_issue.delay(issue.id)

send.short_description = "send"


This code is responsible to queue up a new task to send an issue using celery. It publishes the task to RabbitMQ.

@shared_task()
def send_issue(issue_id):
issue = Issue.objects.get(pk=issue_id)
for subscription in Subscription.objects.filter(newsletter=issue.newsletter):
send_email.delay(subscription.subscriber.email, issue.title, issue.content)

@shared_task()
def send_email(email, title, content):
send_mail(
title,
content,
'newsletters@nancychauhan.in',
[email],
fail_silently=False,
)


The Celery worker uses these tasks. When the producer publishes a task the worker runs the corresponding task.
When we publish the send_issue task we determine the subscriber for the newsletter and publish sub-tasks to send the actual email. This strategy is called fan-out. Fan out is useful as it allows us to retry sending emails to a single user in case of a failure.

# Conclusion

In this post, we saw how to use RabbitMQ as a message queue with Celery and Django to send bulk emails. This is a good fit where message queues are appropriate. Use message queue if the request is indeterministic or the process is long-running and resource-intensive.

You can find the finished project [here](https://github.com/Nancy-Chauhan/newsletter):

Originally published [here](https://medium.com/@_nancychauhan/introduction-to-message-queue-build-a-newsletter-app-using-django-celery-and-rabbitmq-in-30-min-6d484162391d)

Thank you for reading! Share your feedback in the comment box.

Faster Builds with Docker Caching

Nancy Chauhan — Wed, 20 May 2020 06:55:48 +0000

Recently was working around speeding up CI Build, which took time around 50 min. This post is about speeding up builds with Docker caching and Buildkit.

It is a little concept, but it is significant to know as this helped me to reduce timings from 50 min to 15 min.

Docker Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache.

A better way would be to have multiple COPY instructions that install the packages and dependencies first and then copies your code.

To avoid invalidating the cache:

Start your Dockerfile with commands that are less likely to change
Place commands that are more likely to change (like COPY . .) as late as possible
Add only the necessary files (use a .dockerignore file)

Example :

The following Dockerfile is for a simple node js project which works great
but here cache is not used :

FROM mhart/alpine-node

WORKDIR /src
# Copy your code in the docker image
COPY . /src
# Install your project dependencies
RUN npm install
# Expose the port 3000
EXPOSE 3000
# Set the default command to run when a container starts
CMD ["npm", "start"]

Optimized Dockerfile :

Here step 7 is the only step where an image is rebuilt based on
previous steps when we make a change to our source code.

FROM mhart/alpine-node:5.6.0
WORKDIR /src
# Expose the port 3000
EXPOSE 3000
# Set the default command to run when a container starts
CMD ["npm", "start"]
# Install app dependencies
COPY package.json /src
RUN npm install
# Copy your code in the docker image
COPY . /src

BuildKit

If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.

Example:

$ docker pull mjhea0/docker-ci-cache:latest

$ docker build --tag mjhea0/docker-ci-cache:latest .

With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.

To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1. Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Example:

export DOCKER_BUILDKIT=1

# Build and cache image
$ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE=1 .

# Build an image from remote cache
$ docker build --cache-from mjhea0/docker-ci-cache:latest .

Originally Posted here!

DNS Resolution

Nancy Chauhan — Thu, 07 May 2020 04:18:27 +0000

Recently was working around DNS and thought to put it here!

Computers work with numbers. Computers talk to another computer using a numeric address called IP address. Though structured and thus great for computers, it is tough for humans to remember.

DNS acts as the phonebook of the internet 🌐. It converts a web address such as "example.com" to an IP address, which computers use to connect. As a result, we don't have to remember complicated IP addresses 🤩.

We are trying to open example.com on a browser. A Typical DNS lookup goes like this:

The browser first looks up "example.com" in its DNS cache. If it is present, the browser uses the cached IP address and connects to "example.com". If not, then the browser goes to the next step.
Browser issues a gethostbyname (3) and passes the responsibility of name resolution to the operating system (OS). The OS now becomes the resolver.
OS looks for the domain name in the system DNS cache. If found then it returns the IP address to the browser else the OS goes to the next step.
The OS looks into \etc\hosts, known as the hosts file. The hosts file is a method of maintaining hostname to IP address mapping from the ARPANET days. If an entry exists, the OS returns the IP address else it goes to the next step.
The OS tries to connect to your configured DNS Servers and sends a DNS query for "example.com". You can manually set your DNS Servers, or your connected networks can configure it for you. The DNS server now becomes the resolver and has to return a response to the OS of the machine that has sent the DNS query.
The DNS server (resolver) looks into its DNS cache for the hostname. If it finds an entry, it returns the same to the calling machine. Else it goes to the next step.

The DNS server tries to connect to root nameserver (.) You can do dig . to find root nameserver your DNS server is trying to connect. At present, there are 13 root nameservers named with the letters "a" to "m" — a.root-servers.net.

➜ dig -t NS .

; <<>> DiG 9.10.6 <<>> -t NS .
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45206
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.              IN  NS

;; ANSWER SECTION:
.           48  IN  NS  a.root-servers.net.
.           48  IN  NS  d.root-servers.net.
.           48  IN  NS  k.root-servers.net.
.           48  IN  NS  g.root-servers.net.
.           48  IN  NS  j.root-servers.net.
.           48  IN  NS  c.root-servers.net.
.           48  IN  NS  b.root-servers.net.
.           48  IN  NS  m.root-servers.net.
.           48  IN  NS  f.root-servers.net.
.           48  IN  NS  h.root-servers.net.
.           48  IN  NS  l.root-servers.net.
.           48  IN  NS  e.root-servers.net.
.           48  IN  NS  i.root-servers.net.

;; Query time: 80 msec
;; SERVER: 10.254.254.210#53(10.254.254.210)
;; WHEN: Wed May 06 22:51:43 IST 2020
;; MSG SIZE  rcvd: 239

Now the DNS server requests on of the above root nameserver for the TLD
level root nameserver for TLD for ".com".

➜ dig @d.root-servers.net. -t NS com.

; <<>> DiG 9.10.6 <<>> @d.root-servers.net. -t NS com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 106
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1450
;; QUESTION SECTION:
;com.               IN  NS

;; AUTHORITY SECTION:
com.            172800  IN  NS  a.gtld-servers.net.
com.            172800  IN  NS  b.gtld-servers.net.
com.            172800  IN  NS  c.gtld-servers.net.
com.            172800  IN  NS  d.gtld-servers.net.
com.            172800  IN  NS  e.gtld-servers.net.
com.            172800  IN  NS  f.gtld-servers.net.
com.            172800  IN  NS  g.gtld-servers.net.
com.            172800  IN  NS  h.gtld-servers.net.
com.            172800  IN  NS  i.gtld-servers.net.
com.            172800  IN  NS  j.gtld-servers.net.
com.            172800  IN  NS  k.gtld-servers.net.
com.            172800  IN  NS  l.gtld-servers.net.
com.            172800  IN  NS  m.gtld-servers.net.

;; ADDITIONAL SECTION:
a.gtld-servers.net. 172800  IN  A   192.5.6.30
b.gtld-servers.net. 172800  IN  A   192.33.14.30
c.gtld-servers.net. 172800  IN  A   192.26.92.30
d.gtld-servers.net. 172800  IN  A   192.31.80.30
e.gtld-servers.net. 172800  IN  A   192.12.94.30
f.gtld-servers.net. 172800  IN  A   192.35.51.30
g.gtld-servers.net. 172800  IN  A   192.42.93.30
h.gtld-servers.net. 172800  IN  A   192.54.112.30
i.gtld-servers.net. 172800  IN  A   192.43.172.30
j.gtld-servers.net. 172800  IN  A   192.48.79.30
k.gtld-servers.net. 172800  IN  A   192.52.178.30
l.gtld-servers.net. 172800  IN  A   192.41.162.30
m.gtld-servers.net. 172800  IN  A   192.55.83.30
a.gtld-servers.net. 172800  IN  AAAA    2001:503:a83e::2:30
b.gtld-servers.net. 172800  IN  AAAA    2001:503:231d::2:30
c.gtld-servers.net. 172800  IN  AAAA    2001:503:83eb::30
d.gtld-servers.net. 172800  IN  AAAA    2001:500:856e::30
e.gtld-servers.net. 172800  IN  AAAA    2001:502:1ca1::30
f.gtld-servers.net. 172800  IN  AAAA    2001:503:d414::30
g.gtld-servers.net. 172800  IN  AAAA    2001:503:eea3::30
h.gtld-servers.net. 172800  IN  AAAA    2001:502:8cc::30
i.gtld-servers.net. 172800  IN  AAAA    2001:503:39c1::30
j.gtld-servers.net. 172800  IN  AAAA    2001:502:7094::30
k.gtld-servers.net. 172800  IN  AAAA    2001:503:d2d::30
l.gtld-servers.net. 172800  IN  AAAA    2001:500:d937::30
m.gtld-servers.net. 172800  IN  AAAA    2001:501:b1f9::30

;; Query time: 259 msec
;; SERVER: 199.7.91.13#53(199.7.91.13)
;; WHEN: Wed May 06 22:54:16 IST 2020
;; MSG SIZE  rcvd: 828

DNS server then requests one of the above root nameservers for the authoritative nameserver for the domain example.com. This set of nameservers host the addresses of the domain as well as any subdomains it may have.

➜ dig @a.gtld-servers.net. -t NS example.com

; <<>> DiG 9.10.6 <<>> @a.gtld-servers.net. -t NS example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1127
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.           IN  NS

;; AUTHORITY SECTION:
example.com.        172800  IN  NS  a.iana-servers.net.
example.com.        172800  IN  NS  b.iana-servers.net.

;; Query time: 66 msec
;; SERVER: 192.5.6.30#53(192.5.6.30)
;; WHEN: Wed May 06 22:55:10 IST 2020
;; MSG SIZE  rcvd: 88

The DNS server requests the authoritative nameservers for IP addresses of the domain and returns the result to the system that sent it the DNS query.

➜ dig @a.iana-servers.net. -t A example.com

; <<>> DiG 9.10.6 <<>> @a.iana-servers.net. -t A example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5682
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        86400   IN  A   93.184.216.34

;; Query time: 281 msec
;; SERVER: 199.43.135.53#53(199.43.135.53)
;; WHEN: Wed May 06 22:58:40 IST 2020
;; MSG SIZE  rcvd: 56

Using the IP address 93.184.216.34, the web browser connects to the host.

Every stage maintains a cache for some number of seconds based on the TTL that every query returns. In the following DNS query result, the TTL is 86400 seconds

example.com.        86400   IN  A   93.184.216.34

A resolver can thus cache the contents of the query for 86400 seconds. This caching helps to speed up the process and reduces the load on DNS servers.

Originally posted at https://todayilearnt.xyz/posts/nancy/dns_resolution/

Monitoring Java Web Apps using Prometheus and Grafana

Nancy Chauhan — Tue, 21 Apr 2020 03:13:15 +0000

Recently, I have been exploring ways to make systems as monitorable as possible, which means minimizing the number of unknown-unknowns!

The four pillars of the Observability Engineering team’s charter are :

(Source: Twitter’s tech blog )

Monitoring
Alerting/visualization
Distributed systems tracing infrastructure
Log aggregation/analytics

Monitoring an application’s health and metrics makes it possible to manage it in a better way and notice unoptimized behavior. I will be giving you a walkthrough on monitoring and visualizing metrics of a Java application in this blog.
We will be using the following tools to achieve this:

Micrometer: Exposes the metrics from our application
Prometheus: Stores our metric data
Grafana: Visualizes our data in graphs

It’s straightforward to implement all of them with just a few lines of code. We will be performing this on my project, which is a simple KV (key-value) store web service that has been developed in Java. You can find the code here.
To make things even easier, we’ll be using Docker to run Prometheus and Grafana. Later we will provision Grafana Data Sources and Dashboards from the configuration. Let’s get started!

Configuring Java application with Micrometer

Adding Prometheus support to any Java application becomes a lot easier with Micrometer. It provides a clean facade to many monitoring platforms, including Prometheus.

Installing

We need to add the following dependency :

In Gradle:

compile 'io.micrometer:micrometer-registry-prometheus:latest.release'

Configuring

In Micrometer, we need a “Meter,” which is the interface for collecting a set of measurements (which we individually call metrics) about our application.

final PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);

Micrometer packs several Meter primitives including: Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounter, FunctionTimerand and TimeGauge. We will be modifying our code to report various metrics using the above set of meters. You can read more about them from the official documentation.
For instance, the following code defines a counter that can be used to count some events over a short window. Here this counter is used to count the number of getAll requests.

Counter getAllRequestCounter= prometheusRegistry.counter("http.request",
        "uri", "/keyvalue",
        "operation", "getAll");

Secondly, and most importantly, Prometheus expects to scrape app instances for metrics. In addition to creating a Prometheus registry, we will also need to expose an HTTP endpoint to Prometheus’ scraper.
get("/metrics", (request, response) -> prometheusRegistry.scrape());

Note: The above configuration is for Spark Framework in Java. In a Spring environment, a Prometheus actuator endpoint is autoconfigured in the presence of the Spring Boot Actuator.
To record an event, we call the increment method on the counter that we just created :

get("/keyvalue", (request, response) -> {
    getAllRequestCounter.increment();
    response.type("application/json");return new Gson().toJson(keyValueStoreService.getAll());
});

After we are done with configuring our code, let’s proceed towards setting up graphs.

Setting up Prometheus and Grafana

Prometheus is a time-series database that stores our metric data by pulling it using a built-in data scraper periodically over HTTP. It also has a simple user interface where we can query and visualize the collected metrics.

While Prometheus provides some basic visualization, Grafana offers a rich UI where you can build custom graphs quickly and create a dashboard out of many graphs in no time. Grafana can pull data from various data sources like Prometheus, Elasticsearch, InfluxDB, etc.
Here we build docker-compose.yml to install Prometheus and Grafana from Docker:

Let’s now configure Prometheus by setting the scrape interval, the targets, and define the endpoints. To do that, we’ll be using the prometheus.yml file:

You can read more about Prometheus configurations, at the official documentation.

Bringing up everything

Run docker-compose up to start the app, Prometheus and Grafana. Open http://localhost:9090 for Prometheus and http://localhost:3000 for Grafana.

To check if Prometheus is pulling metrics from the web app, open “Status”> “Targets.”

Setting up Grafana

While docker-compose started Grafana, it doesn’t do much yet.

We need to configure Grafana to connect with Prometheus by manually setting up the data source.

Then create a dashboard, add a “Query” and select your Prometheus data source, which you just configured.

In the “Metrics” field, add a PromQL query such as:

http_request_total{application="KeyValue",instance="app:4567",job="key-value",operation="getAll",uri="/keyvalue"}

You can read more about PromQL in the official documentation.

Configure Grafana Provisioning

Instead of manually creating dashboards and data sources, we can utilize Grafana provisioning. You can read more about Grafana Provisioning in the official documentation.
Add two new volumes to docker-compose to read our provisioning configs and dashboards. You can see the completed docker-compose file here.
volumes:

  - "./grafana/provisioning:/etc/grafana/provisioning"
  - "./grafana/dashboards:/var/lib/grafana/dashboards"

Create a new file, datasource.yml under provisioning/datasources

Create another file, dashboard.yml under provisioning/dashboards

Export the dashboard you created earlier and put it under folder. You can find the dashboard here as well.

Bring docker-compose up, and you should be able to see Grafana with your dashboard and data source set.

Here we saw how to configure a Java application to monitor it with Prometheus. You can explore JMX exporter and Micrometer JVM extras to report several metrics about the JVM and many other java libraries.
Let me know about your experiences in the comments!

Today I Learnt: Git Submodules

Nancy Chauhan — Tue, 21 Apr 2020 01:44:03 +0000

It often happens that while working on one project, you need to use another project from within it. For that, we can make use of git submodules.

Submodules are a reference to another repository withing a parent repository. You can make a submodule point to any arbitrary revision of the other repo.

Today I learned how messy git submodules could be! 😾

Using submodules needs many disciplines. Every developer using repo always need to update submodules using git submodule update --init --recursive. Failing to update can lead to very hard to debug and silent errors. And that's not all if you forget to update the submodules, then git commits the old version of the submodule. This effectively undoes the submodule update done by another developer. And git log doesn't even show such an update 😓.

Submodules need references to the URL that git should fetch. We all use SSH authentication in git. To make submodules experience frictionless, we point submodules using another SSH URL. When you start consuming this repo in CI such as Jenkins, which uses https to clone, everything breaks 😢 , so either you force your CI to clone ssh and set up its authentication or force your ci to pull everything in https, which is not possible in all scenarios. Now we are left with a decision to choose between frictionless CI or frictionless development experience. 😕

Also, there are many stories about submodules getting lost during merge conflicts, submodules not supported by some UI tools, etc. which makes them pure evil.

Some people say submodules are "Sobmodules." 🤣

Originally Posted at https://todayilearnt.xyz/posts/nancy/git_submodules/

Docker on mac vs Linux

Nancy Chauhan — Tue, 21 Apr 2020 01:41:10 +0000

🐳Docker is different on Mac and Linux systems. Docker directly leverages the kernel of the host system on Linux. On the other hand, Mac does not provide a Linux kernel, so Docker runs on a small Linux VM running on a mac. Due to this, there are many differences.

Cannot access container IPs directly

On Linux let's try to inspect a running docker image :

ubuntu@primary:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS               NAMES
a230855dd7d2        httpd               "httpd-foreground"   17 seconds ago      Up 16 seconds       80/tcp              determined_saha
ubuntu@primary:~$ sudo docker inspect determined_saha

Let's curl the IP :

                   "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]
ubuntu@primary:~$ curl 172.17.0.2
<html><body><h1>It works!</h1></body></html>

However on mac when hitting the IP address of the container we get :

➜ curl 172.17.0.3
curl: (7) Failed to connect to 172.17.0.3 port 80: Operation timed out

Curl times out because the container is running inside the VM and not sharing the network with the mac.

DEV Community: Nancy Chauhan

Load Balancing

Why do we need load balancing?

Server-side Load Balancing:

Advantages

Client-Side Load Balancing:

Advantages

Benefits of Loadbalancing

Enforcing Coding Best Practices using CI

Unit Testing

Code coverage

Static code analysis

Linting

Linters for Static Analysis

Linters focused on Security

Docker lint check

Secrets checks

Dependency Checks

All-in-one tools

Conclusion

Designing Idempotent APIs

Why Idempotency is critical in backend applications?

Failures and Retries

Solution

Idempotency in HTTP

Conclusion

Top 10 Productive Hacks for Software Developers

Automation

Use Fish or Zsh with Multiplexing terminals

Zsh & Fish

Use and know your IDE very well

Clipboard manager

Gmail labels, and filters to keep annoying mails out of life

Nap and Breaks

A place to put all tasks, Github issues, Jira tickets, private stuff

Nancy-Chauhan / Today-I-Learnt

TIL: 26-02-2021

TIL: 07-03-2021

TIL: 08-03-2021

TIL: 10-03-2021

Touch typing

Linux Commands and Shortcuts

Focus

Building a Prometheus Exporter

How Prometheus Works?

Prometheus Metrics

Using Prometheus

Prometheus Exporters

grofers / jenkins-jobmon

Prometheus exporter to monitor Jenkins jobs

Jenkins Jobmon

Usage

Configuration

Let’s build a generic HTTP server metrics exporter!

Nancy-Chauhan / httpd_exporter

Prometheus exporter for monitoring apache

httpd_exporter

Requirements

Usage

Grafana Dashboard

Introduction to Message Queue: Build a newsletter app using Django, Celery, and RabbitMQ in 30 min

Workers & Message Queues

Build Newsletter App

Requirements

Setup Django

Installation

Setup Postgres and RabbitMQ

Configuring settings.py

Create .env file

Faster Builds with Docker Caching

Docker Caching

BuildKit

DNS Resolution

Monitoring Java Web Apps using Prometheus and Grafana

Configuring Java application with Micrometer

Installing

Configuring

Setting up Prometheus and Grafana

Bringing up everything

Setting up Grafana

Create `.env` file