DEV Community: Deepjyoti Barman

Dynamic Path Matching with Go and mux

Deepjyoti Barman — Sun, 16 Jul 2023 13:29:57 +0000

Go is by far one of the most interesting languages that I have worked with and I have enjoyed working with it almost everytime (except when functions do not support default values). Recently, I came across this problem where I had a use-case to support user defined routes for an API. Problem was supporting these user defined URL's in real time and actually triggerring a handler based on them.

Naive Approach that was considered

I considered this naive approach where I would restart the router (using Mux) everytime an user defined route is added/deleted/updated and the router will know whether the incoming request is for an user defined route or system.

This approach is simple enough but is a very bad practice. Imagine thousands of users updating their routes every now and then and the whole router restarts and the API (as a whole) goes down for a few seconds. This can be a major problem for a production API that is customer facing.

Thus, there was need for figuring out an alternate approach to this problem that would be efficient as well.

How it's Solved

Just a heads up, before I dive into details, the problem was solved by using mux and it's useful functions (thanks to the developers for considering a scenario like this while developing).

In order to understand the solution to the problem, we will have to understand how mux handles an incoming request. The flow is important here in order to solve this problem in a very efficient way.

In mux, every route has two properties (or functions) that can be defined against it. One of them is a matcher which can be defined optionally but not necessarily required. The other is a handler which is the actual function that will be trigerred if the route is matched.

Matchers

mux uses matchers to determine whether a route should match. matchers are just mere functions which return a boolean value. This value indicates whether the route is matched or not matched.

matcher returns true: route is matched and handler should be called.
matcher returns false: route is not matched

NOTE: One thing to know here is that mux goes through each route that is defined in the router until a matcher returns true or the end of routes is reached. When end of routes is reached, a 404 is returned with an error indicating the page is not found.

Matchers are a boon in this particular scenario as this makes solving dynamic user defined routes really simple.

Here's an example matcher that matches the incoming route if it is /hello:



func getHelloMatcher() mux.MatcherFunc {
    return func(req *http.Request, rm *mux.RouteMatch) bool {
        return req.URL.Path == "/hello"
    }
}

Above is a very simple example of how a custom matcher can be defined but in order to match against a dynamic route, it is better to use mux's own functions. mux internally uses a Match function that checks if a route matches the incoming requests path and accordingly returns true or false. It is ideal for us to use this function as they use some complex regex to match the path.

Following code shows how to use mux's internal Match function:



func getHelloMatcher() mux.MatcherFunc {
    return func(req *http.Request, rm *mux.RouteMatch) bool {
        // In order to use a internal `mux` function we will need to
        // define a dummy router that can provide the function.
        copyMuxRouter := mux.NewRouter().StrictSlash(true)

        return copyMuxRouter.Methods(http.MethodGet, http.MethodPost).
                Name("dynamic router").
                Path("/hello").
                Match(req, rm)
    }
}

As explained, above code uses an internal function to match the incoming route against a route /hello. This part of the code can be made dynamic based on user defined route that can be fetched accordigly.

Handlers

After a matcher returns true, the route's handler is called. In mux, all routes will absolutely need a handler defined against it or else mux will not start the router and throw an error at startup.

Pretty much anyone who has used mux is aware of how to define a handler so I will not go over the steps on that. Here's a full example from mux docs on how to define a handler for a route.

Enhancements to the above

The problem that I initially mentioned can be solved by using matchers as I have explained. However, there are certain places where the code can be made efficient. The matcher function that mux supports, uses a RouteMatch type. This is a custom type inside of which there's a Vars key which maps to a map type. This Vars can be used to pass details from the matcher to the handler in order to reduce some redundant steps.

As an example, say the user defined route is stored in a database and the user can also define different methods for the route. These details might come handy in the handler. In our scenario, these details are definitely required in the matcher phase of the code where a database call can be made to get the details for a route.

In the handler phase of the flow, instead of making another database call to get the details of the matched route, those details can simply be passed from the matcher to the handler by using the Vars of the RouteMatch object.

I am not sure if RouteMatch.Vars was intentionally designed to do that but it is definitely a good hack (if you will) to make the code efficient and relatively faster (database network calls are expensive).

Following example shows how some details are passed from the matcher to the handler:



func getHelloMatcher() mux.MatcherFunc {
    return func(req *http.Request, rm *mux.RouteMatch) bool {
        // In order to use a internal `mux` function we will need to
        // define a dummy router that can provide the function.
        copyMuxRouter := mux.NewRouter().StrictSlash(true)

        methods := [http.MethodGet, http.MethodPost]
        isMatched := copyMuxRouter.Methods(methods...).
                Name("dynamic router").
                Path("/hello").
                Match(req, rm)

        if isMatched {
            rm.Vars["MATCHED_ROUTE_METHODS"] = strings.Join(methods, "-")
        }

        return isMatched
    }
}

Following is how to use the MATCHED_ROUTE_METHODS in the handler:



func getHelloHandler() mux.HandlerFunc {
    return func(rw http.ResponseWriter, req *http.Request) {
        // Read the vars
        vars := mux.Vars(req)

        // Get the methods.
        methods := vars["MATCHED_ROUTE_METHODS"]
        fmt.Println("Matched methods are: ", methods)
    }
}

GoLang is becoming one of my favorite languages to work on and I am very close to even replacing Python with GoLang in my next project. I think everyone should give it a try to find out the beauty of it.

gorrila/mux team has been doing a great job building this awesome router. Go give it a star at gorilla/mux.

This post was originally posted at my blog at https://blog.deepjyoti30.dev

Setup your own blog on top of GitHub

Deepjyoti Barman — Sat, 06 Nov 2021 06:07:38 +0000

Have you ever thought of creating your own blog in your own .dev domain that you could just share with anyone and forget?

Well, it has never been easier to create a blog. I myself have a separate blog on my domain and every now and then I end up redesigning the whole thing. However, the tedious part about maintaining a blog is every time you need to keep on moving your posts from one place to another.

This is why I came up with Emanates. Emanates is a tool that lets you build a blog on top of GitHub in just a few minutes.

How does it work?

The idea is simple. There are two key things to know about Emanates:

Every GitHub Issue is a blog post
Every time an issue is created/updated/deleted a GitHub Action is run that builds the latest static blog and deploys it.

Why Emanates?

Now, why should you go with Emanates instead of building your own blog?

Well, Emanates also comes with some of the latest features that people want in a blog:

Comment support
Built in SEO
Support for automatic related posts (Emanates does that automatically, you just need to use the proper labels)
Metadata supported
Cover image supported
Dark Mode (obviously)
Everything is stored on GitHub, no hassles of databases and such.
It is fast (it's a static site)

Show me how it is

If you want to get a taste of how it looks like when it is live, check out the demo page:

https://emanates-demo.netlify.app/

Here are some images:

How should I start

Get started with Emanates by checking the GitHub repo here: https://github.com/emanates/emanates-web

Emanates: Build a blog on top of GitHub Issues and Actions

Deepjyoti Barman — Mon, 23 Aug 2021 13:01:21 +0000

I am happy to announce the launch of Emanates. This idea came in my mind way back when GitHub launched GitHub Actions.

The idea of Emanates is every GitHub Issue is a blog post and when an issue is created, a workflow is triggered (GitHub Action).

This workflow builds the blog and deploys it to your desired location.

Emanates boasts of some nice features like:

Improved SEO
Support for related posts (based on tags added through GitHub Labels)
Meta tag support along with cover image and other things.
Markdown support
Supports comments (through utterances)
Blazingly fast (since it builds a static site)
Dark Mode (obviously!)
Everything on GitHub (no need to trust me)
Open Source

You can check Emanates GitHub here

Demo site for anyone interested

I'm attaching a few images to show how Emanates looks.

A handy deployment script for your pipeline

Deepjyoti Barman — Fri, 16 Jul 2021 12:15:25 +0000

Just after GitHub Actions was launched, I was pretty excited to try it out. So I did what any other developer would have done, I wrote an workflow file to deploy my API's automatically to my server.

After a few tries and catch blocks, ;-), I was able to get it to work properly. Back then, my idea of the pipeline was, the script would ssh into my server, build the dockerfile locally and then deploy it.

However, after talked with a few people, turns out that's not the correct way to use it. I started reading a few articles and I was indeed wrong in thinking the building process is supposed to be done locally on the host machine.

So how to do it then?

If you want to get a proper idea of how to setup a pipeline, read my other article on it

I will just go through the workflow in brief as of now.

Since the pipeline might deploy the docker image in various containers (which obviously adds the possibility of multiple platforms, also in the case of releases), the building process should be rather done in the pipeline.

After the building is done, the image should be pushed to a container hub , yep, you guessed it, Docker Hub or in my case GHCR (since it's free). After the image is pushed, the workflow should deploy the image in various platforms.

I personally do it by using the ssh-action package that allows my workflow to ssh into my server(s), pull the latest image from the hub and deploy it.

Here's where my article comes in place, how do you deploy locally?

Obviously a simple docker pull and docker run won't cut it right? There are ports to be taken care of, volumes mapping, environment files.

I had the same question and I came across an article (that I am unable to find right now), that had a pretty nice script.

Deployment Script

The script I am sharing here is not mine. It was created by someone else and I merely just edited it a bit to fit my requirements. I am unable to find the article of the original author of the script.

#!/bin/bash
########################################
# Put this on a Server
# run chmod +x deploy_app.sh to make the script executable
# 
# Execute this script:  ./deploy_app.sh docker-username/docker-appname:$TAG
# Replace the $TAG with the actual Build Tag you want to deploy
#
########################################

set -e

DOCKER_IMAGE=$1
CONTAINER_NAME="container-name"

# Check for arguments
if [[ $# -lt 1 ]] ; then
    echo '[ERROR] You must supply a Docker Image to pull'
    exit 1
fi

echo "Deploying $CONTAINER_NAME to Docker Container"

#Check for running container & stop it before starting a new one
if [ $(docker inspect -f '{{.State.Running}}' $CONTAINER_NAME) = "true" ]; then
    docker stop $CONTAINER_NAME
fi

# Pull the latest image
echo "Pulling latest image for $DOCKER_IMAGE"

docker pull $DOCKER_IMAGE

echo "Starting $CONTAINER_NAME using Docker Image name: $DOCKER_IMAGE"

docker run -d --rm=true -p 80:5000 --env-file ~/Scripts/.env --name $CONTAINER_NAME $DOCKER_IMAGE

# Show the running processes
docker ps

The above script is basically pulling the latest image and deploying it according to the passed flags like --env-file. However, this also checks if the container is already running. If so, the container is first removed and after the new container is run.

You can directly copy the script to your server, just do the following changes:

Change the CONTAINER_NAME value.
Make the script executable using chmod +x <script_path>

Don't forget to pass the name of the image while running the script, eg: sh script.sh ghcr.io/deepjyoti30/ytmdl-web-v2:latest

It is important to pass the latest tag or the exact tag because otherwise a cached image of the container is used. This means the image is not updated at all. So make sure the tag is passed.

Example workflow with the script

Once you have the script setup nice and cozy on your server, you can setup a workflow like following to ssh into the server and run the script. We will use ssh-action as mentioned above to ssh into the server. Once done, that action will just run the script.

name: Build app and deploy docker

on:
  push:
    branches:
      - "production"

jobs:
  push-to-docker:
    # Write actions here to build and push the image
    # to a hub like Docker Hub or Github Container Registry
    # NOTE: Don't leave these lines as such, the workflow will fail.

  deploy:
    needs: push-to-docker
    runs-on: ubuntu-latest
    steps:
      # SSH into the server
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.SERVER_IP }} # Pass server IP
          username: ${{ secrets.SERVER_USERNAME }} # Pass server username
          key: ${{ secrets.KEY }} # Pass server ssh private key
          script: /bin/bash Scripts/deploy_ytmdl_web.sh ghcr.io/deepjyoti30/ytmdl-web:latest

You can see the above workflow in action in my ytmdl-web repo.

If you happen to come accross the article that had the above script originally, feel free to comment. I will update the post with credits (and links) to that!

This article was originally posted on my personal blog

Github, Python and Docker: The Sweet Trio

Deepjyoti Barman — Wed, 09 Jun 2021 16:18:05 +0000

I have been meaning to write about this for a while. Since most of my work revolves around building API's and deploying them, it is pretty important for me (and people like me) to automate as many things as possible in order to get the best out of my time and available resources.

My current tech stack revolves around the following:

Python (that wasn't hard to guess, huhh!?)
FastAPI (For the API)
Celery (For workers to do tasks in the background)
Docker (Why not!?)
GitHub (Version control)
DigitalOcean (Server)

How do the above fit in the bigger picture?

I will try to keep things as simple as possible.

I use Python as my primary language to build API's. FastAPI is a Python framework that makes building production API's way simpler. Celery is a Python tool/library that queues tasks and does them in the background.

Using the above three makes my API fast and production ready.

I use Docker for containerizing my API. The code for the API and Docker is stored on GitHub. Finally, the docker container is deployed on a VPS on DigitalOcean so that my API can be interected with by the people and most importantly, my apps!

When all of them are used together, I have a fast, production ready API that is accessible by my apps.

Typical Scenario

Let's consider two locations:

My PC (where I write code)
The server (where the API is deployed)

In order for my code to go from the first destination (my PC) to the second destination (my server), it has to go through a few steps.

Now, let's say we don't know something called pipeline exists. A typical way for a code change to go from the start point to the destination would be the following steps:

Code pushed to GitHub.
I ssh into my server.
Pull the latest code changes.
Build the latest docker image.
Deploy.

Here's a simple question, out of all the above steps, which ones require me to do them manually or even a part of them manually?

I need to push the code myself, I need to ssh into my server myself. I need to pull the latest code changes myself. I need to build the docker image myself. I need to deploy the docker image myself.

So, as it turns out, all the steps require me to give them some manual input.

Remove redundant steps

Now, out of the above, lets try to filter out the redundant steps. Redundant steps are those that can be executed by my computer with minimal interaction (can be passed through env variables) for me.

Redundant steps are those that can be executed by my computer with minimal interaction (can be passed through env variables) for me.

We can make ssh into server automated since we just need to pass an ssh key.
We can automate pulling the latest code changes.
We can automate the docker image build and push process since it's the same step being repeated.
We can automate the deployment of the latest docker image since that doesn't require any input.

What is GitHub Actions (or anything similar)?

GitHub Actions is a Continuous Integration service. It is very much similar to Travis CI and Jenkins but it is built by GitHub so that obviously comes with perks if we're using GitHub.

We pass GitHub Actions a few steps to follow (remember those redundant steps?). It will gracefully follow all the steps and the end result should be our code being deployed.

Basically, we are defining a path that our code will follow in order to reach their destination (the server) but without any human interaction.

One thing we need to keep in mind is that GitHub Actions will actually run the steps we ask it to run in a server somewhere but not in our server neither in our computer. This is why we say pipelines build on the fly.

We can easily specify which OS GitHub Action uses in order to run the steps. (More on that below).

How do we tell GitHub Actions to do something?

In order to let GitHub know that we want it to follow a few steps and do an action, we can do that by passing a yaml file. This file needs to follow the GitHub workflow syntax.

We can define jobs in the file. Each job can have a few steps that are supposed to be followed. Jobs can be defined in the following way:

jobs:
  # Name of the job
  push-to-docker:
    runs-on: ubuntu-latest # Use latest Ubuntu for the steps
    steps:
      # Do something
      - name: Do something

Removing redundant steps

Now that we know that we can use GitHub Actions to automate our steps, let's see how we can make sure that GitHub Actions follows each step.

NOTE: From now on, we need to keep in mind that the steps are being written for GitHub Actions.

Automate pulling the latest code changes

There is a neat action that is provided by GitHub itself. GitHub Actions Checkout checks out the latest code from the repository we're running the action on.

This means, we can just use this package to pull our latest changes. Now in whatever server GitHub Action is running, we will basically clone the latest changes from our repo to that server.

Following is the code to tell Actions to checkout the latest code from the current repo.

jobs:
  example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2

The above code tells action to start a new job named example-job on the latest version of Ubuntu and run the following steps.

The first step pulls the latest changes of the current repo to that OS.

Automate building the docker image and pushing it

Now, since we have the latest code changes avaiable, we can easily build a docker image and push it to Docker Hub or GitHub Container Registry.

In order to do that, we can use the package build-push-action.

However, docker or GitHub doesn't just let anyone push an image to the container registry. This means, we need to login to docker in order to push the latest built image. We can do that by using the login-action package from docker.

Adding the above steps to the yaml file will make it look like following:

jobs:
   example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2
      # Login to docker
      # This can be used for both Docker Hub and
      # GitHub container registry.
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          # Remove the following line if you want to
          # login to docker hub.
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          # secrets are GitHub actions that can be added
          # from settings of the repo.
          password: ${{ secrets.CR_PAT }}
      # Build the docker image and push it.
      - name: Build image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          # Remove the `gchr.io/` if you're pushing to
          # Docker Hub
          tags: ghcr.io/owner/packageName:latest

With the above code, we have already removed three steps that we were doing manually. Now we can just ssh into our server, pull the latest image and deploy it. But wait a minute, shouldn't we automate that too?

Automate SSH into server and deploy latest image

Now we have just one more step left. We want our GitHub Action to deploy the latest image in our server as well. Since our action is already running on a machine, can't we just pass a few steps to make the machine ssh into the actual server (the destination) and then deploy the changes?

Yes we can! We will use the ssh action package for that. This package will do what we have been doing manually. SSH into the server and deploy.

Let's add a new job that will wait for our first job to complete and then ssh into our server and deploy the changes:

  # Name of the job is deploy
  deploy:
    needs: job1
    runs-on: ubuntu-latest
    steps:
      # SSH into the server
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          # The server IP of our VPS from DigitalOcean
          # or any other provider. This is again
          # passed in as a secret.
          host: ${{ secrets.SERVER_IP }}
          # Username of the user sshing into the server
          username: ${{ secrets.SERVER_USERNAME }}
          # The private ssh key in order to get access
          key: ${{ secrets.KEY }}
          script: docker pull ghcr.io/owner/packageName:latest && docker run ghcr.io/owner/packageName:latest

So the above job takes care of deploying as well.

Note that you might have a more complex deploy script for Docker in order to consider things like if the container is already running and ports etc.

Final steps

Now that we have our steps file in place, we need to tell GitHub to run it when we push something. We can do that by saving the file in the .github/workflows/ directory of our project. So if the name of the file is build.yml, we will have the following directory structure.

-- project
  |-- .github
    |-- workflows
      |-- build.yml

Once the file is located like above, it will be automatically picked up by GitHub. However, we might not want the action to run when we push to certain branches. The file should just run on a few branches (let's say staging and production), we can do that by the following:

on:
  push:
    branches:
      - "production"
      - "staging"

We can also give the action a name by the name syntax:

name: Build API and Deploy

Adding secrets

Now, you might not want sensitive data to be passed on along with the code. Especially if the repo is public. In a case like this, we can use GitHub's secrets feature to store sensitive data like ssh private key etc.

This question on StackOverflow explains how to add secrets on GitHub

Conclusion

We started with 5 steps, each one of them required manual interaction. Now we are left with one step (git commit) that requires interaction. All the other steps are automatically handled by our CI service (GitHub). CI is a very useful aspect of deployment, especially if you're a web developer.

When we add all the jobs and steps to one file, it turns out as following:

name: Build API and Deploy

on:
  push:
    branches:
      - "production"
      - "staging"

jobs:
  example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2
      # Login to docker
      # This can be used for both Docker Hub and
      # GitHub container registry.
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          # Remove the following line if you want to
          # login to docker hub.
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          # secrets are GitHub actions that can be added
          # from settings of the repo.
          password: ${{ secrets.CR_PAT }}
      # Build the docker image and push it.
      - name: Build image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          # Remove the `gchr.io/` if you're pushing to
          # Docker Hub
          tags: ghcr.io/owner/packageName:latest
  # Name of the job is deploy
  deploy:
    needs: example-job
    runs-on: ubuntu-latest
    steps:
      # SSH into the server
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          # The server IP of our VPS from DigitalOcean
          # or any other provider. This is again
          # passed in as a secret.
          host: ${{ secrets.SERVER_IP }}
          # Username of the user sshing into the server
          username: ${{ secrets.SERVER_USERNAME }}
          # The private ssh key in order to get access
          key: ${{ secrets.KEY }}
          script: docker pull ghcr.io/owner/packageName:latest && docker run ghcr.io/owner/packageName:latest

Adding the above workflow requires just a few seconds but it highly removes the manual work for me. I can now just make some changes and merge them to a branch and it will be automatically picked up by my pipeline and deployed in a matter of seconds.

If you'd like to see one of these in action.

CI can also be used for things like testing. The actions can be run on various events like Pull Requests.

This article was originally posted at my personal blog

Easily Test Mixins With Jest

Deepjyoti Barman — Fri, 14 May 2021 10:23:16 +0000

So recently I decided to add tests to one of my repos. Since I am new to JS to some extent, I started looking for what the best ways were to add tests to a frontend project. Well, as it turns out there are many libraries (duhh, it's JS afterall). However, there are various types of testing as well like Unit Testing or Component Testing.

This post is not about those tests though. After looking a bit for the best library to use with VueJS, I decided on using jest. I am not an expert but it doesn't take an expert to know that Jest was built for ReactJS. Well, for starter, Jest is built by Facebook and has a great integration with React. js. So yeah, Jest might not be the first choice of Vue users.

I loved the fact that it was so easy to setup and as I started writing tests I understood that it's actually very easy to use as well. So all in all, I ended up using Jest as the testing framework for my project.

Testing Mixins

After I had written tests for most of my components, I finally decided to write tests for my mixins. This is when I came to a stop. I started looking at the docs in order to know what the best way to test mixins would be. However, there wasn't much documentation regarding that.

This is when I realized something.

What are mixins?

A mixin is a class containing methods that can be used by other classes without a need to inherit from it

As stated on Wikipedia, a mixin is a class containing methods that can be used by other classes without a need to inherit from it.

In case you are not familiar to mixins (why are you even reading this article about writing tests for mixins then?), mixins are functions that developers can inherit in some other code and use them.

This also means that mixins cannot be used independently. What I mean is, let's say you have a mixin defined in VueJS. Now you want to use this mixin. You can easily import it and use. However, in your source code is there any use of the mixin without it getting imported? No.

Mock Component

We will create a mock component, a bare bones component basically, to inherit the mixin to that and test it.

Now that our doubt about mixins is out of the way. It is clear, we will need a component that can import the Mixin in order to test it. So what should we do?

We can just make a simple Mock Component. In my case, I went with a simple Mock Component written in Vue in the following way:

<template>
  <div class="test"></div>
</template>

<script>
/**
 * Return a component to be used as dummy in order
 * to mock things like mixins.
 */
export default {
  name: "MockComponent"
};
</script>

Once we have a component, we can easily use it to use our mixins and use it accordingly.

Using mock component with Mixin

Let's say we have a mixin with a function getBoolFromValue(), we can easily write a jest test for that now:

describe("Test mixin function", () => {
  const wrapper = shallowMount(MockComponent, {
    mixins: [mixinName]
  });

  it("should return proper value from bool", () => {
    expect(getBoolFromValue("on")).toBeTruthy();
  })
})

Why not use a component from the code?

Well, using a component already defined in the code would work too. However, why add all the bulk while testing a mixin. We want the mixin to be tested as easily as possible. As well, adding a component might add some bulky imports etc, so it is way easier to just keep a MockComponent that is just basically an empty template to test Mixins.

You can check the tests on my repo as well

This post was initally posted on my personal blog

Deploy Celery & RabbitMQ with Compose

Deepjyoti Barman — Wed, 07 Apr 2021 13:42:48 +0000

Off late, I had been working a lot with FastAPI. Recently, however I had a requirement. I needed to run tasks in the background after the request was made. So something like, when the request is recieved, add it to the task list and return a response. The task will be done in the background.

I somewhat knew about Celery, however never had the need to work with it. So I finally decided to use Celery and boy oh boy, was I surprised.

What is Celery

So, Celery is a task scheduler. What I described above in the first paragraph, Celery does exactly that! We pass it a task and it runs that in the background.

It basically runs the task as a synchornous function which effectively makes the function run in the background. You can read more about it here.

However, in order for Celery to run properly, it needs a broker.

What is a broker?

A broker will keep a list of all the tasks that are to be executed and will accordingly supply Celery with the tasks. Celery will then use the task and work on it.

Directly from Google:

A broker is a person or firm who arranges transactions between a buyer and a seller for a commission when the deal is executed.

This pretty much sums it up. In our case, we are the seller, celery is the buyer and we will use a broker in between in order to handle all the tasks. Brokers are also called as message queues or task queues.

Some of the brokers that Celery works with are:

RabbitMQ

In this article, I will primarily use RabbitMQ as the broker. You can read on how to use Redis with Celery.

Easiest way to setup RabbitMQ is to use a docker file. Using the following command, a container with RabbitMQ can be deployed within seconds.

docker run -d --rm -it --hostname my-rabbit -p 15672:15672 -p 5672:5672 rabbitmq:3-management

In the above command, the management image is used. You can check other available images here

Breaking down the above command:

We are mapping the 15672 port of the container to our host
We are mapping the 5672 port of the container to our host.

This is because the 15672 port serves the GUI for rabbitmq and 5672 is how Celery will communicate with it.

Worker

Now that we have our broker in place, let's use a Dockerfile to deploy Celery. Celery, since it does tasks in the background, is referred to as worker.

We will build the worker with the following Dockerfile:

FROM python:3.6

# copy contents of project into docker
COPY ./ /app/

# We will use internal functions of the API
# So install all dependencies of the API
RUN cd app && pip install -r requirements.txt

WORKDIR /app

ENTRYPOINT celery -A worker worker --loglevel=INFO

Using the above Dockerfile, we can deploy the worker.

Using a compose file

Now that we have two of the services ready, we are ready to write our docker compose file. Read more about docker compose here.

Usually, the worker is run along with an API and the API makes calls to the worker in order to run the worker tasks in the background.

In our case, we will be creating two containers:

RabbitMQ container
Worker container

We want our worker to access the rabbitMQ container through the network and accordingly use it as a broker.

Most of the time, you'll probably also need an API container that will also interact with the worker using the network.

Following is the compose file:

version: "3.7"

services:
  # Deploy the broker.
  rabbitmq_server:
    image: rabbitmq:3-management
    ports:
      # Expose the port for the worker to add/get tasks
      - 5672:5672
      # OPTIONAL: Expose the GUI port
      - 15672:15672

  # Deploy the worker
  worker:
    # Build using the worker Dockerfile
    build:
      context: .
      dockerfile: worker.Dockerfile
    # Need to access the database
    # OPTIONAL: If you worker needs to access your db that is deployed
    # locally, then make the network mode as host.
    network_mode: host
    # Pass the rabbitmq_uri as env varible in order to
    # connect to our service
    environment:
      # NOTE: Below we are using 127.0.0.1 because this container
      # will run on the host network, thus it will have access to the
      # host network.
      # If it would not have run locally, we would have had to
      # connect using the service name like following:
      # amqp:rabbitmq_server:5672
      rabbitmq_uri: amqp://127.0.0.1:5672
    # Make it wait for rabbitmq deployment
    depends_on: 
      - rabbitmq_server

With the above file, you can deploy it as follows:

docker-compose -f docker-compose.yml up --detach --scale worker=2 --build

In the above command, we are scaling the worker service to have 2 containers.

Gotcha's to look out for

Connection URI for RabbitMQ

Let's say we have RabbitMQ deployed in a container called rabbitmq. Now, from our worker container we need to access RabbitMQ in order to add tasks. In this case, we will have to connect to RabbitMQ using a connection URI. This URI will be something like:




> Note that we have name of the container in the URI. This will map the URI to the network of that container.

Typically this URI should be something like `amqp://localhost:5672`

However, now, let's say we need to run our container in the network. This can be easily done using the `network_mode: host` field in the compose file or the `--network=host` arguement to the deploy command.

In cases like this, our container will have the network of the host which means the RabbitMQ container will be accessible as it will be accessible to the network which will be:



```amqp://127.0.0.1:5672```



> Note that we exposed the port 5672 when deploying the rabbitmq container.

This article is posted on [my personal blogging site](https://blog.deepjyoti30.dev/celery_compose)

Automate Tests With GitHub Actions for Python

Deepjyoti Barman — Fri, 26 Mar 2021 06:16:44 +0000

Since the addition of GitHub Actions, a lot of things, that previsouly we had to use third party services for, have become easier. Recently, an user raised an issue in my downloader-cli repository. He suggested that addition of tests would be greatly helpful. More importantly because that way when people make a PR, the tests would tell the developers if the PR can be accepted.

So, I decided to add some nice tests for my app. downloader-cli is written in Python and it is a small, simple, downloader library that features a highly customizable, responsive progressbar for the commandline.

How will it work

The idea is simple. We will add a GitHub workflow file. This file will have the details as to when and what to do. For the sake of this example, I will make the tests run when a push is made to the master branch or a pull request is made to the master branch.

In case you don't already know, GitHub's workflow files are written in YAML and are pretty simple to get started with.

You just need to create a directory named .github and a subdirectory inside that named workflows. Inside the subdirectory, you can add as many workflow files as you want, given that all of them follow proper syntax and are written in YAML.

Adding Conditions to run

We will create a file named tests.yml and add the basic conditions to make it run on PR's and Pushes.

name: Run tests

on:
  push:
    branches:
      - "master"
  pull_request:
    branches:
      - "master"

The above code does the following:

Add condition to run on push on branch master
Add condition to run on pull_request on branch master

Add the job

So once we tell GitHub about when to run the action, we also need to pass them some commands on what to do when running the action. These can be called as jobs. One action file can have multiple jobs and jobs can depend on other jobs so that they will run only after the job they depend on is run.

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        # Run in all these versions of Python
        python-version: [3.5, 3.6, 3.7, 3.8, 3.9]

    steps:
        # Checkout the latest code from the repo
      - name: Checkout repo
        uses: actions/checkout@v2
        # Setup which version of Python to use
      - name: Set Up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
        # Display the Python version being used
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"
        # Install the package using the setup.py
      - name: Install package
        run: python setup.py install
        # Install pytest (you can use some other testing utility)
      - name: Install pytest
        run: |
          python -m pip install --upgrade pip
          pip install pytest
        # Run the tests. I'm using pytest and the file is in the tests directory.
      - name: Run tests
        run: pytest tests/test*

In the above code, we add a job with the name test.

This job runs-on the latest Ubuntu Version as deonoted by ubuntu-latest.
We are using the matrix strategy to run it for different python versions.

Then we have added a few steps. The steps do the following:

Checkout the latest code from the repo
Setup the python-version as passed by the matrix strategy.
Display the python version (just cross check).
Install the package. (In my case it is using the setup.py file.)
Install pytest.
Run the tests.

Push the code

Once you have the file setup, you can try try out the changes by pushing the code to the master branch.

Just do a push using git and go to GitHub and open the Actions tab. You will see a job running with the yellow indicator on the left. (The yellow means it is running).

Conclusion

If you want to take a look at the actions of my repo, you can check them here.

Also, GitHub Actions can be used for various automations. Most useful usecase (at least for me) being the ability to add CI/CD pipelines which have made a lot of my deployments painless. You can see an example of that in my ytmdl-web repo.

Also, in case you want to take a look at the file from my downloader-cli repo, you can check it here.

This post was originally published at my personal blog

I rebuilt my blog with Nuxt, now it loads in seconds!

Deepjyoti Barman — Sat, 20 Mar 2021 16:51:35 +0000

TL;DR: Rebuilt my blog (this page) with Nuxt in order to improve SEO and performance and give the whole thing a new look. Used Tailwind too (well, duh!)

I felt like it was about time I tried my hands on Nuxt. Considering that Vue is my goto frontend framework, Nuxt was inevitable to be skipped. Though, I cannot just learn a language, I need to build something in order to get to know the language all around. So I started wondering what would be a perfect project to build with Nuxt.

Why Nuxt?

So in order to know what project would be perfect for Nuxt, I started researching a bit on why Nuxt would be better as compared to Vue. Obviously, I came across SSR a lot more that I expected to. So it was time, I dived into SSR and built something that could benefit from it.

If you would just do a google search Why is SSR good, you'd be easily prompted to a lot of sites, however all these sites will have one thing in common, SSR is good for SEO. You see where this is going?

Out of all the apps that I have currently, this blog is the one that will be able to leverage SEO the most and yeah I was kinda bored of the old design, really started disliking it after a while.

So that was that. I started rebuilding my blog. The backend is still the same. The same old API built on FastAPI.

Features

Yeah, I had to list down the features that I think are way better in this version. Some of the key highlights are:

Way faster
Improved SEO
Better theme
Better dark mode
Comments (thanks to utteranc.es)
Everything is better

Yeah, I honestly think this version is pretty good. I mean the older one was not bad by any means, but a lot of things were just not perfect for a blog. Things like the SEO takes the whole page a long way along.

Theme

Since I have recently started using Tailwind, I went with tailwind again and I have to admit, I like it a bit more every time I use it. It just makes so many things so much easier. Things like handling a dark theme. Talking about dark themes, consider checking out the dark theme. Just open the menu (if you're on mobile) and click on the sun icon.

The themes are also set based on the device theme (if the user has not explicitly set a theme). I wanted to keep an option with auto that would theme the site based on the users mobile, however, I just couldn't find the motivation to add that. Perhaps in the next release?!

Comments

Okay, this is a big one. I was just going through a blog site and I came accross utterances, and I have to say it's a pretty neat idea. Like the whole thing is very easy to setup.

However, I will have to call it both an advantange and a disadvantage that it is built on GitHub. This means, if the blog has readers that do not have a GitHub account, would not be able to comment.

However, if the blog has readers that are mostly techies (read people with GitHub account), then it is like a wish come true. I think being built on GitHub, it is like an extra push for techies to comment.

But, it is what it is, it depends on the readers whether it would be a good idea.

Redesigned share menu

So, the share menu is now redesigned. It still continues the old approach however. If your device supports native share, then the custom menu will not be shown. However, most PC's does not really support it (can't vouch for Windows, totally doesn't work on i3-gaps), so the custom menu is a great addition.

Check it out by clicking on the share button at the end of the post, or at the top of the post.

Conclusion

I think, all things, considered, this was a pretty good chance for me to learn Nuxt and I totally enjoyed. I am already considering Nuxt for my upcoming projects. It's ofcourse good for SEO, however the whole SSR thing makes it very convinient to make the site performant. Also, really appreciate that the app is automatically built as a PWA. Anyway, some more posts coming up in the following days, consider subscirbing to the newsletter!

Also, the repo is now open source, in case you are interested, check it out.

This post was originally posted at my blog

Python has a built in search function. Here's how to use it

Deepjyoti Barman — Mon, 04 Jan 2021 14:58:27 +0000

I was going through the Python docs and guess what I came across? A function that helps you get closest matches to a list of other strings. Now why would that function be useful, you'd ask. Let me give you a proper example.

The problem

About 1 and a half years ago, I was working on playx. It is a tool that lets you stream music from the commandline directly from YouTube. While working on it, we decided that adding a feature to cache the files would be pretty nice. Now, while implementing that, we came across a problem. We needed to check if a song is already cached.

Now, the issue is, on YouTube, one song is uploaded by various channels which means the titles vary and since we were caching the songs using the title of the video from YouTube, that meant that if we did a direct match of the titles, it would be a very inefficient way to check if the song has been cached already.

For example, if we have the song name Sundown. This song might have been saved as Sundown - Gordon Lightfoot. However, the next time user tries to play the same song which had a different title, something like Sundown - Lyrics only | Gordon Lightfoot. In this case, a simple match would indicate that both songs are different and the tool would end up caching both even though the songs are same.

If you got an idea of the problem from reading the above, I think you would agree with me that it is similar to searching a particular file in a list of files.

This was originally posted on my personal blog and can be read here

Happiness status of your GitHub repo: repostatus

Deepjyoti Barman — Wed, 09 Dec 2020 14:21:23 +0000

People always ask "why this repo" and never ask "how is this repo", so I created an app that finds how happy a repository is.

TLDR; The app runs a sentiment analysis engine on your repo and finds how happy it is. Check repostatus

How?

So how exactly do you find the sentiment of a non living thing? Well, even I had that thought in my mind. Any repository is made up of people that contribute to that repo, people that interact in the repo's comments.

So, if we are able to run a sentiment analysis engine on the interactions of the people that are contributing to that repo, we might get somewhere?

repostatus extracts three important part of the repo (by using GitHub's API):

the commit messages
the comments on the issues
the comments on PR's

Once, we have these three things, we can combine them, filter out the unnecessary data and run our engine on it.

Tech

The backend is written in Python (FastAPI) and the frontend is written in Vue

First things first, the sentiment analysis engine used by repostatus is the textblob library. It is very easy to use and works great.

So now that we have the engine at our ease, what's next.

Backend

My goto tech stack is Python so no wonder I went with that for the API.

I used FastAPI for the backend. Off late, I have started liking FastAPI more and more and it was an obvious no brainer to go with it for the backend. It's ease of use with the efficiency is just awesome. If you haven't checked it out, do that, I'm sure you'll love it if you're a Python developer.

Services

Services that the API will offer are:

Internal API for the webapp
Public API
Badge API (Yep, you can use repostatus badges on your README).

I wanted to provide a Public API so that people would be able to use it for their own fun projects. The API is capable of working with both private and public repo's. The details for that can be found here

The badge is another thing that I thought would be a nice little addition. This badge works similar to how the travis build badges work or any badge. You can simply use the URL to embed it into your repositories README.

More details about the badge can be found here.

Frontend

I love using VueJS. It was obvious that I would use that to build the frontend. Now for the frontend I wanted to make sure that it doesn't restrict the user too much.

Thus, repostatus works with both private and public repos. For private repos, GitHub's OAuth is used which gives us access to that particular repo and then we run the engine over the repo.

One issue that I faced while implementing the OAuth was that I wanted to make the process seamless. If you go the the app now and select on the OAuth option, you will see the process is pretty neat.

Here's what it does:

Opens a new window and asks the user to give access
User gives access and GitHub redirects the user to my sites callback endpoint.
Window closes and the app shows all the users repos.

The above steps makes it look real seamless. However, the hard part for me was to figure out how do I know when the OAuth is done and then show the window.

Seamless OAuth

So in order to make it seamless, I implemented the following flow:

User clicks on OAuth button, new window is opened and the app keeps waiting for it to close.
In the new window, the user is redirected to the callback URL which returns a nice HTML page that shows the user that the window will close in 5 secs. After 5 seconds the window closes and the app knows that the OAuth is done.
The app then tries to find the repo's of that user and accordingly shows the user all the repos.

I know, it's not much. I have to say, though, I really liked implementing this one little feature and making it so seamless. I'm not even kidding, I just kept on doing OAuth on repeat after implementing it, for a while.

Badge

An example of the badge can be seen below.

Isn't it cool?! It supports options like style of the badge where for-the-badge can be used. It changes the color of the badge based on the happiness status of the repo.

How is happiness exactly calculated

As I mentioned earlier, the happiness of any repo depends on certain parts of the repo. Thus, once the commit messages, issue comments and the PR's comments are extracted, they are run through a filter.

Filtering the data

This step makes sure that the data is cleared off of any unreadable content, like an image that the user might have posted in the comments. Or some code inside a code block that the user might have added in an issue report.

Running the engine

Once the data is cleared off all the unreadable content, it is passed to the textblob library that runs the engine on the data.

Now, this library returns a float score. This score is between -1 and +1 where +1 indicates happy and -1 indicates sad.

Thus, when the engine is run on the comments, let's say we get a score of 0.8, this means, based on the comments, the repo is not that happy but it's more happier as compared to being sad. So repostatus considers that based on the comments the repo is happy.

We do the above process on other aspects of the repo too, the commit messages etc.

Once, all the individual scores are available, they are added up and an average is calculated. This average, mathematically, is also between -1 and +1. This is the overall happiness status of the repo.

Based on this score, it is decided what color is to be assigned to the repo.

Caching

Since the engine takes a pretty hefty amount of memory, the score of any repo is cached for 15 days.

The badges are cached for 24 hours and only after that they are updated.

This caching was necessary in order to make sure the performance of the API was not effected.

Try repostatus here

Source

repostatus is open source. Source for the backend and the frontend can be found below

trotsly / repostatus

Get Happiness status of your repo

Backend for repostatus. Repostatus lets you calculate the happiness status of your repository.

What we do?

We consider various parts of the repo like commit messages, comments on issues, pulls etc and run a sentiment analysis engine on the data in order to find out the happiens status.

Setup

You'll need to setup an environ variable named GITHUB_TOKEN that will contain an access token. In order to get the token, follow this article and accordingly save it to the environment.

One way to save something to environment is:

from os import environ
environ.set('GITHUB_TOKEN', '<your_token>')

Otherwise, it can also be set through the rc file, i:e zshrc, bashrc etc

Tests

For the tests, we are using pytest

If you wish to run the tests yourself, make sure you have it installed. The tests can be run by the…

View on GitHub

I created my own blog management system. Here's how I did it.

Deepjyoti Barman — Tue, 17 Nov 2020 16:28:14 +0000

I have had a personal blog for a while now. I had written a few posts there but it was far from perfect. It was built using basic HTML, CSS and JS. I had seen all this awesome sites with unique designs and I thought, why not create my own?

I went with a front-end for back-end approach which means the back-end needed to be robust in order for the content to load properly and fast.

I built my API using FastAPI for Python and the webapp using VueJS.

Building the API

Structering the API

I laid down the endpoints that will possibly be needed in order for the blog to work properly.

Here are some of them

/posts: Allow GET, POST, UPDATE, DELETE where only GET will be public and other methods would need a secret token in order to access them
/subscribe: Allow POST, DELETE where only POST will be public.
/related: Allow GET to get the related posts to a post.

For the database, I went with mongoDB. So the idea is to store the posts as markdown in the database and let the API access it. The frontend will then just make a request and get all the data from the API.

Implementing the API

It took me a few days to get the API ready. FastAPI was really helpful with their openapi docs to provide a nice interface in order to test the API without using curl.

The posts and the subscribe endpoint are pretty self explanatory, here's how I created the related endpoint.

Finding posts related to a post

Since all the posts will have tags linked to them, I used those to calculate a score for the post.

The /related/<post_id> endpoint was structered to pass a post_id that would tell us which post to consider the root post. Once we have this post, we can fetch all the other posts and calculate a related score.

This score is calculated in the following way

# Consider root_tags are tags of the source post
# other_tags are the tags of the other post that.

def calculate(root_tags: List, other_tags: List) -> int:
    # Remove duplicate tags if present
    root_tags = set(root_tags)
    other_tags = set(other_tags)

    # Calculate the score now
    score = len(root_tags.intersection(other_tags)) / len(root_tags)

    return score

Above code does the following:

Intersect the root tags with the other tags set giving us the common tags
Score is the division of the number of tags common between the two posts and the total number of tags present in the actual post.

This way we get a score that would be between 0 and 1. Once we have this score, we can sort the posts based on the result and the posts that have a higher score are more related to a post as compared to other posts.

Building the webapp

The webapp is built using VueJS. The whole idea of the frontend for backend approach is, the frontend will be dependent on the backend for the data.

Structuering the app

Before building the app, I went through a few points that the app should be able to do

It should have a home page that will show the users all the posts available
It should be able to show each post
It should be fast
It should be able to load posts directly through a route (for example: blog.com/nana should load the post nana and not just the webapp that is hosted on blog.com)

The first part is pretty simple. I just used the /posts route in order to get all the posts and then displayed them in a nice way.

Rendering the post dynamically

Here's the flow of how a post is rendered

If the user clicks on a post from the home page, the content of the post is passed to the router and accordingly rendered in the post view.
If the post is opened using a route, the route is used to find the post and accordingly the content is shown.

The above basically does two things:

Makes the load speed faster if the user opens a post from the home page
Adds the ability to load a post using the route.

To pass the contents of the post by the route, use a route prop to pass a object that will hold the contents. In the Post view, check if this object is available or not, if not use the route to make a request and fetch the content.

What about SEO?

Well yeah, I know SEO is important. For loading the meta tags I used vue-head which renders the meta tags dynamically after the post is loaded using the API.

This is pretty important since the meta tags are used by all the bots crawling the page. Also, Google bots are now able to crawl dynamically rendered content whcih means it should not be an issue if the tags are loaded dynamically using JS.

Problems

Except the common occurence of bugs, I did not had any problems with the back end. However there was one issue that made me question the whole idea. How do bots that do not have the ability to crawl dynamically rendered content crawl the page.

For example, twitter bots crawl a page in order to show a nice card view. If the bots are not able to crawl the page then the card won't be there. Not just Twitter, a similar functionality is used by various other social share bots like the ones from Facebook and LinkedIn.

How to let bots crawl the page

Well, so how did I fix this issue? At first, I obviously thought this would be inevitable because there's no way the bots would be able to detect dynamically rendered content. One solution was to go with server side rendering but I'm better off not diving down that road.

So the solution that I went with was to write a static file server in Python.

What would our server do?

It should be able to return the static html, JS, CSS files.
It should be able to return a rendered HTML with just the meta tags if the request is made by a bot.

I built the server using Flask for Python. It detects the request making entity using the User-Agent header being passed and accordingly returns an HTML. If the request is being made by a bot, it returns some HTML content that has the meta representing the post.

Else it returns the proper static files.

You can read about it here

You can check my blog page here

This post is also published in my personal blog

DEV Community: Deepjyoti Barman

Dynamic Path Matching with Go and mux

More details of the problem

Naive Approach that was considered

How it's Solved

Matchers

Handlers

Enhancements to the above

Setup your own blog on top of GitHub

How does it work?

Why Emanates?

Show me how it is

How should I start

Emanates: Build a blog on top of GitHub Issues and Actions

You can check Emanates GitHub here

Demo site for anyone interested

A handy deployment script for your pipeline

So how to do it then?

Deployment Script

Example workflow with the script

This article was originally posted on my personal blog

Github, Python and Docker: The Sweet Trio

How do the above fit in the bigger picture?

Typical Scenario

Remove redundant steps

What is GitHub Actions (or anything similar)?

How do we tell GitHub Actions to do something?

Removing redundant steps

Automate pulling the latest code changes

Automate building the docker image and pushing it

Automate SSH into server and deploy latest image

Final steps

Adding secrets

Conclusion

This article was originally posted at my personal blog

Easily Test Mixins With Jest

Testing Mixins

What are mixins?

Mock Component

Using mock component with Mixin

Why not use a component from the code?

This post was initally posted on my personal blog

Deploy Celery & RabbitMQ with Compose

What is Celery

What is a broker?

RabbitMQ

Worker

Using a compose file

Gotcha's to look out for

Connection URI for RabbitMQ

Automate Tests With GitHub Actions for Python

How will it work

Adding Conditions to run

Add the job

Push the code

Conclusion

I rebuilt my blog with Nuxt, now it loads in seconds!

Why Nuxt?

Features

Theme

Comments

Redesigned share menu

Conclusion

Python has a built in search function. Here's how to use it

The problem

This was originally posted on my personal blog and can be read here

Happiness status of your GitHub repo: repostatus

How?

Tech

Backend

Services

Frontend

Seamless OAuth

Badge

How is happiness exactly calculated

Filtering the data

Running the engine

Caching

Try repostatus here

Source