DEV Community: Zhalok Rahman

Essential docker commands

Zhalok Rahman — Fri, 05 May 2023 04:25:47 +0000

1. Showing all containers

docker ps -a

2. Showing running containers

docker ps

3. Showing all images

docker images

4. Showing all conatiner ids

docker ps -a -q

5. Showing all image ids

docker images -q

6. Stopping all containers

docker stop $(docker ps -q)

7. Deleting all containers

docker rm $(docker ps -a -q)

8. Building image from docker file

docker build <directory in which the dockerfile exists>

9. Building image from docker file with a tag name

docker build -t <tag name> <directory in which the dockerfile exists>

10. Running an image in a container

docker run <image_id>or
docker run <tag name (if provided while building)>

11. Running an image in a container with interactive shell terminal

docker run -i -t <image_id or tag_name> bash

Things I wish I knew as a hackathon participant

Zhalok Rahman — Thu, 04 May 2023 04:56:47 +0000

Hackathon is one of the most popular type of competition in the field of programming. It's basically a project development and showcasing competition where you get to implement a project idea on the spot and solve some particular real life problems, also add new features based on judge recommendations.

Typically in a hackathon the team consists of 3-5 members. But of course if it is based upon the scale of the project that is required. One of the key points on winning a hackathon is the proper co operation and proper co-ordination as the workload is being distributed among the team members.

Based on my experiences of participating on hackathons I would like to share some of the tricks that I wish I knew before.

1. Don't focus on scalability of your architecture

You are not here to build a production level application that will have 1 million users. You just need to convince the judges that you do have the proper knowledge and potential to turn your idea into reality.

2. Use pre-built components from UI libraries as much as possible

You don't need to build a cutting edge User Interface by applying a lot of styling. You just need to present that the features are working properly. If you put a lot of time in designing a cutting edge UI but don't get the time to make them functional, the judges will not be happy.

(However, there are some project showcasing competition where people get a prize by just creating a figma design. But I don't consider them as hackathons, they are startup business idea competitions for me. Well actually they are business idea competitions but sometimes they name it as hackathon.)

3. Use cloud services as back-end and database instead of developing own server-side application.

Don't spend time on developing APIs for basic operations such as authentication and CRUD operations. Instead use cloud services like Firebase,

4. Integrate modern techs such as Speech Processing, Computer Vision, Maps, etc. with cloud services which are not free for everyone.

Cloud services provide a lot of useful services that are easy to integrate and function. But the problem is they are not free. They require card information even if you want o create a free trial account.
Well here's the hack, you can use your student email account to apply for github student developer pack. Once you have that you have a superpower as student dev. You can have 6 month free access to Microsoft Azure. Azure comes up with almost all the services you will possibly need to make your project an unique one from others'.

5. Use docker if you really need to migrate applications to a single machine.

You may have developed front-end application and back-end application or maybe also a mobile application on different machines. But while presenting you need to demonstrate all the applications in a single machine. Therefore it's really a good and safe choice to dockerize the applications.

6. Create project in GitHub and keep track of the tasks.

As you will be working on team, It's really important to keep track of the features that are implemented and that are in progress and assign task of implementing features to team members, it can be very useful to have a GitHub project. Or you can use other project management tools like Jira and Trello.

7. Learn how to work on branches and resolve merge conflicts in GitHub and VScode.

If you are working in parallel, you definitely need to work on different branches. But in order to sync you also need to merge your codes into the default branch _(which is main
branch in most of the cases) _. Therefore you will have conflicts. Learn how to pull and resolve conflicts in local machine with VSCode or any IDE that you are using. They provide a GUI tool for you to work around gits or any other version control systems.

8. While testing use the IP addresses of your machine in local network for communication between apps running on different machines. No need to merge them on single machine.

If you need to connect your front-end application with your back-end application which are running on separate devices you can use their IP addresses on the local network and port numbers. But make sure all the devices are connected to the same local network (e.g same WiFi). You can find the local IPs of the machine using some commands on your terminal which vary from machine to machine, therefore you need to do some googling based on your machine environment.

9. Finally the most important one: Present recorded workflow.

Make small recording of your work flow of your project using
loom and present that to the judges instead of demonstrating the project on your machine. Because if you do that there may occur some errors due to unknown reasons which is not possible to fix at that time.

That's all from me. If you think I missed something, share your experience with me ...

Optimizing complex relational query in MongoDB using aggregation pipeline

Zhalok Rahman — Fri, 28 Apr 2023 10:39:25 +0000

we all know that MongoDB is a nosql database and is mostly used for non relational database.
But you can also design a database with strong relationship between tables (or models for mongo) using MongoDB as well. But what will be the problem then? The problem will be if we use the basic APIs of MongoDB like FindOne(), FindById() etc, then we may end up writing a lot of database calls on a single API endpoint.

For example, let's assume that we have a simple classroom management system where there will be a classroom and under one classroom there can be multiple teachers and students.
So, now if we want to design a database schema for this design what we need to do? Let's first understand the relation between classroom teachers and students. Hence the relation between classroom-teacher is many to many and so as the relation between classroom-student.

Now let's design a classroomSchema using mongoose. How we will do that?
A naive approach can be like this:

but whats the problem? this database is not normalized and may lead to performance issue while scaling up.
So what can we do?

We can come up with a bit efficient solution by storing the object id of the teachers and students documents instead of keeping the whole documents.

Great, but now if we need to build a rest api endpoint which will return all the classrooms along with all the teachers and students associated with them

we can write the following code for that:

but what's the problem with this solution? this solution will lead to a larger response time because there are 3 database calls here and the complexity of the logic being applied is O(n^2) (quadratic) which both are going to make latency in the response. To get out of this problem, we need to make only one database call and get all the necessary information that we need directly from the database query instead of writing any additional logic.

But how can we do that? we are not storing the full document in our array of the classroom document, then how can we do combine classroom, teacher and student document and then make query?

We can use aggregation pipeline. Aggregation pipeline allows you to make all kinds of query that you can perform in SQL with just some different syntax. I don't know why the mongoDB tutorials I found online did not include the concept of aggregation pipeline yet it is necessary to understand.

MongoDB aggregation pipeline allows you to write multiple queries on a single database call.

Right now our goal is to get all the classrooms along with their enrolled students and teachers. Which means we need to combine classroom, teacher and student schemas together and then run the find() query.

we can write the following code for that

we can see that the model.aggregate() function receives an array of objects. In each object we can write individual query. We used first $lookup query to join classroom with teachers schema and second $lookup query to join classroom with students schema.

Let's see these in action and observe the difference.

Let's first create three endpoints
one for creating teachers
one for creating students
one for creating classrooms with 3 teachers and 3 students

Each of these endpoints will create 1000 teacher and students and 100 classroom entries in the database

now we will build two endpoints to fetch the classrooms.

GET /classrooms/unoptimized

GET /classrooms/optimized

now lets make the unoptimized query

see the response time. its around 850 ms

Now lets see the optimized endpoint response time

Just see the magic
we have managed to reduce the response time from 850 ms to just 15 ms.
HOW COOL IS THAT

Full Code -->

Hackathon Winning Experience…

Zhalok Rahman — Fri, 26 Aug 2022 05:01:00 +0000

Recently, our team has won a hackathon competition HackNSU4 organized by the ACM student chapter, North South University, Bangladesh. It was really an amazing achievement for the whole team as we had to manage a lot of academic activities and also had to travel 6 hours to participate in this competition.
However now I would like to share our experience in the competition and some tips for anyone who is eager to participate in any hackathon in the future.
In this hackathon, at first, we were asked to submit an idea on some specific scopes. There were mainly 4 scopes on which we could submit our idea: transport efficiency, Reducing energy crisis, and reducing cyber harassment.
we submitted our idea in the first scope where we had the idea of an advanced ticketing system. our idea was basically to reduce the number of unused seats on the Bangladesh railway and prevent tickets from being sold on the black market.
However, our idea was selected and we were invited to participate in the final round. In the final round, there was no pre-code allowed. We had to start building our application from scratch. This was an 8-hour hackathon and within this time we had to complete the prototype and also the presentation and the speech.
The most focused part of the competition was the presentation because it is definitely not possible to implement all the features in the prototype and show a demonstration. However, we still worked really hard and created a prototype for web and also mobile applications.
With the grace of Almighty God, we have managed to become the champion with all our struggles but also I think there were some parts where we could have worked a bit more strategically.

Prototype: While making the prototype only focus on the features that are the highlighters of our idea instead of implementing general stuff like login, and signup page. Because this is a prototype, you need to only show that you know how to make this happen and you can code to make this happen.

Dummy Data: You don't need any database. So don't waste your time configuring your database. As we were working with transactions we were using a MySQL database and I had to spend a lot of time just connecting my application with my database. But you really don't need this because this is a prototype.

Front-end: Again just make a dummy front-end. It doesn't need to function properly and fetch any data. Just set some internal logic and data with hard coding and show that you can do this with the application, that's it

My Docker Experience as a Beginner.

Zhalok Rahman — Thu, 14 Jul 2022 04:46:29 +0000

Recently I have started to get my hands dirty with docker. Here I would like to share some of my silly mistakes as a beginner and how I solved them.
Basically, I was trying to dockerize my MySql database instance as me and my partner are working on completely different systems.
Starting from installing docker in my system to creating the run the MySQL in a container was fine. But the problem arose when I wanted to connect my node js application with my dockerized database.
Now before I explain the problem that I was facing I would like to share a bit of knowledge on how to connect a node js application with a MySQL database in short. For connecting node js application to any MySQL database server, we typically use a MySQL client using an npm package called "MySQL". https://www.npmjs.com/package/mysql. Now for establishing the connection between my node js app and MySQL server, there is an API provided by the "MySQL" npm package called "createConnection()". In the function, there are some options that need to be passed. A typical createConnection function looks something like this:

`const mysql = require("mysql");
const connection = mysql.createConnection({
host:"localhost",
port:"3306",
user:"root",
password:"03041959",
insecureAuth:true,
database:"BookLook",

});`

The problem was that I did not understand what the port number and the host needed to be used to connect with my dockerized database. Before jumping into the actual solution that worked, I would first like to go through the failed solutions that I initially tried and why they failed.
Solution1: I first ran the MySQL server in the docker container, then found the hostname using the command "select @@hostnames" and I got the docker container ID as the result, so I thought probably a MySQL server running in a docker container should have the hostname as the docker container ID. Then I changed the host value from "localhost" to "". But it did not work out.
The reason that this solution failed is that even if the MySQL instance is running inside a docker container, they are still on the same machine. Therefore, the dockerized MySQL instance is also running on localhost. So the host was basically not the problem. Then what was it? Why I was not able to connect and make queries to my dockerized MySQL instance using the NodeJS app. Well, the problem was with the port.
Here I would like to note something that may help in future similar problems. If you are experiencing communication issues between two processes running on the same server, you should be 90% certain that the issue is related to the port. Ports allow processes to communicate with one another. However, there are other ways for inter-process communication, such as "shared memory". However, that's an OS topic and I am not diving into it.
Now that I understood that the problem was with the port, I did some digging with the Docker communication via the port. Then I figured out that Docker has a concept called "port mapping". To learn about port mapping, you can read this article:

https://docs.docker.com/config/containers/container-networking/

In short, the docker mapping creates two ports. One for the Docker host and another one in the Docker container for running the application. The Docker host post is mapped to the Docker container port. So whenever you are trying to communicate with any dockerized application, you need to communicate via a docker host port and that will be redirected to the docker container port (where the application is running.
) that corresponds to the Docker host port. How can you specify the port mapping that is also in the link mentioned above?
Ok, so let's explain the final successful solution, I simply mapped the port 3306:3306. But while mapping the port I was facing some errors as this port was already in use by localhost as my MySQL-server was already running at 3306 port. Therefore, I closed the MySQL server and then ran my Docker container with the above mentioned port mapping. Boom, it worked.