DEV Community: Siddharth

Microservices and What is a Messaging Queue?

Siddharth — Fri, 10 Nov 2023 14:42:23 +0000

Well as a backend developer, we know that maximum of the projects out there are almost the same either a clone of a website like YouTube, Facebook, Instagram, etc. No doubt they teach you a lot of things from database design to writing the API logic. But to make your resume stand out we need to add some jazz to it and this can be done by using a messaging queue like RabbitMQ and breaking down our project into a microservice architecture.

So I know I have thrown a lot of terms out there and as a beginner, you might not be able to understand some or all of them. So let's tackle them one by one.

Microservice architecture

Now what is a microservice putting it in simple terms as of now all the YouTube tutorials you see and everything you develop are contained in a single Project. Like, let us take an e-commerce website we have all things like authentication, payments, delivery, reviews everything under one big project. Now let us say if our payment service goes into fault due to some reason now this will eventually affect our application and might bring down the whole system.

Now as an engineer Do we really need it Nooo. If payment fails at Amazon, or Flipkart they don't break down they still work well try-catch block helps sometimes but with severe errors it will break. So let's break our main project into multiple services based on functionality like

Authentication
Payments
Products

Create a separate service for each of them and eventually have communication between them moreover even if one goes down others are still working so your whole website won't go down. This will help you a lot in the longer run creating a robust architecture for your website and also it's really scalable at the same time. So in brief breaking down and modularising your project based on your business logic is more of a microservice architecture. Well, do it up to some level breaking it to a granule level will cause a lot of problems and then you will spend a whole lot of time just understanding it rather than coding it out. To read more about microservices follow this article here

Messaging Queue

Imagine a messaging queue as a kind of digital post office for computer programs. It's a system that helps different parts of a computer program or different computer programs talk to each other in a safe and organized way.

Here's how it works:

Sending Messages: Think of one part of a program as sending a message (like an email) to another part. This message can contain information, instructions, or data.
Message Storage: Instead of sending the message directly, it's placed in a queue, like putting your letter in a mailbox. The message waits there until the other part of the program is ready to read it.
Receiving Messages: The other part of the program checks the queue and takes out the message when it's ready. It then does what the message says or uses the information in it.

The cool thing about messaging queues is that they allow parts of a program to talk to each other without waiting for an immediate response. This is like sending an email - you don't need an instant reply, and it allows different parts of a program to work at their own speed.

Messaging queues also help when you have lots of messages coming in or going out. They can keep messages organized, make sure they're not lost, and send them to the right place.

So, messaging queues are like the post office for computer programs, helping them communicate in a smooth, organized, and efficient way.

Now that we know the basics of what is microservice architecture and what is messaging queue we have the basic blocks to create something in our own project. Even Now you didn't get it Don't worry I will help you develop it as well.

Now let's say you have a project that has an emailing functionality so write now it's integrated into it. So now Let's create a separate microservice that does the email-sending functionality for you. This can be easily done using two ways. Let's explore them both ...

Using the normal API call

Normally we know that the frontend works by pinging the backend and sending out a response to it we can use that as well because we have two different services the frontend and backend and they are communicating with each other via API call. So why can't we have to backend sending API calls to each other and exchanging information between them?

Now to make it more interesting I have used two different languages here we are creating our email and everything in nodejs/javascript and sending it out via a Django service here is the code below

const express = require('express');
const axios = require('axios');
const bodyParser = require('body-parser');

const app = express();
const port = 3000; // Choose a suitable port

// Use middleware to parse JSON request bodies
app.use(bodyParser.json());

// Define a route for receiving email data from the frontend
app.post('/send-email', (req, res) => {
  // Access email data from the request body
  const emailData = req.body;

  // Forward the email data to the Django email service
  axios.post('http://django-email-service-url/send-email/', emailData)
    .then(response => {
      console.log('Email sent successfully');
      // Respond with a success message (customize as needed)
      res.status(200).json({ message: 'Email sent successfully' });
    })
    .catch(error => {
      console.error('Error sending email:', error);
      // Respond with an error message (customize as needed)
      res.status(500).json({ error: 'Error sending email' });
    });
});

app.listen(port, () => {
  console.log(`Backend server is running on port ${port}`);
});

Now the emailing part in django service
routes.py

from django.urls import path
from . import views

urlpatterns = [
    # Define the URL route for sending an email
    path('send-email/', views.send_email, name='send_email'),
]

views.py

from django.http import JsonResponse
from django.core.mail import send_mail
from django.views.decorators.csrf import csrf_exempt

@csrf_exempt
def send_email(request):
    if request.method == 'POST':
        to = request.POST.get('to')
        subject = request.POST.get('subject')
        message = request.POST.get('message')

        try:
            send_mail(subject, message, 'your@email.com', [to], fail_silently=False)
            return JsonResponse({'message': 'Email sent successfully'}, status=200)
        except Exception as e:
            return JsonResponse({'error': str(e)}, status=500)

    return JsonResponse({'error': 'Invalid request method'}, status=405)

So right now as we can see that we made two microservices communicate with each other booom.You have achieved your first microservices up and running.

But it looks kind of messy this way and it is not right as well cause if there are multiple requests to send an email there might be an issue of API throttling secondly it's not idle when we have a lot of data that needs to be communicated so and let's make it much cleaner using a messaging queue and for this tutorial we will be using Rabbit MQ.

Using RabbitMQ

Step 1: Install RabbitMQ
First, you'll need to install RabbitMQ. Follow the installation instructions for your specific operating system on the official RabbitMQ website: RabbitMQ Installation Guide
Step 2: Set Up a Node.js Producer
In this step, we'll create a Node.js service that acts as a message producer, sending email data to a RabbitMQ exchange.

npm install amqplib express body-parser

Create a Node.js script to send email data to RabbitMQ:

const express = require('express');
const amqp = require('amqplib');
const bodyParser = require('body-parser');

const app = express();
const port = 3000;

app.use(bodyParser.json());

// RabbitMQ server connection
const rabbitmqUrl = 'amqp://localhost';
const exchangeName = 'email_exchange';

app.post('/send-email', async (req, res) => {
  const emailData = req.body;

  try {
    const connection = await amqp.connect(rabbitmqUrl);
    const channel = await connection.createChannel();
    await channel.assertExchange(exchangeName, 'direct', { durable: false });

    // Publish email data to RabbitMQ
    channel.publish(exchangeName, '', Buffer.from(JSON.stringify(emailData));
    console.log('Email data sent to RabbitMQ');

    res.status(200).json({ message: 'Email data sent to RabbitMQ' });
  } catch (error) {
    console.error('Error sending email data:', error);
    res.status(500).json({ error: 'Error sending email data' });
  }
});

app.listen(port, () => {
  console.log(`Node.js service is running on port ${port}`);
});

Step 3: Set Up a Django Consumer Now, let's create a Django service that acts as a message consumer, consuming email data from RabbitMQ and sending emails.

pip install pika

Create a Django script to consume email data from RabbitMQ:

# consumers.py

import pika
import json

def send_email(email_data):
    # Implement your email sending logic here
    pass

def process_email(channel, method, properties, body):
    email_data = json.loads(body)
    send_email(email_data)
    print(f'Email sent: {email_data}')

def consume_emails():
    rabbitmq_url = 'amqp://localhost'
    connection = pika.BlockingConnection(pika.URLParameters(rabbitmq_url))
    channel = connection.channel()
    channel.queue_declare(queue='email_queue')

    channel.basic_consume(queue='email_queue', on_message_callback=process_email, auto_ack=True)
    print('Email consumer is waiting for messages. To exit, press Ctrl+C')
    channel.start_consuming()

You'll need to call consume_emails to start the consumer, which will listen for email data and process it.

Step 4: Running the System Start RabbitMQ: Run the RabbitMQ server on your local machine.

Start the Node.js producer: Run your Node.js service that sends email data to RabbitMQ.

Start the Django consumer: Run your Django service that consumes email data from RabbitMQ and sends emails.

With this setup, the Node.js service will send email data to RabbitMQ, and the Django service will consume the data from RabbitMQ and send emails.

Remember to implement the actual email-sending logic in the send_email function in your Django consumer. Also, adjust the RabbitMQ settings as needed for your production environment.

So with this, you can easily use a messaging queue to send out messages between services and work asynchronously. You can easily tell your interviewer how you have implemented a messaging queue in a normal college project.Giving you an edge over other candidates

Django with Rest Tutorial -2

Siddharth — Tue, 07 Feb 2023 17:45:18 +0000

Hey everyone this is the 2nd tutorial of Django and in this we will be creating database of Hogwarts and get Harry and his friends registered in it. So lets start

Quick Recap

Created a project named harrypotter/Hogwarts and created an app for grffyindor for it.And added all the apps in INSTALLED apps in settings.py
Commands used until now

django-admin startproject Hogwarts
python manage.py startapp grffyindor

So now lets move forward from here.

Folder Structure:

So here is what the folder structure might look like in your machine and now coming to all files that are mentioned in here

urls.py This file contains all the urls and function mapped to it for now it would be having admin/ url only that comes in by default we will creating our own urls soon!!

settings.py This is an important file it contains all settings of your project from connection to database to the baked authentication that djangon provides plus if you want to create custom authentication using JSON web tokens we need to edit here first here

So for Hogwarts folder these are the files that you need to know about as of now further as we move we will discuss other folders

Now to our app Grffiyndor
views.py This file mostly contains all the functions that helps us to send data to frontend from database and modify it on our needs

models.py This is a really important file here we create all the tables that we need to store our data soon we will be creating our table for storing details of the database.

tests.py This file is majorly used for testing and creating testcases for our app

Now these are majorly all the folders you need to know.

Making our First API

So create a folder named urls.py in Grffyindor once you have done that go to urls.py of Hogwarts and write the following line of code
Hogwarts urls.py

from django.contrib import admin
from django.urls import path, include
#make sure to import include
urlpatterns = [
    path("admin/", admin.site.urls),
    path("grffyindor/", include('grffyindor.urls'))
]

So basically here when we are adding including grffyindoor/ path and writing include('grffyindor.urls') it means that all the urls starting with grffyindoor/ will be checked in grffyindor urls file for further path

Now once you have done this open urls.py of Grffyindor and write the following piece of code:

Grffyindor urls.py

from django.urls import path
from . import views
urlpatterns = [
    path('hello/', views.hello)
]

So here we have created a path hello/ when we go to grffyindor/hello/ our hello function fires off which helps in processing the data.

After doing this open views.py file and lets write our first view to return something to frontend

from django.shortcuts import render
from django.http import HttpResponse
# Create your views here.


def hello(request):
    return HttpResponse("Welcome to Grfyindoor")

So here we have created a function hello() which takes an object by default it should be request and its passed automatically here the request object refers to the incoming request from client we will be exploring this request soon now moving inside we are just returning a Welcome message now you might ask that why did we write HttpResponse() instead of just writing returning that string firstly we can't send a string it should be a json/xml file so that the frontend language could understand it. And a response has certain headers too now we can manually add them but its alot of work so we use HttpResponse.

Now lets fire up our server

python manage.py runserver

and open chrome or whatever browser you have and go to the url you created i.e http://127.0.0.1:8000/grffyindor/hello/

Congrats!!! You are Welcomed to Hogwarts and you created an api all by yourself, Lets take a break and celebrate it.

Creating a Database

So lets start with our database
Django uses a ORM for creating a database and communicating with it you can think it of as a middle man.And each table is a class in django and you can communicate with it using an object as simple as that. So lets open models.py file in grfyindoor and lets create a table Student.With some simple propertys like student name,age and gender and we will add more once we move forward

from django.db import models
from .constants import GENDER_CHOICES

class Student(models.Model):
    Name = models.TextField()
    Age = models.IntegerField()
    Gender = models.CharField(max_length=2, choices=GENDER_CHOICES)

So here we have created 3 field Name that is text ,Age that is Integer and Gender which is choice field we have created the choices in a constants file in the same directory this is done to make your code much cleaner and readable and the max_length basically specify its length that will be stored in DB.The constants.py file is defined below

GENDER_CHOICES = (
    ("M", "MALE"),
    ("F", "FEMALE"),
    ("O", "OTHER")
)

Once you have made these changes run the following commands

python manage.py makemigrations
python manage.py migrate

So the first command basically tells the database management system to create the database and finally we migrate i.e save it once we do that you can go and check the migrations folder with 0001_intial.py file it basically has our first migration and as we do more changes more files will be created and we can also go back to a certain migration if there was mistake just like github we can go back to an older version.

So this is all for this tutorial. Will be talking more about models and database in the next part of the tutorial
Until then

Django with Rest Tutorial -1

Siddharth — Thu, 22 Dec 2022 18:06:05 +0000

Hey everyone so this is my first blog on Dev. So I was learning Django and came across Django rest Frame work. I know that there are aloooooooooot of tutorials on Django Rest on YouTube but no one talks much in depth about the whole concept of serialization and desrialization and in this tutorial series i will be taking about it from scratch and will be creating small applications to make you understand Django Rest Frame work in easy way.
So lets get started

Pre-requisites:

Python(Basics)

Tools & Dependcies:

VsCode
Postman
Chrome(or any other latest browser)
Python 3 or greater
Django
Django rest-framework

Installation:

pip install django
pip install django
pip install djangorestframework

Creating a web-server/django app

So if you are coming from a background of nodejs where we create webserver.The same is achieved here by creating an app/project.The simple command to create a django project is:
django-admin startproject <projectname>
Once you run this command a project will be created.But before you do so i prefer to create a seprate virtual environment using venv package for your web applications so at anytime you want to host your appication you can easily get its dependecies.

So let's start with basic theory first.
What is an API?
API is basically like an waiter that takes order from the frontend and requests the data from backend and delivers back to the frontend.

Ps:You don't need to tip the api.
So moving on to the smaller question What is a rest api Basically REST is a set of architectural constraints that is used in developing an api that will be taking more about as we start to develop our api.

Creating our application

Run the following commands

 django makeproject harrypotter
 django startapp grffyindorr

So first command creates our project called harrypoter and then we have made an app grffyindoor.
So what is an app basically app helps in managing things separately when we work in a team for particular component
For eg: Instagram has feature of chat so we can create an app in insta as chat and all the people working on it can edit that app without worrying about other things

Now lets move to installed apps an
Add 'rest_framework, grffyindorr' to your INSTALLED_APPS setting.

INSTALLED_APPS = [
    ...
    'rest_framework',
    'grffyindorr',
]

Now lets start our app

django manage.py runserver

You will get this output

System check identified no issues (0 silenced).
December 22, 2022 - 17:16:09
Django version 4.1.3, using settings 'rest.settings'
Starting development server at http://127.0.0.1:8000/

Open that url on your server and you will see a rocket launching on it
Congragulations 🎉🎉 You have succesfully created an application in django and ran it

Stay tuned for next part where will create our database and start writing our first api