DEV Community: Mazen Ramadan

Distributing Task Queuing using Django & Celery

Mazen Ramadan — Mon, 24 Apr 2023 21:08:44 +0000

Have you ever had a backend system that needed to perform time-consuming tasks, such as sending emails, processing data, or generating reports?
These tasks can take a long time to complete and tie up server resources, which can lead to slow response times and a poor user experience.

This is where task queuing comes in. Task queuing is the practice of offloading time-consuming tasks to a separate system, where they can be executed asynchronously in the background. This frees up server resources and allows your web application or backend system to continue responding quickly to user requests.

Celery is a popular Python library for task queuing that makes it easy to set up and manage a distributed task queue. It provides a simple yet powerful way to manage the execution of asynchronous tasks and can integrate with a wide variety of other Python libraries and frameworks.

In this article, we'll discuss what Celery is and write a simple task queue system using Celery & Django.

Let's jump into it!

What is Task Queuing

Task queuing is a concept that has been around for many years, and it has become an important technique for managing large-scale distributed systems. In a distributed system, there are many tasks that need to be performed, and these tasks can be time-consuming and require a lot of resources. By using task queuing, tasks can be distributed across multiple workers or servers, allowing them to be executed concurrently and efficiently.

One of the benefits of using a task queuing system is that it can improve the performance and scalability of your system. For example, if you have a web application that needs to generate a report for each user, you can use a task queue to distribute the report generation tasks across multiple workers. This can significantly reduce the time it takes to generate reports and make your application more responsive.

Task queuing also provides a way to handle errors and retries. If a task fails, it can be retried automatically, ensuring that it eventually completes successfully. This can be particularly useful when working with unreliable resources or when performing complex calculations that may fail due to errors or resource constraints.

What is Celery & How it Works

Celery is a popular Python-based task queuing system that has been around since 2009. It is designed to be easy to use, flexible, and scalable, making it a popular choice for both small and large-scale applications.

Celery works by using a combination of a message broker and a worker pool. The message broker is responsible for storing the tasks and messages that are sent between the workers, while the worker pool is responsible for executing the tasks.

When you define a task in Celery, you create a Python function and decorate it with the @celery.task decorator. When this function is called, Celery adds the task to the message broker, which can then be picked up by a worker and executed. Celery supports a variety of message brokers, including RabbitMQ, Redis, and Amazon SQS, allowing you to choose the one that best suits your needs.

Celery also provides a variety of features for monitoring and managing your tasks. For example, you can configure the maximum number of retries for a task, set a time limit for how long a task can run, and monitor the progress of your tasks using Celery's built-in monitoring tools or third-party tools like Flower.

By using Celery, you can greatly improve the performance and scalability of your Python applications and ensure that time-consuming tasks are executed efficiently and reliably in the background.

Hands-on Task Queuing using Django & Celery

In section, we'll create a simple task queue using Django, Celery & RabbitMQ. Note that we need RabbitMQ as a broker to send tasks from Django to Celery.

If you don't have RabbitMQ installed you can install from here and make sure that that RabbitMQ service is running.

First, create an empty folder, create a virtual env named venv and activate it:
python -m venv venv venv\Scripts\activate
Note that you might need another command to activate the venv if you are using Mac.

Next, install Django & Celery:
pip install django celery
After that, create a new Django project:
django-admin startproject main .
In the main directory of the main project create a file named
celery.py and add the following code:

main/celery.py



import os
from __future__ import absolute_import, unicode_literals
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')

app = Celery('main')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

The previous code sets up a Celery instance with configurations defined in Django settings file and enables Celery to discover and execute tasks defined in different Django apps:

Imports the os module which provides a way of using operating system dependent functionality like reading or writing to the file system.
Imports tools that allows importing of unicode literals in the code.
Imports the Celery module.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings') This sets the DJANGO_SETTINGS_MODULE environment variable to main.settings, which is the location of the Django settings file.
Creates an instance of the Celery object.
Loads the Celery configuration from the Django settings file, specified by the argument. This is done to make sure that Celery settings are separate from other Django settings.
Discovers and imports all of the tasks defined in the tasks.py files within each installed Django app. It enables the Celery app to find and execute tasks defined in different Django apps.

Now let's create a new Django app inside the main project:
python manage.py startapp app1

Before we forget, let's register this app into the main Django project:

main/settings.py



INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'app1'
]

Inside the app directory, let's create a python file tasks.py, which contains the tasks:

app1/tasks.py



from __future__ import absolute_import, unicode_literals
from celery import shared_task

@shared_task
def add(x, y):
    return x + y

Imports tools that allows importing of unicode literals in the code.
Imports the shared_task decorator from the Celery module, which is used to create a task that can be executed asynchronously using Celery.
Creates a simple task function which adds 2 values. We also use the @shared_task decorator to wrap this function and turn it into an async Celery task.

Finally, let's run the celery queue:
Windows: celery -A proj worker -l info --pool=solo
Mac/Linux: celery -A main worker --loglevel=info

You will see something similar to this:

Now let's trigger the task from the Django shell, open a new terminal and add the following code:

from app1.tasks import add add.delay(4, 4) add.apply_async((3, 3), countdown=20)

We import the add function which will trigger task to Celery via RabbitMQ.
We use the function to add 2 values.
We use the same function but we wait 20 seconds before we deliver the message to Celery.

If we take a look RabbitMQ interface we'll see that it had received the 2 messages:

Here is out result:

You can find the code used here.

Conclusion

In summary, Celery is a powerful and flexible task queuing system for Python that can help you manage and distribute tasks across multiple workers or servers. By using Celery, you can improve the performance and scalability of your applications, handle errors and retries, and monitor and manage your tasks with ease.

How to implement a Load Balancer Using Nginx & Docker

Mazen Ramadan — Thu, 06 Apr 2023 21:32:36 +0000

Scaling becomes a necessary part for your system when your system grows in popularity.
There are two types of scaling:
Vertical Scaling - Adding more resources (CPU, RAM, storage) to your single server.
Horizontal Scaling - Spinning up more servers and distribute the load between them, which is called load balancing.

In this article, we will talk about load balancing, what it is and how to implement a load balancer using Nginx and Docker.
Let's get started!

What is Load Balancing ?

Load balancing is the process of distributing incoming network traffic across multiple servers, applications, or network resources to ensure that no single resource is overwhelmed with too much traffic. The goal of load balancing is to optimize resource usage, maximize throughput, minimize response time, and ensure high availability and reliability of the system.

In a typical load balancing scenario, incoming requests are first directed to a load balancer, which acts as a front-end for the system. The load balancer then forwards the requests to the appropriate server, application, or resource based on a set of predefined rules, policies, or algorithms. This ensures that all resources are used efficiently, and no single resource is overburdened. Load balancers are effective at:

Preventing requests from going to unhealthy servers.
Preventing overloading resources.
Helping to eliminate a single point of failure.

There are several different types of load balancing that can be used, depending on the specific requirements and constraints of the system:

1. Round-robin

This is the simplest and most common type of load balancing, where incoming requests are distributed to servers in a cyclical order. Each server is given an equal chance to process a request, regardless of its current load or performance.



servers = [server1, server2, server3]
next_server_index = 0

def handle_request(request):
    server = servers[next_server_index]
    next_server_index = (next_server_index + 1) % len(servers)
    server.process_request(request)

2. Weighted round-robin

This is a variant of round-robin load balancing, where each server is assigned a weight based on its processing capacity, and requests are distributed accordingly. Servers with higher weights are given a higher proportion of incoming traffic.



servers = [{'server': server1, 'weight': 2},
           {'server': server2, 'weight': 3},
           {'server': server3, 'weight': 1}]
next_server_index = 0

def handle_request(request):
    server_info = servers[next_server_index]
    server = server_info['server']
    server.process_request(request)
    server_info['weight'] -= 1
    if server_info['weight'] == 0:
        next_server_index = (next_server_index + 1) % len(servers)
        for info in servers:
            info['weight'] = info['weight'] + 1

3. Least connections

In this type of load balancing, incoming requests are sent to the server with the fewest active connections, in order to avoid overloading any one server. This is particularly useful for applications that involve long-lived connections, such as streaming media or gaming.



servers = [server1, server2, server3]

def handle_request(request):
    min_connections = float('inf')
    min_server = None
    for server in servers:
        if server.connections < min_connections:
            min_connections = server.connections
            min_server = server
    min_server.process_request(request)

4. IP hash load balancing

This method uses the IP address of the client to determine which server to send the request to. The IP address is hashed, and the resulting value is used to select the server. This ensures that requests from the same client are always sent to the same server, which can be useful for maintaining session state or other application-specific requirements.



servers = [server1, server2, server3]

def handle_request(request):
    client_ip = request.get_client_ip()
    hashed_ip = hash(client_ip)
    server_index = hashed_ip % len(servers)
    server = servers[server_index]
    server.process_request(request)

5. Layer 4 load balancing

This type of load balancing operates at the transport layer (TCP/UDP) of the network stack and uses information such as source IP, destination IP, source port, and destination port to distribute traffic across servers.



servers = [server1, server2, server3]

def handle_request(request):
    dest_ip = request.get_dest_ip()
    dest_port = request.get_dest_port()
    server_index = hash(dest_ip + ':' + dest_port) % len(servers)
    server = servers[server_index]
    server.process_request(request)

6. Layer 7 load balancing

This type of load balancing operates at the application layer of the network stack and uses information such as HTTP headers, cookies, and URL paths to distribute traffic across servers. This allows for more intelligent routing based on application-specific criteria, such as session affinity, content-based routing, or SSL offloading.



servers = [{'server': server1, 'route': '/api/*'},
           {'server': server2, 'route': '/auth/*'},
           {'server': server3, 'route': '/*'}]

def handle_request(request):
    for info in servers:
        if request.matches_route(info['route']):
            server = info['server']
            server.process_request(request)
            break

After we know about load balancing, let's now to move the practical guide.

Implement Nginx Load Balancer

Let's consider having 3 Python servers each one is deployed to a container. After that we will use Nginx as a load balancer for the 3 servers.

Here is our files structure:



nginx-load-balancer
    |
    |---app1
    |     |-- app1.py
    |     |-- Dockerfile
    |     |-- requirements.txt
    |
    |---app2
    |     |-- app2.py
    |     |-- Dockerfile
    |     |-- requirements.txt
    |
    |---nginx
    |     |-- nginx.conf
    |     |-- Dockerfile
    |
    |------ docker-compose.yml

We have 2 basic Flask servers which returns a text to clarify which server we are connected to:

app1/app1.py



from flask import request, Flask

app1 = Flask(__name__)


@app1.route('/')
def hello_world():
    return '<h1>Hello from server 1</h2>'


if __name__ == '__main__':
   app1.run(debug=True, host='0.0.0.0')

Each Flask server is deployed to a docker container:

app1/Dockerfile



FROM python:3

COPY ./requirements.txt /requirements.txt

WORKDIR /

RUN pip install -r requirements.txt

COPY . /

ENTRYPOINT [ "python3" ]

CMD [ "app1.py" ]

In the previous code, we tell docker to install Python3 image and copy the requirements file. After that, we change work directory of docker and install dependencies. Finally, we copy the code and run the server.

We will do the same for Nginx:

nginx/nginx.conf



upstream loadbalancer {
    server 172.17.0.1:5001 weight=5;
    server 172.17.0.1:5002 weight=5;
}

server {
    location / {
        proxy_pass http://loadbalancer;
    }
}

Here we use round-robin balancing algorithm. The routing criteria is specified using the weight parameter. In this case, we want to balance the load equally for both servers.

After that, let's dockerize Nginx configruation. It will copy the above conf file on the related path inside the container when starting it.

nginx/Dockerfile



FROM nginx
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf

Finally let's create a docker-compose file, it will be a wrapper that spins up the whole architecture:

docker-compose.yml



version: '3'
services:
  app1:
    build: ./app1
    ports:
    - "5001:5000"
  app2:
    build: ./app2
    ports:
    - "5002:5000"
  nginx:
    build: ./nginx 
    ports:
    - "8080:80"
    depends_on:
      - app1
      - app2

Here is what this file do:

It will build images for app1, app2, Nginx based on our Dockerfiles and then spin up containers from those images.
The opened port inside app1 and app2 containers are 5000 (default port used by flask), these ports will be mapped to 5001 and 5002.
The load balancer will route traffic to the appropriate application based on that port.
The load balancer (Nginx) will expose his internal 80 port to 8080, so we can access the application from http://localhost:8080

Finally, you can spin up the architecture using this command from the base directory:



docker-compose up

Requests well be routed to both servers equally:

You can find the code used here.

What is Redis and How To Implement Redis Cache in Node JS.

Mazen Ramadan — Sat, 18 Feb 2023 21:10:55 +0000

With the development of the internet, websites now have tons of data. This is why we need a solution to reduce latency and improve performance.

In this article, we will discuss what is Redis, what is used for and how to implement a simple caching solution in NodeJS.
Let's get started!

What is Redis?

Redis (Remote Dictionary Server) is an open-source in-memory database which used primally for caching. It is a NoSQL key-value data store, the keys can store strings, lists, sets and bitmaps.

Unlike normal databases which stores data in HDDs. Redis stores data in memory, which makes read and write operations very fast. This leads to a better speed, reliability and performance for applications.

So since redis store data in memory, do we lose this data ?

To prevent data loss from happening, there is a built-in persistence module which writes the in-memory state to a dump file on the disk. These dump files are loaded when the system is starting; once it's up and running, the data is available again.

What is redis used for ?

Caching

Redis is commonly used as a cache to store frequently accessed data so that applications can be responsive to users. Redis provides a variety of caching patterns. It includes the capacity to set how long you want to keep data and which data to evict first.

The typical workflow for this case is when a user tries to a access data, the application searches for this data in the cache. If the data is there, it simply returns it. If not, redis will fetch this data from the database and return it back to the cache.

Primary Database

If the data size and risk profile are well known and match with Redis’ disk model, it can serve as a primary database.

A problem you might encounter while using redis as a primary database is running out of memory. The simple solutions for this are clustering and sharding.

Clustering means that you have a primary or master Redis instance, which can be used to read and write data. You can also have multiple replicas of that primary instance for reading the data.

Sharding means that you split your data into smaller chunks, where each shard is responsible for its own subset of data.

Queues

Redis can be used as a message queue that enables applications and services to communicate and exchange data with each other. It can validate, store, route and deliver messages to it's destination without knowing the details about the receiver.

In a nutshell, message queues works under a Pub/Sub model.
The publisher is responsible for publishing messages to the queue.
The Subscriber is the endpoint that waits the delivery of messages.

Lua Scripting

Redis lets users upload and execute Lua scripts on the server.
This allows users to add features to Redis themselves in the form of fast executing scripts.

Redis also supports storing various data structures like streams, geospatial, hyperloglog and bitmaps which can be used for various use-cases.

How To implement Redis Cache in Node.JS

In this section, we will use redis with nodejs. The idea is simple, we create an API that catches the number of public repositories of a GitHub user. Then, we will cache this result using redis.

Let's start by installing the required NPM packages:

npm install express node-fetch redis

If you haven't install redis yet, you can install it from here.

Create an empty JavaScript file and add the following code:

const express = require('express');
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
const redis = require('redis');

const PORT = 8000;
const REDIS_PORT = 6379;

const client = redis.createClient({
    legacyMode: true,
    PORT: REDIS_PORT
});

client
    .connect()
    .then(async () => {
        console.log("You are connected")
})
    .catch((err) => {
        console.log("error happened", err)
});

const app = express();

We require the modules we will use and redis port and server port. Next, create a client and connect to that client.
Note that you need redis server to be running before create a redis client.
The last step is to create an express app.

Now, let's a function that gets the total number of repositories.

// send a request to Github for data
getRepos = async (req, res, next) => {
  try {
    console.log('Fetching Data...');
    const { username } = req.params;
    const response = await fetch(`https://api.github.com/users/${username}`);
    const data = await response.json();
    const repos = data.public_repos;
    // Set data to Redis
    await client.setex(username, 3600, repos);
    res.json({username, repos});
  } catch (err) {
    console.error(err);
    res.status(500);
  };
};

In this function, we get the username from the requests parameters and send a request to this user profile using node-fetch.

After that, set the (key, data expirations in seconds, value) of the client using the .setex function. Finally return the response in json.

Now let's create a cache middleware that returns the cached data.

cache = async (req, res, next) => {
  const { username } = req.params;
  client.get(username, (err, repos) => {
    if (err) throw err;
    if (repos !== null) {
      res.json({data:"cached data", username, repos});
    } else {
      next();
    };
  });
};

Like we did before, we get the username from the parameters. Then redis searches for keys that have same username.
If redis finds data it will return it, otherwise, the middleware will execute what comes next which is the previous function.

The last step is to register create a route in start the server.

app.get('/repos/:username', cache, getRepos);

app.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
});

Let's test this out!

As you can see from the image, we sent a request and it took about 372 ms to fetch the data.

Now let's send the same request again to get the cached data.

This was fast!
It took only 5 ms to get the cached data from memory.

Conclusion

In this article we talked redis and what it is used for, We also went through an implementation of redis in nodejs.

In a nutshell redis a NoSQL key/value store, which can be used a cache database. Redis also can be used as a message queue and a primary database.

If you find this article valuable, consider sharing it with other people!

TCP vs UDP Protocols

Mazen Ramadan — Thu, 09 Feb 2023 18:53:33 +0000

TCP and UDP are the most used transport protocols. They enable accessing websites resources and data transmissions, but which one should you use ?
In this guide, we will compare TCP vs UDP and how they differ from each other. Let’s dive in!

What are Internet Protocols?

Before we discuss TCP and UDP, we need to understand what an internet protocol is. The Internet Protocol is a set of rules for routing and addressing parts of data so that it can travel across networks.

Before data is sent across a network, it is divided into smaller pieces called Packets. Every device that connects to the internet is assigned with an IP address, which packets are directed to. Once packets arrive at the device, they are handled differently depending on which transport protocol is used with the IP. The most common transport protocols are TCP and UDP.

What is TCP?

TCP is a Transmission Layer Protocol, it provides a reliable connection and protected data transmission between the connected devices. It first established a secure link before sending data.

While TCP is slow in data transmission, it has more functionalities including flow and error control. As TCP is reliable, it is used with web protocols like HTTP and FTTP to secure data over the network.

The TCP protocol is connection-based, it means that it creates and maintains connection between sender and receiver. This process results in a more significant overhead size, which means that it consumes more bandwidth.

Now let’s discuss the pros and cons of TCP protocol.

Pros

Acknowledgment Sometimes between data transmission, data may get lost failing to reach its destination. Due to this acknowledgment, the sender can re-transmit this data again.

Guaranteed Delivery If the server didn’t receive the data or the data was corrupted during transmission. The sender keeps retrying until data is delivered.

Congestion Control TCP uses a separate congestion control policy, this happens when the sender sends too many data packets in a given time.

Unique Identification Each device connected has a unique IP address, making it identifiable over the network.

Cons

Slow Start As TCP starts by establishing a secure connection between two devices, it’s always slow at the beginning.

Slow Handshake When a connection is established, TCP performs a three-way handshake. This leads the connection establishment to slow down.

More Bandwidth As TCP has many features, it adds more packets to enable these features. This leads to more bandwidth consumption.

What is UDP?

UDP is User Datagram Protocol, it’s connectionless meaning that it doesn’t establish a connection between devices like TCP. UDP is used for situations where speed is a critical factor like video streaming or online games.

UDP doesn’t guarantee the transfer of all data. This is why it’s called “fire-and-forget” protocol. The UDP header has fixed size, this means that it’s unreliable but fast.

Now let’s discuss the pros and cons of UDP.

Pros

Less Bandwidth UDP uses a small packet size, this means less bandwidth and memory consumption.

Fast UDP is fast but it doesn’t guarantee the delivery of all packets.

Cons

Unreliable As UDP doesn’t establish a connection between devices, connection can be unreliable.

Unguaranteed Delivery Packets may not be delivered or delivered twice.

No Congestion Control Since UDP transmits a large amount of data, it can cause congestion without controlling it.

TCP vs UDP

After we talked about TCP and UDP, let’s compare TCP vs UDP to know which is better for your case.

TCP is a connection-oriented protocol, it relies on a passive open state. This means that both devices need to accept connection between each other.
Comparatively, UDP is a connectionless protocol. This means that data transmission involves an endpoint without waiting the other side to accept a connection

TCP sends data in a particular sequence, This allows the system to track the order in which data is transmitted.
On the other hand, UDP doesn’t follow a sequence, data packets are sent independently with no fixed order

Since TCP needs to establish a connection, it’s slow compared to UDP which is connectionless.

TCP adds more resources to a response which makes it consume more bandwidth than UDP.

Another difference between TCP and UDP is that UDP doesn’t guarantee delivery of data.

Conclusion

In this guide we discussed:

What TCP and UDP are.
The pros and cons of TCP and UDP.
TCP vs UDP and which is better for your case.

In a nutshell, TCP is more reliable and secure than UDP but it’s slower than UDP.

TCP is better where reliability and security are important like chatting applications.

UDP is better where speed is more important like streaming and online games.

Request Response vs Message Queues vs Publish Subscribe Patterns

Mazen Ramadan — Thu, 13 Oct 2022 22:15:50 +0000

In this post I will discuss the difference between every pattern between Request Response , Message Queues and Publish Subscribe Patterns. I will also discuss the pros and cons and when to use each of them.

Request Response

Request–response or request–reply is one of the basic methods computers use to communicate with each other in a network, in which the first computer sends a request for some data and the second responds to the request. More specifically, it is a message exchange pattern in which a requestor sends a request message to a replier system, which receives and processes the request, ultimately returning a message in response. It is analogous to a telephone call, in which the caller must wait for the recipient to pick up before anything can be discussed. This is a simple but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel; it is especially common in client–server architectures.

This pattern is typically implemented in a purely synchronous fashion, as in web service calls over HTTP, which holds a connection open and waits until the response is delivered or the timeout period expires. However, request–response may also be implemented asynchronously, with a response being returned at some unknown later time.

Pros

1. Stateless (HTTP)
This means that every HTTP request the server receives is independent and does not relate to requests that came prior to it.

2. Scalable
You can scale it horizontally where you can duplicate the receiver or use a load-balancer.

3. Latency
Only when the connection is established, the handshaking process will take place in HTTP. Hence, there will be no handshaking procedure following a request. This significantly reduces latency in the connection.

Cons

1. Administrative Overhead
It is bad for multiple receivers as it causes administrative overhead in the connection.

2. Server Availability
Even if HTTP receives all the data that it needs, clients does not take measures to close the connection. Therefore, during this time period, server will not be present.

3. Integrity
Integrity is not there, so someone can easily alter with the content. HTTP is insecure as there's no encryption methods for it.

Message Queues

A message queue is a form of asynchronous service-to-service communication used in serverless and microservices architectures. Messages are stored on the queue until they are processed and deleted. Each message is processed only once, by a single consumer. Message queues can be used to decouple heavyweight processing, to buffer or batch work, and to smooth spiky workloads.

In modern cloud architecture, applications are decoupled into smaller, independent building blocks that are easier to develop, deploy and maintain. Message queues provide communication and coordination for these distributed applications. Message queues can significantly simplify coding of decoupled applications, while improving performance, reliability and scalability. improving performance, reliability and scalability.
Some of the most popular open-source message queues are Apache
Kafka, RabbitMQ and Celery.

pros

1. Decoupling
If system A sends a message to B, C and other systems, it can send the message through the message queue, and B, C and other systems can get it when used.

2. Asynchronous
System A returns a response after sending a message, waits until B, C and other systems return business data, and then updates the return response, without always displaying the waiting state. Significantly enhance the user experience.

3. Peak cut
When a large number of business needs flood in, direct operations will have a great impact on the database. In order to avoid database downtime, by introducing a message queue, the data to be processed is stored in the message queue until the demand peaks Re-execute the zone, effectively avoiding database downtime and causing system crash.

Cons

1. Increases system complexity
After the introduction of message queues, it is necessary to ensure that there are no issues such as repeated calls and sequence of message delivery.

2. Consistency problem
After the A system is processed, it returns successfully. Everyone thinks that your request is successful; but the problem is that if there are three BCD systems, the two systems BD successfully write the library, and the C system fails to write the library. , Resulting in inconsistencies between the data in the data database and the ideal data.

Pub/Sub Messaging

Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all of the subscribers to the topic. Pub/sub messaging can be used to enable event-driven architectures, or to decouple applications in order to increase performance, reliability and scalability.

The Publish Subscribe model allows messages to be broadcast to different parts of a system asynchronously. A sibling to a message queue, a message topic provides a lightweight mechanism to broadcast asynchronous event notifications, and endpoints that allow software components to connect to the topic in order to send and receive those messages.

To broadcast a message, a component called a publisher simply pushes a message to the topic. Unlike message queues, which batch messages until they are retrieved, message topics transfer messages with no or very little queuing, and push them out immediately to all subscribers. All components that subscribe to the topic will receive every message that is broadcast, unless a message filtering policy is set by the subscriber.

Pros

1. Decouple and Scale Independently
Pub/Sub makes software more flexible. Publishers and subscribers are decoupled and work independently from each other, which allows you to develop and scale them independently.

2. Simplify Communication
Communications and integration code is some of the hardest code to write. The Publish Subscribe model reduces complexity by removing all the point-to-point connections with a single connection to a message topic.

3. Scales with Multiple receivers
you can add as many different receivers as you like and scale them smoothly.

Cons

1. Message delivery issues
The broker in a pub/sub system may be designed to deliver messages for a specified time, but then stop attempting delivery, whether or not it has received confirmation of successful receipt of the message by all subscribers. A pub/sub system designed in this way cannot guarantee delivery of messages to any applications that might require such assured delivery.

2. Well-defined Policy
Requires a well-defined policy for message formatting and message exchange; otherwise, message consumption can become mangled and error-prone.

Different methods for handling errors in Express

Mazen Ramadan — Mon, 15 Aug 2022 18:14:38 +0000

It gets confusing sometimes when you have to deal with errors in different methods. In this post I will discuss different methods for handling errors in Express.js and when to use each of them.

Why do we need to handle errors anyway ?

Not all of users know that might be something wrong with their inputs, they might just think that the service is down.
It lets the user know, in a relatively friendly manner, that something has gone wrong and that they should change how they behave.
it allows the programmer to put in some niceties to aid in the debugging of issues.
Some errors leads the server to crash down.

Now I will use the same function in every method so the difference between each will be clear.

The function is simple, it is an API that POST data into the database.

router.post('/article', async (req, res) => {
  title = req.body.title
  content = req.body.content
  const newPost = await new Post({
    title: title,
    content: content
  })
  newPost.save()
  res.status(201).json(newPost)
})

Now this code works great ONLY if the user used it in a correct way, if he didn't provide any content or title the server will crash. of course we don't need this!

1- Using Try-Cath statement

This is the most popular one and you probably figured it out

router.post('/article', async (req, res) => {
    title = req.body.title
    content = req.body.content
    try {
        const newPost = await new Post({
            title: title,
            content: content
          })
          newPost.save()
          res.status(201).json(newPost)
    } catch (err) {
        console.log(err)
        res.status(400).json({"Something went wrong":err.message})
    }
})

In this method we TRY to do whatever the function should do, if there any errors the CATCH statment will catch it then we will return it to the user with the status code of 400 "bad request" and the message of the error so he can know what he had done wrong.

2- Using promises in functions

Most of functions in express are promises that takes arguments like the .save() the function we use here.
We can get the error if any by getting the error in a param like this .save((err)) Here we catch the error and we can make use it like this.

router.post('/article', async (req, res) => {
    title = req.body.title
    content = req.body.content
    const newPost = new Post({
        title: req.body.title,
        content: req.body.content
    })
    newPost.save((err) => {
        if (!err) {
            res.status(201).json(newPost)
        } else {
            console.log(err)
            res.status(400).json({
            "Something went wrong":err.message})
        }
    })
})

Here we check if we don't have any errors with (!err) if we have any we handle it like before.

3- Using promises

router.post('/article', async (req, res) => {
    title = req.body.title
    content = req.body.content
    const newPost = new Post({
        title: title,
        content
    })
    newPost.save()
    .then (result => {
        res.status(201).json(newPost)
    })
    .catch(err => {
        console.log(err)
        res.status(400).json({"Something went wrong":err.message})        
    })
})

This is so similar to try-catch but here we are using promises.

4- Using custom methods

In this method we check or errors that we know that it could happen often so we handle it in a very custom way example of this is when you try get a an article and you know that user might try to get a article that doesn't exist so you handle this error specifically. In this example we will we assume that the user doesn't provide a content or a title in the body of the request.

router.post('/article', async (req, res) => {
    title = req.body.title
    content = req.body.content
    if (!content || !title) {
        res.status(400).json({
        "error":"You must provide content and title"})
    }
    // Rest of the code with any method we used
})

In this example we check if the user didn't provide content or title with the if statement (!content || !title) this will tell the user that they need to provide content and title.

Conclusion

You can use any method of there is no big difference it's just personal prefare.

You can visit the docs of error handling in express docs page for more info.
If you liked the article you can support me with a like.