In an ever changing world of software development,utilizing microservices and distributed systems is the de facto standard for architecting software solutions, there is always a need to employ efficient means of message transfer between these various microservices.
One of the fundamental aspect of developing these services infact, is to incorporate an Inter-service communication between them. And although there are numerous methods of approaching this; Message Brokers provide an efficient, scalable and cheaper means of data sharing betwixt these services.
In this article we are briefly going to uncover the basics of message brokers, and build a distributed alert system that publishes alert messages to listed email addresses and a twitter account; using fastapi golang/gin and rabbitmQ.
Message Brokers
A broker in layman's term is someone who negotiates/arranges a transaction between two entities (be it people, businesses e.t.c) similarly; Message brokers are an inter-service communication technology, that utilizes standard messaging protocols to promote rapid development, and data-sharing between distributed applications. Message brokers provide data-sharing, data-marshaling, and data persistences features to traditional middleware applications out of the box, and they as well guarantee the transfer of messages with low latency. They ensure delivery of messages even when a consumer(client) is inactive, this enhances reliability/uptime of application's flow since each service is independent. A typical case for example is a service responsible for sending emails to recently signed up users, now assuming the service went down at the time a user is being registered, the downtime doesn't interupt the user's experience since the native backend only has to publish to the sign-up queue, which the consuming application can process from whenever it is restored.
Publishers, Consumers and Queues
Message brokers primarily rely on three fundamental concepts for message transfer, they are Publishers, Consumers and Queues and although they might differ in name and behaviour pending on the message brokers you use, they can still be used interchangably(or not) with Producers, Subscribers and Channels; So lets have a brief walkthrough of each of these concepts.
Publishers: In messaging, publishers (or producers) represents an application that sends/publishes messages to a message broker instance. Depending on the protocol, the publisher sends the message to a broker which further routes the message to the approriate queue/channel.
Consumers: consumers (or subscribers) represents application that readily listens on a queue or channel for new messages from the publisher. In order to consume messages, a consumer has to be registered to a queue.
Queues: queues as we know them are data structures that provides two kind of operations; enqueue whereby data is enterred into the queue through the rear or tail, and dequeue whereby data is removed from the queue through the front or head. In message brokers, queues are referenced via their names.
Data Sharing
Since the essence of this article revolves around data tranfer, i'd quickly like to deviate your attention to commonly employed design practices when defining how applications should communicate with each other.
In the process of building a distributed system where message sharing is priority, you should be able to answer the following questions:-
- What data does your services need to share?
- What service(s) are the primary providers of the data, can they be identified?
- Does the nature of the data imply the introduction of a third-party software to share it?
Answering these questions are quintessential to the development of solid distributed ecosystems. When we start to write the codes for this article, we'd incoporate this services and see if our solution answer these questions.
When and Why Should We Use Message Brokers?
Unlike the typical client-server paradigm, where a client (eg. web browser) sends a requests to a server, and blocks till it gets a response back from the server, message brokers allows a client to continue with other operations, whilst it processes the former requests. This is incredible because it saves time, and enhances the user's experience; Imagine a scenerio where you have to successfully send a user their registration email, before redirecting them to their profile or the login page, the time spent waiting on the email service would imply a very poor user experience.
Suffice to say it's best to use message brokers whenever you're building something whose average processing time spans beyond the traditional http requests/responses duration. Keep in mind time isn't the only yardstick to consider when trying out message brokers, but it is one to always have in mind.
Building an alert dispatch service
Up until this point we've seen ideal situations where we would want to build our applications using message brokers. This section aims at furthering your practical knowledge on the use of message brokers, and we are going to be building an Alert Dispatch Service, which is essentially an API that recieves a json post request:
the API then serializes this json and publishes it to a message broker (RabbitMq) which in turn broadcasts it to any consumer listening on the queue, the consuming service serializes the recieved data and publishes the message to a twitter account and a list of email addresses.
The Api is written in python fastapi framework and the consumer is written in golang, this example elaborates the use cases of message brokers as we can see how we've enabled communication between two independent microservices, written in entirely different languages; whether this service has a use case in real world scenerios isn't considered as i only thought of a perfect example for using message brokers, which would be straight to beginners.
Fastapi Service (Publisher)
As i mentioned earlier, the Api and publisher service is written with python's modern web api framework fastapi, it is quite easy to set up. First we need to set up our virtual environment, do this by running python -m venv <name_of_your_virtual_env> you can activate your virtual environment on windows by running
<name_of_venv>/bin/activate.bat or source <name_of_venv>/bin/activate on linux/macOs. Next we need to install fastapi by running:
pip install fastapi
this installs fastapi, alongside other dependencies it needs to function properly. Then we need to install uvicorn which will act as the ASGI (Asynchronous server gateway interface) server for our application; to do this run pip install uvicorn. Once you've completed this setups, we can move on to writing our codes;
From the code above,on line one and two we imported the FastAPI class from fastapi and also the BaseModel class from pydantic
we then instantiate the fastapi class with a variable called app on line 4, we will see the use of this app variable shortly.On line 8 we create an Alert class which is the schema of the json data our api expects, and within the class we declare two fields, topic which is the topic of the alert and message which is the alert's content. Notice how our alert class inherits the pydantic.BaseModel class.
Next we write our api function on line 14, the app variable defines the http method required by the function and also declares it's route, then the function basically checks if an http request body contain's an alert type, notice how all the Serialization is handled neatly by Fastapi behind the scenes; if the alert is valid we calls our rabbitmQ publisher function else, we throw an error at the client. The Publisher function:
For the publisher, we import two packages Pika which is a python library for communicating with rabbitmQ and json to enable us serialize the alert instance. Then we create a connection to our rabbitmQ instance, and declare a Queue named alert_queue, when you declare a queue, rabbitmQ checks to see if the queue already exists, if it doesn't it creates the queue else it just goes ahead to publish the message. Next we declare a dictionary, to map our Alert's value, this is to enable us send messages in json format to our message broker, which can be delivered and serialized by our consumer service. Finally we publish the message to the alert_queue and return from the function body.
Golang Service (Consumer)
The consumer service is written in go, to elaborate the essence of this article i.e using message brokers to transfer messages between different, independent microservices.
Unfortunately i won't be able to detail the code bit by bit since this article is becoming too lengthy, but it's quite straight-forward if you write Go regularly;
Basically this code, connects to our rabbitmQ using the streadway/amqp package which is the golang standard library for communicating with rabbitmQ, next we declare a unique channel to enable us pass across message to a specific queue. Then we call the Consume function which takes in a parameter of *amqp.Channel which is the channel we created previously.
So the Consume function basically reads all the messages currently available in the queue and serializes them to an alert variable, then it sends a tweet to a specified twitter account with the alert's message as the tweet's content, it also sends an email with the alert's topic as subject and the alert's message as the email's content to my email address. I'm obviously not gonna go into the details of how to programmatically create a tweet or send an email. But if you're interested in learning how to send Emails with golang, you should definitely take look at my article Sending E-mails with Go. All the sample codes written here are publicly available on this github repository with instructions to run them.
Conclusion
Being able to effectively exchange messages across multiple decoupled software components is vital for every distributed ecosystem; message brokers provides an easy, scalable and reliable means of achieving this.
In this article we've briefly covered the basics of publishing and consuming messages with rabbitmq, we went further to build an alert dispatch service that distributes alerts to listed email addresses and a twitter account, using python and golang.
Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!
Top comments (1)
Great article, thanks for sharing! Just a small note, if we are talking about MQTT brokers, Mosquitto MQTT broker might be one of the most popular brokers out there. Here is a bit of info on this one: cedalo.com/mqtt-broker-pro-mosquitto/