Jake Page for Glasskube

Posted on May 14 • Updated on May 16

RabbitMQ on Kubernetes

#devops #rabbitmq #kubernetes #opensource

RabbitMQ is an open-source message broker software that implements a handful of messaging protocols, originally the AMQP (Advanced Message Queuing Protocol), and also includes web-based ones such as STOMP (Simple Text Orientated Messaging Protocol), MQTT (Message Queuing Telemetry Transport), and WebSockets to decouple applications that share asynchronous data. RabbitMQ not only serves as an attractive messaging system choice due to its robustness and well-maintained open-source nature but also stands out for its ease of use and configuration. Before creating our first RabbitMQ instance and cluster, let's explore some fundamental concepts around messaging and check out some common use cases.

ℹ️ As of writing the latest RabbitMQ version is 3.13.2. See change log here.

Fundamentals

Sync vs Async messaging 📟

In synchronous messaging, the sender waits for a response before proceeding, which can lead to bottlenecks or inefficiencies. Asynchronous messaging, on the other hand, frees up the sender by allowing messages to be sent without an immediate response, breaking the synchronous relationship between apps or microservices is known as decoupling.

Messaging systems excel in scenarios requiring asynchronous communication. Decoupled applications that exchange high message volumes and depend on reliable delivery are prime candidates to use it. From distributed systems to microservices architectures and event-driven setups, messaging brokers like RabbitMQ facilitate seamless communication across multipe diverse technological architectures.

Producers and consumers 📲

In its simplest form, the messaging graph looks like this: Producer - Queue - Consumer

Messaging Brokers 🎛️

The Messaging broker is the core component of the messaging system, orchestrating the management of various messaging core entities, such as connections, channels, queues, and exchanges among others.

As the central hub, the broker encompasses all RabbitMQ features, ranging from clustering and mirroring to GUI and plugin management. It facilitates communication between clients and the broker itself, adhering to the open standard AMQP protocol for message transportation.

Other key components 🧩

A producer application establishes a connection with the messaging broker via a TCP connection. Within this connection, multiple channels can be created, serving to logically separate data flows. Each channel facilitates the transmission of data to a queue. These queues act as intermediary storage, from which consumer apps connect to consume messages from. If the Pub/Sub messaging pattern is used, an exchange might also be present.

RabbitMQ also introduces the concept of virtual hosts (vhosts) to partition and isolate messaging resources within the broker, enabling distinct environments or applications to operate independently within the same RabbitMQ instance, and enhancing security and manageability.

In scenarios demanding heightened fault tolerance and message reliability, messaging replicas come into play. These replicas, used in quorum queues, ensure data redundancy and resilience by maintaining synchronized copies of messages across multiple nodes or clusters.

Messaging patterns 🌐

Publish / Subscribe

The Pub/Sub pattern is used to broadcast messages to multiple subscribers who are interested in receiving notification updates. Here, instead of publishing a message to a single queue, messages are processed by an exchange which then routes the messages to multiple queues based on predetermined routing rules and bindings. Each queue is bound to the exchange with a routing key.

Worker queues

Worker queues are a way to distribute and parallelize the processing of tasks or messages across multiple workers or consumers. It can be simpler than the Pub/Sub pattern since it doesn’t have the added abstraction of an exchange agent. In this pattern, tasks or messages are published to a single queue. Multiple workers consume messages from this queue, with each worker processing tasks independently.

Oh by the way

If you are a fan of supporting Open Source projects working to make Kubernetes package management better for everyone then please consider supporting Glasskube, by giving us a Star on GitHub 🙏

Benefits of messaging systems 👍

Scaling

Queues can be added to expand capacity if needed. Even multiple broker nodes can be coupled to form messaging clusters.

Batching

Batching involves grouping multiple messages into a single batch or bundle before transmitting them over the network. This can positively impact performance, efficiency, and resource utilization.

Architecture decoupling

Applications can be developed and scaled independently, optimizing resource usage and interoperability.

Reliability and Persistence

By clustering, replicating and node federation, no node or storage volume is fully responsible for data persistence. No node need be a single point of failure.

Use cases

So which architectures or apps might benefit the most from messaging systems? Think of those that require asynchronous communication, scalability, reliability, and decoupling between components. Solutions like social media apps, financial services, IoT products, telecommunication apps, and supply chain management tools are just a few.

Messaging app example: 💬

For instance, every time you receive a message on WhatsApp or via text, the message payload is processed asynchronously using queues.

Financial app example: 📈

The same goes for when you use a financial app to put a buy order on your favorite stock. The buy details will be delivered from the front end of the producer app to the queue so that a backend consumer app can pick it up and continue processing your buy order.



       +-------------+
       |  Financial  |
       |    App      |
       +------+------+               +-----------------+
              |                      |                 |
              v                      |    RabbitMQ     |
  +-----------+-----------+          |    Broker       |
  |                       |          |                 |
  |  User Initiates Buy   +--------->+   Queue         |
  |      Order            |          |                 |
  |                       |          +-----------------+
  +-----------+-----------+                    |
              |                                |
              v                                |
       +-------------+                         |
       |  Consumer   |                         |
       |    App      |<------------------------+
       +-------------+

Features of RabbitMQ 🧰

Ease of use

Compared to other messaging systems, RabbitMQ’s simplicity of configuration and use stands out among the rest.

Delivery Acknowledgement and consumer confirms

Typical of synchronous communication but less present in async is a received acknowledgment when a message was successfully consumed. RabbitMQ has a feature that can be enabled to achieve this.

Distributed network

RabbitMQ as a messaging broker fits into distributed architectures, using producer and consumer apps connecting from remote hosts, this makes it desirable for the RabbitMQ broker to also be distributed there are three ways to accomplish this: clustering, federations, or the shovel plugin.

Tools + plugins

RabbitMQ has a series of tools and plugins to make managment and configuration much easier. Just as the management plugin which delivers a useful GUI from which mange management and monitoring tasks can be done as well as deployment plugins. Some useful plugins

rabbitmq_management: for management console access
rabbitmq_prometheus: exports metrics in Prometheus-compatible format, facilitating monitoring and observability.
rabbitmq_federation: enables cluster and exchange formation and data replication
rabbitmq_peer_discovery_k8s: used for automatic node discovery (cluster formation) in k8s environments
RabbitMQ comes with some useful CLI tools rabbitmqcli ,rabbitmq-plugins,rabbitmqadmin and rabbitmq-queues. To be executed inside the RabbitMQ nodes they enable cluster management, plugin configurations, and an array of customizations.

RabbitMQ Installation methods

Docker 🐳



# latest RabbitMQ 3.13
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management

ℹ️ Check out a great in-depth installation guide from Marcel Dempers using Docker

Open Source RabbitMQ Server on multiple operating systems

Linux: (Debian, RHEL, CentOS Stream, Fedora, Amazon Linux 2023)
Windows
MacOS

RabbitMQ Cluster Kubernetes Operator ☸️

We will focus on this installation method since it’s the recommended method for working with RabbitMQ in Kubernetes environments.

Install the RabbitMQ Cluster Kubernetes Operator

The RabbitMQ Cluster Kubernetes Operator automates the provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.

Installation

Installing the RabbitMQ cluster operator can easily be achieved with the Glasskube package manager.

Install Glasskube

If you already installed glasskube you can skip this step.
If not, glasskube can easily be installed the way you usually install packages for your operating system.

MacOs:



brew install glasskube/tap/glasskube

Linux (Ubuntu/Debian)



curl -LO https://releases.dl.glasskube.dev/glasskube_v0.4.0_amd64.deb
sudo dpkg -i glasskube_v0.4.0_amd64.deb

More installation guides here

Install the RabbitMQ cluster operator

Start the UI via the command line:



glasskube serve

Install RabbitMQ cluster operator via the Glasskube UI.
A package can be installed with a simple command.



glasskube install rabbitmq-operator

The process will wait until the package got successfully installed.

You will need access to a Kubernetes cluster running Kubernetes 1.19 or later (You can easily create a local cluster by using Minikube) kubectl isn’t strictly speaking a dependency for installing packages via Glasskube, but it is the recommended way to interact with the cluster. Therefore, it is highly recommended. Installation instructions are available for macOS, Linux, and Windows.

ℹ️ Glasskube will automatically install the cluster operator in a newly created namespace called rabbitmq-system

Confirm Service Availability

Before configuring your app to use RabbitMQ Cluster Kubernetes Operator, ensure that RabbitmqCluster Custom Resource is deployed to your Kubernetes cluster and is available.

To confirm this availability, run



kubectl get customresourcedefinitions.apiextensions.k8s.io

Create a RabbitMQ Instance

The RabbitMQ Cluster Kubernetes Operator creates the necessary resources, such as Services, StatefulSets, Secrets and ConfigMaps in the same namespace in which the RabbitmqCluster was defined.

First, create a YAML file to define a RabbitmqCluster resource named definition.yaml.

ℹ️ Note: The YAML file can have any name, but the steps that follow assume it is called definition.yaml.

Then copy and paste the below snippet into the file and save it:



apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: definition

Next, apply the definition by running:



kubectl apply -f definition.yaml

Then verify that the process was successful by running:



kubectl get all -l app.kubernetes.io/name=definition

Configure a RabbitMQ Instance

To configure a RabbitMQ instance in other words the cluster itself, open definition.yaml or edit the configuration in place by running:



kubectl edit rabbitmqcluster definition

By default all of the required manifests and objects to support your RabbitMQ have been created, to edit them you simply have to edit the Kubernetes manifest files.

Access the RabbitMQ dashboard on Kubernetes

You can access the RabbitMQ management dashboard in multiple ways but editing the service object and using a load balancer. The quickest way that does not require you to change any default configuration is by port-forwarding the definition-server-0 pod.



kubectl port-forward -n rabbitmq-system definition-server-0 8080:15672

Then you can access the dashboard login page to access the console. To find the credentials to access your deployment you will have to decode the username and password in the provisioned Kubernetes secret.



kubectl get secret definition-default-user -n rabbitmq-system -o yaml

The output will be something like this:



apiVersion: v1
data:
  default_user.conf: ZGVmYXVsdF91c2VYXVsdF91c2VyXzhMZklqOGFFQmROZmcKZGVmYXVsdF9wYXNzID0gd3IyWTl3MVJLMUtOdWN2S2ptS0FaX2R3dlRDSEhiYWEK
  host: ZGVmaW5pdGlvbi5yYWJiaXRtcS1zeXN0ZW0uc3Zj
  password: d3IyWTl3MVJLMUtOdWN2S2ptR3dlRDSEhiYWE=
  port: NTYMg==
  provider: cmFiml0bXE=
  type: cmFiYmlbXE=
  username: ZGVmYXVsdF91c2VMZklqFFQmRsbHd4b1BOZmc=

Decode the base64 password and username values and pass output into the login page



# Password
➜  ~ echo -n d3IyWTl3MVJLMUtOdW2S2ptS0FaX2R3dlRDSEhiYWE=  | base64 --decode
wr2Y9w1RK1KNucvKjmKAZ_dwvTCHHba%
# Username
➜  ~ echo -n ZGVmYXVsdF91c2VyXhMZklqOGFFQmRsbHd4b1BOZmc=  | base64 --decode
default_user_8LfIj8aEBdllwxoPNf%

ℹ️ To monitor your RabbitMQ instance check out this documentation.

Create a queue 🚃

There are multiple ways to declare queues in RabbitMQ, depending on your workflow and preferences.

Option 1: Queue Declaration in a Producer App

In most cases, you'll declare a queue within the producer app written in your preferred programming language. When the app publishes its first message, the queue declaration will automatically create the queue if it doesn't already exist. This approach allows for dynamic queue creation as needed.

Option 2: RabbitMQ Configuration Files

Alternatively, you can use RabbitMQ configuration files to declare multiple types of RabbitMQ objects, including vhosts, channels, exchanges, and queues. This method provides a more structured approach to managing RabbitMQ resources and configurations.

Option 3: Management UI

For manual management, you can create queues directly from the RabbitMQ Management UI. It provides a user-friendly interface for interacting with RabbitMQ, allowing you to perform administrative tasks without using the command line.

User and Virtual Host Creation (Optional)

Before creating queues, it's recommended to set up users and virtual hosts to organize messaging workflows effectively. For example:



rabbitmqctl add_user jake jakepassword
rabbitmqctl set_user_tags jake administrator
rabbitmqctl set_permissions -p / jake ".*" ".*" ".*"

rabbitmqctl add_vhost vhost-1
rabbitmqctl set_permissions -p vhost-1 jake ".*" ".*" ".*"

Queue Creation via Management UI

Access RabbitMQ Management Console: Open the RabbitMQ Management UI in your web browser.
Login with Credentials: Use the credentials of the user you created (e.g., username: jake, password: jakepassword) to log in to the Management UI.
Navigate to Queues Section: Click on the "Queues" tab in the Management UI to view existing queues.
Add Queue: Scroll to the bottom of the page and click on the "Add a new queue" button to create a new queue.

Specify the queue name, type, and other desired properties like node preference.
Optionally, configure queue bindings, policies, and other advanced settings based on your requirements.

Save Changes: Click on the "Add queue" or "Save" button to confirm and create the new queue.

By following these steps, you can easily create and manage queues in RabbitMQ according to your specific needs.

The queue is now ready:

Build a RabbitMQ cluster 🐇

When building a production application with a messaging component to its architecture, considerations around high availability must be made. RabbitMQ offers a clustering feature that consists of joining multiple RabbitMQ brokers and replicating the configuration and the data contained inside the message queues.

ℹ️ To understand how clusters are formed we will need to understand a few concepts.

By using the operator, much of the concepts we explore below will be configured for you by default. Nonetheless, it’s still valuable to understand what is happening under the hood.

Authentication

For a RabbitMQ node to join forces and form a cluster with another node that need to share an Erlang cookie environment variable. This will ensure that nodes share the configuration data needed to consider themselves a cluster. This doesn’t ensure queue data replication though. This is where the concept of mirroring comes in.

Quorum queues

The RabbitMQ quorum queue is a modern queue type, which implements a durable, replicated FIFO queue based on the Raft consensus algorithm. Previously Classic Mirrored Queues were recommended but they have since been deprecated. Quorum queues are implemented much the same way yet they offer high availability via replication and focus on data safety. Policies can be applied to queues and it’s through these policies that we can configure the mirroring specification.

Queue declaration

In the past, replication of queues was specified by using policies in conjunction with Classic Queues. Quorum queues are created differently but should be compatible with all client applications which allow you to provide arguments when declaring a queue. The x-queue-type argument needs to be provided with the value quorum when creating the queue. For example, using the Elixir AMQP client1, declaring a Quorum Queue is as follows:



Queue.declare(publisher_chan, "my-quorum-queue", durable: true, arguments: [ "x-queue-type": "quorum" ])

ℹ️ Manage quorum queues easily by using the rabbitmq-queues CLI. Here are some useful CLI commands.

Automatic Failover

Each quorum queue is made up by a primary replica, known as the leader in Raft terminology, along with potentially multiple secondary replicas, referred to as followers.

Initially, a leader is elected when the cluster is formed, and subsequently, if the current leader becomes unavailable.

Suppose a node hosting the leader of a quorum queue fails or stops for any reason. In that case, another node hosting one of the followers for that quorum queue will assume leadership and resume operations.

When failed or rejoining followers come back online, they undergo a process of resynchronization, commonly known as "catching up," with the leader. Unlike traditional mirrored queues, where a temporary replica failure requires a full resynchronization from the current leader, only the differential changes are transferred if a rejoining replica lags behind the leader.

Fault tolerance

For optimal performance in RabbitMQ clusters, it's beneficial to constrain the quorum queue size to a smaller, odd number of nodes.

It's also worth mentioning that performance tends to degrade noticeably with quorum queue node sizes that exceed 5. Therefore, it’s strongly advised against deploying quorum queues on more than 7 RabbitMQ nodes.

Add instances to the cluster 🪴

ℹ️ Upon deployment, the cluster operator automatically generates a set of Kubernetes manifest files defining the default configurations for the RabbitMQ instance. You can inspect these configurations using the following command: Run k get all -n rabbitmq-admin to see all created objects. These default configurations can be customized and updated to better align with your specific requirements and preferences.

The RabbitMQ Kubernetes operator simplifies the setup process by handling essential tasks such as Mirroring, Failover, and Node federation automatically. This means that creating a RabbitMQ cluster is as straightforward as adjusting the number of replicas in the RabbitmqCluster definition file. With these capabilities built-in, managing RabbitMQ clusters becomes much more seamless and hassle-free.

Here I updated the RabbitmqCluster manifest to scale to 3 replicas:

ℹ️ When specifying the number of replicas for the RabbitmqCluster object an even number of replicas is highly discouraged. Odd numbers (1, 3, 5, 7, and so on) must be used.



apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rabbitmq.com/v1beta1","kind":"RabbitmqCluster","metadata":{"annotations":{},"name":"definition","namespace":"rabbitmq-system"}}
  creationTimestamp: "2024-05-07T17:04:53Z"
  finalizers:
  - deletion.finalizers.rabbitmqclusters.rabbitmq.com
  generation: 6
  name: definition
  namespace: rabbitmq-system
  resourceVersion: "2729957"
  uid: f9b4cad8-68b4-489e-b7a1-baa3b91882d5
spec:
  delayStartSeconds: 30
  image: rabbitmq:3.13.1-management
  override: {}
  persistence:
    storage: 10Gi
  rabbitmq:
    additionalConfig: |
      load_definitions = /etc/rabbitmq/definitions.json
  replicas: 3 # added three replicas which will for the RabbitMQ cluster
  resources: 
    limits:
      cpu: "2"
      memory: 2Gi
    requests:
      cpu: "1"
      memory: 2Gi
  secretBackend:
    externalSecret: {}
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800
...

Not only have the nodes been added to the cluster but the queue has been automatically replicated to all cluster nodes too

Messaging brokers such as RabbitMQ play a central role in event-driven architectures, offering flexibility for task processing, workflow orchestration, and service integrations while offering a selection of messaging patterns to choose from to best suit your architecture. An obvious choice when prioritizing delivery reliability and fault tolerance, due to its native acknowledgment and message persistence features.

Getting started with RabbitMQ is easier than ever especially now that Glasskube integrates the cluster controller for easy installation and management. This guide aims to offer the necessary insights for setting up a functional instance or RabbitMQ cluster within a Kubernetes environment. While there's so much more to learn regarding RabbitMQ such as this and that, I hope this primer provides ample groundwork to kickstart your RabbitMQ journey.

If you like this sort of content and would like to see more of it, please consider supporting us by giving us a Star on GitHub 🙏