RabbitMQ Monitoring with Prometheus and Grafana

In this project, I completed two setups for RabbitMQ monitoring with Prometheus and Grafana. The first setup is a local Docker Compose setup for quick testing while the other is a 3-node RabbitMQ cluster setup on AWS EC2 for scalability and real-world applications.

Overview of the Project

RabbitMQ is a popular message broker that plays a crucial role in distributed systems. This project aimed to:
I. Set up a RabbitMQ environment both locally and on AWS.
II. Monitor RabbitMQ metrics using Prometheus and visualize them with Grafana.
III. Test the message queuing system using Python scripts.

---

Part 1: Local RabbitMQ Setup Using Docker Compose

Why Local Setup?

The local setup serves as a quick testing environment before deploying to the cloud.

Steps to Set Up:

I. Clone the RabbitMQ Repository:

   git clone https://github.com/rabbitmq/rabbitmq-server.git
   cd rabbitmq-server/deps/rabbitmq_prometheus/docker

II. Run Docker Compose:
Use Docker Compose to start the RabbitMQ cluster and Prometheus instance, along with a basic workload to generate meaningful metrics. This will start a RabbitMQ cluster, Prometheus, and Grafana with predefined configurations, collecting metrics from RabbitMQ.

   docker-compose -f docker-compose-metrics.yml up -d
   docker-compose -f docker-compose-overview.yml up -d

III. Access RabbitMQ Dashboard:

• URL: http://localhost:15672
• Default credentials: Username: guest Password: guest

IV. Access Grafana Dashboard:

• URL: http://localhost:3000
• Default credentials: Username: admin Password: admin

---

Part 2: RabbitMQ 3-Node Cluster on AWS EC2

Why a Cluster?

To ensure high availability and load distribution in production.

Prerequisites

3 AWS EC2 instances running Ubuntu 24.04 for RabbitMQ and one each for Prometheus and Grafana. Configure the Security Groups as follows:

• RabbitMQ Security Groups:

  • Allow SSH (port 22) from your IP.
  • Allow RabbitMQ Management UI (port 15672) from your IP.
  • Allow RabbitMQ Prometheus Metrics (port 15692) from Prometheus instance SG.

• Prometheus Security Groups:

  • Allow HTTP (port 80) from your IP.
  • Allow HTTPS (port 443) from your IP.
  • Allow Grafana (port 3000) from your IP.

• Grafana Security Groups:

  • Allow HTTP (port 80) from your IP.
  • Allow HTTPS (port 443) from your IP.

Step-by-Step Guide

I. Install RabbitMQ on All 3 Nodes:
Use a bash script as EC2 user data during instance launch or save it and run it on each EC2 instance. The bash script I used is named rabbitmq_installation.sh in my git repository. It includes all the plugins needed to help Prometheus scrape metrics from RabbitMQ.

II. Cluster Configuration:

• Verify Erlang cookie consistency:
  

   sudo cat /var/lib/rabbitmq/.erlang.cookie
  

  Ensure the same cookie is on all 3 RabbitMQ EC2 nodes.

• Join nodes to form a cluster:
  Pick one EC2 as reference and run the commands below on the other two EC2's.
  

   sudo rabbitmqctl stop_app
   sudo rabbitmqctl reset
   sudo rabbitmqctl join_cluster rabbit@<master-node-hostname>
   sudo rabbitmqctl start_app
  

III. Verify Cluster Status:

   sudo rabbitmqctl cluster_status

---

Part 3: Monitoring with Prometheus and Grafana

Prometheus Setup on AWS EC2

I. Install Prometheus Using a Bash Script:
I used a Bash script named prometheus.sh saved in my git repository to install Prometheus and it includes all the plugins needed. Save this script on your Prometheus EC2 and run the script.

   ./prometheus.sh

II. Configure Prometheus:
Edit prometheus.yml to scrape RabbitMQ metrics:

   scrape_configs:
     - job_name: 'rabbitmq'
       static_configs:
         - targets: ['<rabbitmq-node-1-ip>:15692', '<rabbitmq-node-2-ip>:15692', '<rabbitmq-node-3-ip>:15692']

III. Run Prometheus:

   /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

Grafana Setup on AWS EC2

I. Install Grafana Using a Bash Script:

   #!/bin/bash
   sudo apt update -y
   sudo apt install -y wget unzip
   wget https://dl.grafana.com/enterprise/release/grafana-enterprise-11.4.0.linux-arm64.tar.gz
   tar xvf grafana-enterprise-11.4.0.linux-arm64.tar.gz
   sudo mv grafana-11.4.0 /usr/local/grafana
   echo 'Grafana installed.'

II. Configure Grafana:

• Add Prometheus as a data source in Grafana
• Import RabbitMQ dashboards - the file is named RabbitMQ-Overview.json

III. Verify Metrics on Grafana:

• URL: http://<grafana-ip>:3000
• Navigate to imported RabbitMQ dashboards to monitor metrics.

---

Part 4: Testing the Message Queue

Using Python scripts, I tested message publishing and consumption:

import pika

# Connection setup
credentials = pika.PlainCredentials('guest', 'guest')
connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq-node-ip', credentials=credentials))
channel = connection.channel()

# Declare queue
channel.queue_declare(queue='test_queue')

# Publish message
channel.basic_publish(exchange='', routing_key='test_queue', body='Hello RabbitMQ!')
print("Message published.")

# Consume message
def callback(ch, method, properties, body):
    print(f"Received: {body}")

channel.basic_consume(queue='test_queue', on_message_callback=callback, auto_ack=True)
channel.start_consuming()

Grafana showing metrics scrapped by Prometheus

---

Final Thoughts

This project showcased the power of RabbitMQ in a distributed setup and the importance of monitoring using Prometheus and Grafana. Whether for testing or production, the tools and techniques used here ensure a scalable and observable messaging system.