DEV Community

Cover image for RabbitMQ Monitoring with Prometheus and Grafana
Fidelis Ikoroje
Fidelis Ikoroje

Posted on

3

RabbitMQ Monitoring with Prometheus and Grafana

In this project, I completed two setups for RabbitMQ monitoring with Prometheus and Grafana. The first setup is a local Docker Compose setup for quick testing while the other is a 3-node RabbitMQ cluster setup on AWS EC2 for scalability and real-world applications.

Overview of the Project

RabbitMQ is a popular message broker that plays a crucial role in distributed systems. This project aimed to:
I. Set up a RabbitMQ environment both locally and on AWS.
II. Monitor RabbitMQ metrics using Prometheus and visualize them with Grafana.
III. Test the message queuing system using Python scripts.


Part 1: Local RabbitMQ Setup Using Docker Compose

Why Local Setup?

The local setup serves as a quick testing environment before deploying to the cloud.

Steps to Set Up:

I. Clone the RabbitMQ Repository:

   git clone https://github.com/rabbitmq/rabbitmq-server.git
   cd rabbitmq-server/deps/rabbitmq_prometheus/docker
Enter fullscreen mode Exit fullscreen mode

II. Run Docker Compose:
Use Docker Compose to start the RabbitMQ cluster and Prometheus instance, along with a basic workload to generate meaningful metrics. This will start a RabbitMQ cluster, Prometheus, and Grafana with predefined configurations, collecting metrics from RabbitMQ.

   docker-compose -f docker-compose-metrics.yml up -d
   docker-compose -f docker-compose-overview.yml up -d
Enter fullscreen mode Exit fullscreen mode

docker_compose_metrics

Rabbit_Metrics

Docker_Containers

III. Access RabbitMQ Dashboard:

  • URL: http://localhost:15672
  • Default credentials: Username: guest Password: guest

IV. Access Grafana Dashboard:

  • URL: http://localhost:3000
  • Default credentials: Username: admin Password: admin

Image description

Grafana_metrics_2

Grafana_metrics_3


Part 2: RabbitMQ 3-Node Cluster on AWS EC2

Why a Cluster?

To ensure high availability and load distribution in production.

Prerequisites

3 AWS EC2 instances running Ubuntu 24.04 for RabbitMQ and one each for Prometheus and Grafana. Configure the Security Groups as follows:

  • RabbitMQ Security Groups:

    • Allow SSH (port 22) from your IP.
    • Allow RabbitMQ Management UI (port 15672) from your IP.
    • Allow RabbitMQ Prometheus Metrics (port 15692) from Prometheus instance SG.
  • Prometheus Security Groups:

    • Allow HTTP (port 80) from your IP.
    • Allow HTTPS (port 443) from your IP.
    • Allow Grafana (port 3000) from your IP.
  • Grafana Security Groups:

    • Allow HTTP (port 80) from your IP.
    • Allow HTTPS (port 443) from your IP.

Step-by-Step Guide

I. Install RabbitMQ on All 3 Nodes:
Use a bash script as EC2 user data during instance launch or save it and run it on each EC2 instance. The bash script I used is named rabbitmq_installation.sh in my git repository. It includes all the plugins needed to help Prometheus scrape metrics from RabbitMQ.

RabbitMQ_Nodes_Running

II. Cluster Configuration:

  • Verify Erlang cookie consistency:

     sudo cat /var/lib/rabbitmq/.erlang.cookie
    

    Ensure the same cookie is on all 3 RabbitMQ EC2 nodes.

  • Join nodes to form a cluster:
    Pick one EC2 as reference and run the commands below on the other two EC2's.

     sudo rabbitmqctl stop_app
     sudo rabbitmqctl reset
     sudo rabbitmqctl join_cluster rabbit@<master-node-hostname>
     sudo rabbitmqctl start_app
    

III. Verify Cluster Status:

   sudo rabbitmqctl cluster_status
Enter fullscreen mode Exit fullscreen mode

3-EC2 Nodes Joined


Part 3: Monitoring with Prometheus and Grafana

Prometheus Setup on AWS EC2

I. Install Prometheus Using a Bash Script:
I used a Bash script named prometheus.sh saved in my git repository to install Prometheus and it includes all the plugins needed. Save this script on your Prometheus EC2 and run the script.

   ./prometheus.sh
Enter fullscreen mode Exit fullscreen mode

II. Configure Prometheus:
Edit prometheus.yml to scrape RabbitMQ metrics:

   scrape_configs:
     - job_name: 'rabbitmq'
       static_configs:
         - targets: ['<rabbitmq-node-1-ip>:15692', '<rabbitmq-node-2-ip>:15692', '<rabbitmq-node-3-ip>:15692']
Enter fullscreen mode Exit fullscreen mode

III. Run Prometheus:

   /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Prometheus_UI_Showing_RabbitMQ_Cluster

Grafana Setup on AWS EC2

I. Install Grafana Using a Bash Script:

   #!/bin/bash
   sudo apt update -y
   sudo apt install -y wget unzip
   wget https://dl.grafana.com/enterprise/release/grafana-enterprise-11.4.0.linux-arm64.tar.gz
   tar xvf grafana-enterprise-11.4.0.linux-arm64.tar.gz
   sudo mv grafana-11.4.0 /usr/local/grafana
   echo 'Grafana installed.'
Enter fullscreen mode Exit fullscreen mode

II. Configure Grafana:

  • Add Prometheus as a data source in Grafana
  • Import RabbitMQ dashboards - the file is named RabbitMQ-Overview.json

III. Verify Metrics on Grafana:

  • URL: http://<grafana-ip>:3000
  • Navigate to imported RabbitMQ dashboards to monitor metrics.

Grafana_running


Part 4: Testing the Message Queue

Using Python scripts, I tested message publishing and consumption:

import pika

# Connection setup
credentials = pika.PlainCredentials('guest', 'guest')
connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq-node-ip', credentials=credentials))
channel = connection.channel()

# Declare queue
channel.queue_declare(queue='test_queue')

# Publish message
channel.basic_publish(exchange='', routing_key='test_queue', body='Hello RabbitMQ!')
print("Message published.")

# Consume message
def callback(ch, method, properties, body):
    print(f"Received: {body}")

channel.basic_consume(queue='test_queue', on_message_callback=callback, auto_ack=True)
channel.start_consuming()
Enter fullscreen mode Exit fullscreen mode

Grafana showing metrics scrapped by Prometheus
Grafana_showing_metrics

Grafana_showing_metrics_2


Final Thoughts

This project showcased the power of RabbitMQ in a distributed setup and the importance of monitoring using Prometheus and Grafana. Whether for testing or production, the tools and techniques used here ensure a scalable and observable messaging system.

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay