DEV Community

Cover image for Message Schema Evolution in RabbitMQ: Using Virtual Hosts as Deployment Boundaries
İbrahim Gündüz
İbrahim Gündüz

Posted on • Originally published at ibrahimgunduz34.Medium

Message Schema Evolution in RabbitMQ: Using Virtual Hosts as Deployment Boundaries

#architecture #devops #microservices

This article is a continuation of a previous post in which we focused on deployment safety through cache key versioning. If you haven’t read it yet, we recommend starting there:

Message schema evolution is another critical factor that can threaten deployment safety if it is not handled correctly. Changes to message structures, such as adding, removing, or renaming fields, can easily lead to incompatibilities between producers and consumers during rolling deployments. Without clear deployment boundaries, these incompatibilities may cause message processing failures, data loss, or subtle runtime bugs that are difficult to detect.

Let’s look at the following example to better understand the risks involved.

Example Case:

v1: Delayed Capture After Authorization

  • The backend performs the authorization synchronously during the checkout flow and immediately informs the user of the result.
  • After a successful authorization, the backend publishes an event indicating that the payment has been authorized. (PaymentAuthorized) 
{
  "paymentId": "p-123",
  "authorizedAmount": 100.00,
  "currency": "EUR"
}
Enter fullscreen mode Exit fullscreen mode
  • The consumer processes this event and performs a full capture of the authorized amount. 
capture(authorizedAmount)
Enter fullscreen mode Exit fullscreen mode

v2: Partial capture based on stock availability

As the system evolves, some products may be out of stock at fulfillment time.

  • The backend still performs the authorization synchronously and returns the result to the user.
  • Before capturing the payment, the system checks stock availability via the inventory service.
  • The backend now publishes an event that includes the actual amount to be captured, which may be lower than the authorized amount. (PaymentCaptureRequested) 
{
  "paymentId": "p-123",
  "authorizedAmount": 100.00,
  "captureAmount": 80.00,
  "currency": "EUR"
}
Enter fullscreen mode Exit fullscreen mode
  • The consumer captures only the specified amount. 
capture(captureAmount)
Enter fullscreen mode Exit fullscreen mode

What Really Happens During a Rolling Deployment?

Scenario 1: v2 consumer processing v1 messages

  • Because the message does not contain captureAmount field, the consumer may fail to process the message.

Result: Capture fails or the consumer behaves unpredictably

Scenario 2: v1 consumer processing v2 messages

  • The v1 consumer does not recognize the captureAmount field or may ignore it entirely.
  • Since authorizedAmount is an expected field, the v1 consumer proceeds to capture the full authorized amount.

Result: The customer is charged more than the value of the fulfilled items.

The root cause of these failures is not incorrect business logic, but the fact that multiple, incompatible versions of producers and consumers are allowed to process the same messages during a deployment. To prevent this, RabbitMQ virtual hosts can be used to ensure that only compatible versions are able to communicate with each other.

Using RabbitMQ Virtual Hosts as Deployment Boundaries

To achieve clean isolation and a risk-free cutover, the following steps can be applied to your applications and deployment pipeline.

1. Externalize RabbitMQ Configuration

  • Update both producer and consumer applications to inject the RabbitMQ virtual host and credentials via external configuration. 
spring.rabbitmq.virtualHost=${RABBITMQ_VHOST}
spring.rabbitmq.username=${RABBITMQ_USER}
spring.rabbitmq.password=${RABBITMQ_PASSWORD}
Enter fullscreen mode Exit fullscreen mode

2. Provision Version-Scoped RabbitMQ Resources

  • Add a step to your deployment pipeline that provisions a version-scoped RabbitMQ virtual host, user, and permissions before deploying the application. 
$ rabbitmqctl add_vhost payments-v1.0
$ rabbitmqctl add_user payments-v1.0-app YourStrongPassword
$ rabbitmqctl set_user_tags payments-v1.0-app
$ rabbitmqctl set_permissions -p payments-v1.0 \
  payments-v1.0-app \
  ".*" ".*" ".*"
Enter fullscreen mode Exit fullscreen mode

3. Inject Configuration via Environment Variables

  • Provide the required configuration values through the application environment. 
...
  environment:
    RABBITMQ_VHOST: ${RABBITMQ_VHOST}
    RABBITMQ_USER: ${RABBITMQ_USER}
    RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
...
  command: java -jar ...
Enter fullscreen mode Exit fullscreen mode

4. Deploy and Verify

  • Deploy the new version of the application.
  • Perform health checks to ensure all nodes are running correctly.
  • Verify that messages from the previous version have been fully consumed.

5. Clean Up Old Resources

Once the deployment is complete and the previous version is no longer needed:

  • Remove the old virtual host.
  • Delete the associated user.

This ensures the broker remains clean and prevents obsolete deployment boundaries from accumulating over time.

Credits:

Top comments (0)