Preventing Concurrent Migrations in a Distributed Environment

Introduction:

In a distributed environment, where multiple apps share a single database, running migrations simultaneously can lead to conflicts and deployment failures. In this article, we'll explore a solution to prevent concurrent migrations and ensure smooth deployments. We'll discuss the issue, the approach used, and the code changes implemented to tackle this problem.

Problem:

When running migrations in Rails, they are meant to run sequentially based on their timestamps. However, in a distributed environment like Heroku, multiple apps can trigger migrations simultaneously, causing conflicts and errors. This occurs when one app tries to run a migration while another migration is already in progress, resulting in an ActiveRecord::ConcurrentMigrationError exception.

Approach:

To address this issue, we'll use a combination of advisory locks and environment variables to control the execution of migrations during deployment. The basic idea is to allow only one app to run migrations while the others skip them.

Implementation:

Let's walk through the code changes needed to implement this solution.

Step 1: Create a release-tasks.sh Script
We'll start by creating a shell script called release-tasks.sh in the project's root directory. This script will handle the execution of migrations during the release phase.

bash

#!/bin/bash
set -e

if [ "$RUN_MIGRATIONS" = "true" ]; then
  echo "Running migrations"
  bundle exec rake db:migrate
else
  echo "Skipping migrations"
fi

The script checks the value of the RUN_MIGRATIONS environment variable. If it is set to "true," the script runs the migrations using bundle exec rake db:migrate. Otherwise, it skips the migrations.

Step 2: Grant Execution Permissions to the Script
To allow execution of the release-tasks.sh script, we need to grant it execution permissions. Run the following command:

bash

chmod +x release-tasks.sh

Step 3: Update the Procfile
In the project's Procfile, we need to modify the release command to use our custom script. Open the Procfile and update the line as follows:

release: ./release-tasks.sh

This change ensures that our release-tasks.sh script is executed during the release phase.

Step 4: Set Environment Variables for Each App
In Heroku, go to each app's settings and set the RUN_MIGRATIONS environment variable. For the app that should run migrations, set RUN_MIGRATIONS to "true". For the other apps that should skip migrations, either leave it unset or set it to any other value.

Conclusion:

By implementing this solution, we have effectively prevented concurrent migrations in our distributed environment. By using advisory locks and environment variables, we ensure that only one app runs migrations while others skip them during deployment. This approach promotes a more stable and error-free deployment process.

Remember to test this solution thoroughly in a non-production environment before applying it to your live applications. It's crucial to consider the specific requirements and constraints of your own setup.

I hope this article helps you in handling concurrent migrations in your distributed environment. Feel free to reach out if you have any questions or need further assistance.

Happy coding!

Comments

[Youri van der Lans]

Coming back here to report on our approach. We've used a similar task as described in this post but then using Ruby.

The task either runs the migration or it runs bundle exec rails db:migrate:status in a loop to see if there are any pending migrations.

The upside of this approach is that when migrations fail the other heroku app is not deployed.

It has to be noted that there is no way to know if migrations failed inside the release phase where the migration status is being polled (unless you'd want to do complicated API calls to heroku). We have chosen to let the polling run until the release process is killed by heroku, after 1 hour. Although it is also possible to stop the release phase manually.

[Youri van der Lans]

We're running into the same issue on Heroku and we're also thinking about a custom release rake task to either run or not run the migrations.

What happens when the migrations fail though? I think with this approach one app will be deployed and the other will not.

From the Heroku docs:

If a release phase task fails, the new release is not deployed, leaving the current release unaffected.

devcenter.heroku.com/articles/rele...

I've not found a solution to this issue but thought it would be worthwhile to mention.