Using Vercel's instant rollback feature in your own CI/CD pipeline

Vercel, a Platform as a Service (PaaS) for hosting Next.js web applications, offers some really convenient features like instant rollbacks. This post explores how to use this feature in a custom CI pipeline to allow colleagues without a Vercel login to carry out rollbacks.

Vercel is built on a serverless architecture, so it's cheap for them to keep copies of all your previous deployments. This allows for some cool features, like a vercel bisect CLI command that works like git bisect, except instead of checking out file changes locally to find a bug, it gives you URLs of each deployment to test before telling you which deployment introduced an issue.

 Vercel instant rollbacks

Another cool feature is instant rollbacks. If you deploy a change that causes issues, rather than doing a git revert or a hotfix and then re-running your build/test/deploy pipeline you can instead instantly revert to the last deployment, saving lots of time and stress. You can do a rollback via logging into Vercel's admin UI, via their CLI, or via an undocumented REST endpoint.

A downside to Vercel is that it's a premium service with a premium cost for the higher-end tiers. You're charged per-seat (per developer), so in many organisations not everyone will have a Vercel login. This is a bit of a problem for instant rollbacks – what if the developer who spots the production issue doesn't have a Vercel account so can't use their admin UI, CLI, or REST API?

To work around this issue at work I recently added a 'Rollback' step to the end of our CI pipeline. Every developer has access to the pipeline so will be able to initiate a rollback via GitLab CI instead of Vercel. The pipeline is authorised to connect to Vercel's API using an API token associated with an existing Vercel account.

How I did it

Firstly I generated a Vercel API token and added it to GitLab CI as an environment variable called VERCEL_TOKEN. I also created a VERCEL_TEAM_ID and VERCEL_PROJECT_ID using the team and project IDs from the Vercel web interface.

Vercel's REST API documentation is missing docs for a 'rollback' endpoint, but Vercel's CLI is hosted on GitHub and you can see how they are using their own REST API (a good sign!) for the rollback CLI command:

  await client.fetch(`/v9/projects/${project.id}/rollback/${deployment.id}`, {
    body: {}, // required
    method: 'POST',
  });

The rollback process, via API

Before we can send a rollback request to the API we need to find the previous production release's deployment ID. Here's how I did that in Bash, using jq:

# Fetch the rollback deployment ID
ROLLBACK_DEPLOYMENT_ID=$(curl -s "https://api.vercel.com/v6/deployments?teamId=$VERCEL_TEAM_ID&projectId=$VERCEL_PROJECT_ID&limit=2&rollbackCandidate=true&state=READY&target=production" -H "Authorization: Bearer $VERCEL_TOKEN" | jq -r '.deployments[1].uid')
echo "Rolling back to the previous deployment ID: $ROLLBACK_DEPLOYMENT_ID"

Note the query string parameters in that request:

• limit=2, as we want to get the previous production deployment, not the current one
• rollbackCandidate=true, as we need a deployment we can rollback to
• state=READY, to filter out deployments that might be in-progress
• target=production, because we are not interested in preview deployments

We pipe (|) the silent curl output to jq and use it to get the second uid from the results. We now have a variable called ROLLBACK_DEPLOYMENT_ID that we can include in a POST request to Vercel's undocumented rollback endpoint:

# Initiate a rollback to the previous deployment
curl -s -X POST "https://api.vercel.com/v9/projects/${VERCEL_PROJECT_ID}/rollback/${ROLLBACK_DEPLOYMENT_ID}?teamId=${VERCEL_TEAM_ID}" -H "Authorization: Bearer $VERCEL_TOKEN" -H "Content-Type: application/json"

If both those commands are successful then your site should rollback to the previous production release, hopefully before too many of your users have noticed the issue.

GitLab CI example

I added the above commands to a scripts/rollback-current-prod-deployment.sh file, made it executable, and then referenced it in a GitLab CI job like this:

Rollback current prod deployment:
  image: registry.gitlab.com/gitlab-ci-utils/curl-jq:latest
  stage: deploy
  needs:
    - job: Upload to Vercel prod URL
      optional: true
  script:
    - './scripts/rollback-current-prod-deployment.sh'
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      when: never
    - if: $CI_COMMIT_BRANCH != "main"
      when: never
    - when: manual

This job:

• only runs manually
• only runs after the 'Upload to Vercel prod URL' job
• uses a Docker image that contains both curl and jq. (Note that despite the name, this image is not an official GitLab product)

It'd be a very similar approach to use this with GitHub Actions or any other CI/CD pipeline provider.

How to take this idea further

I made a few other additions that are application-specific so I haven't covered here.

An 'undo rollback' job

I created a job to undo the rollback, in case it was triggered accidentally or was no longer needed. This means the original production deployment can be quickly restored, without re-running the whole pipeline

I did this by storing the original deployment ID and writing it to a file that a second pipeline step could read from.

Confirmation that the rollback worked by displaying the deployed version after the rollback succeeded

I created an /api/version endpoint in my application that returned the version of the site in a JSON object. I query this after the rollback job so I can give the pipeline user confirmation that the rollback succeeded. I set the API route to have no caching with a max-age header (public, max-age=0, must-revalidate), but just to be sure I added some cache busting to the CI job too:

# Fetch version information twice to help bust cache and stale-while-revalidate
cache_buster=$(date +%s)
curl -s -o /dev/null "https://example.com/api/version?cache_buster=$cache_buster"
VERSION_INFO=$(curl -s "https://example.com/api/version?cache_buster=$cache_buster")
echo "Version after rollback: $(echo $VERSION_INFO | jq -r '.version')"