Spacelift team for Spacelift

Posted on Oct 14 • Originally published at spacelift.io

How to Migrate From Terraform Cloud

#terraform

When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates.

Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.

Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit.

Getting started with Spacelift Migration Kit

To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:

Install Python3.10 or newer and Poetry
Clone the Spacelift Migration Kit.
Install other dependencies and the spacemk command in a Python virtual environment
Create your config.yml
[Optional] Audit your current setup
Export the configuration
Generate the Terraform/OpenTofu configuration
Create a repository based on the generated code
Create a stack based on the repository
State Migration
[Optional] Customization
Useful information

Let's cover these steps in detail.

How to migrate from Terraform Cloud

1. Install prerequisites

We need to ensure that a version newer than python3.10 is installed and also that poetry is installed.

Installing poetry on MacOS using pipx

brew install pipx
pipx install poetry
pipx ensurepath

To install it on any other operating system, check out this tutorial.

2. Clone the Spacelift migration kit

Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.

git clone git@github.com:spacelift-io/spacelift-migration-kit.git

3. Install spacemk

Go to the Spacelift migration kit directory, and run the following:

poetry install
Creating virtualenv spacemk-liMdtQjr-py3.12 in /Users/flaviuscdinu/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file

Now after you have installed spacemk, activate the virtual environment:

poetry shell

4. Create your config.yml

Inside the repository, there is a config.yml.example file, containing the values you need to configure for the migration:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: # Valid values: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token:
   include:
     workspaces: ^example-.*$

generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"

github:
 api_token:
 endpoint: https://api.github.com

spacelift:
 api:
   api_key_endpoint: https://<ACCOUNT NAME>>.app.spacelift.io/graphql
   api_key_id:
   api_key_secret:

Rename this file, or create a new one that should be called "config.yml".

In this file, you must add the login details to your Terraform Cloud account and your Spacelift account. You can also use environment variables inside this file, to avoid using credentials inside the file (these variables should be prefixed with the "$" sign).

Here is an example that uses environment variables:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token: $TFC_TOKEN

generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"

github:
 api_token: $GITHUB_TOKEN
 endpoint: https://api.github.com

spacelift:
 api:
   api_key_endpoint: https://saturnhead.app.spacelift.io/graphql
   api_key_id: $SPACELIFT_API_KEY_ID
   api_key_secret: $SPACELIFT_API_KEY_SECRET

You will need to export these values inside of your terminal (e.g.):

export TFC_TOKEN=tfctoken

Based on the values above, we will include all workspaces in the migration.

If you don't already have a Spacelift API key, you can go to Settings → API keys and generate one:

After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.

To create an API key for Terraform Cloud check this tutorial, and to create an GitHub token, check this tutorial.

5. [Optional] Audit your current setup

To audit your current setup, you can simply run the spacemk audit command:

Agent Pools: 1 (including 1 with warnings)
Modules: 1
Organizations: 1
Policies: 1 (including 1 with warnings)
Policy Sets: 1
Projects: 4
Providers: 0
Tasks: 0
Teams: 601
Variable Set Variables: 0
Variable Sets: 0
Workspace Variables: 0
Workspaces: 1

This will give you all the information about your Terraform Cloud account, and everything that can be exported.

6. Export your configuration

To export your configuration, simply run spacemk export. This command will generate a json file in the tmp directory, called data.json.

In this file, you will see information about everything that will be migrated and how they are mapping out to Spacelift resources.

{
 "context_variables": [],
 "contexts": [],
 "modules": [
   {
     "_migration_id": "aks",
     "_relationships": {
       "space": {
         "_migration_id": "saturnhead",
         "_source_id": "saturnhead",
         "name": "saturnhead",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "name": "aks",
     "status": "setup_complete",
     "terraform_provider": "az",
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "provider": "github_custom",
       "repository": "terraform-az-aks"
     },
     "visibility": "private"
   }
 ],
 "spaces": [
   {
     "_migration_id": "saturnhead",
     "_relationships": {},
     "_source_id": "saturnhead",
     "name": "saturnhead",
     "requires_terraform_workflow_tool": false
   },
   {
     "_migration_id": "default_project",
     "_relationships": {},
     "_source_id": "...",
     "name": "Default Project",
     "requires_terraform_workflow_tool": false
   },
   ...
   {
     "_migration_id": "sdsd",
     "_relationships": {},
     "_source_id": "...",
     "name": "sdsd",
     "requires_terraform_workflow_tool": false
   }
 ],
 "stack_variables": [],
 "stacks": [
   {
     "_migration_id": "blog_examples",
     "_relationships": {
       "space": {
         "_migration_id": "default_project",
         "_source_id": "...",
         "name": "Default Project",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "autodeploy": false,
     "description": null,
     "has_secret_variables_with_invalid_name": false,
     "has_variables_with_invalid_name": false,
     "name": "blog-examples",
     "slug": "blog-examples",
     "terraform": {
       "version": "1.5.5",
       "workflow_tool": "TERRAFORM_FOSS"
     },
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "project_root": "migration",
       "provider": "github_custom",
       "repository": "blog-examples"
     }
   }
 ]
}

This file can be reviewed and modified to accommodate your needs.

💡 You might also like:

7. Generate the Terraform/OpenTofu configuration

To generate the Terraform/OpenTofu configuration, you will need to use the spacemk generate command.

terraform {
 required_providers {
   spacelift = {
     source  = "spacelift-io/spacelift"
     version = "~> 1.0"
   }
 }
}

resource "spacelift_space" "saturnhead" {
 inherit_entities = true
 name             = "saturnhead"
 parent_space_id  = "root"
}

resource "spacelift_stack" "default_project_blog_examples" {
 branch            = "main"
 manage_state      = false
 name              = "blog-examples"
 project_root      = "migration"
 repository        = "blog-examples"
 space_id          = spacelift_space.default_project.id
 terraform_version = "1.5.5"

 github_enterprise {
   namespace = "flavius-dinu"
 }
}

...

resource "spacelift_module" "saturnhead_aks" {
 branch             = "main"
 name               = "aks"
 repository         = "terraform-az-aks"
 space_id           = spacelift_space.saturnhead.id
 terraform_provider = "az"

 github_enterprise {
   namespace = "flavius-dinu"
 }
}

This will generate the Spacelift configuration required for the migration. Again, the code can be modified to accommodate your needs.

If you are not using the default VCS integration, you will also need to specify the id property in the vcs block (in my case, I will add an id to the github_enterprise block):

github_enterprise {
   namespace = "flavius-dinu"
   id        = "flavius-dinu"
 }

8. Create a repository based on this code

In your VCS, simply create a repository with the exported code. Ensure the vcs provider you are using is configured in Spacelift, and the code repositories you've exported from TFC are available in this vcs provider.

To configure a new VCS provider inside your Spacelift provider, follow the steps from this tutorial for your VCS.

Create the repo with the exported code, this is how it looks like for my GitHub configuration:

9. Create a stack based on the repository

Go to your Spacelift account, select Stacks, and the Create Stack option:

Add a name to your stack, select a space, and add optional labels and description.

In the next step, select the repository containing your migration code as shown above, and specify the branch you want to use.

Next, you can choose your vendor, it can be either Terraform, or OpenTofu, so I will accept the default Terraform FOSS workflow.

Ensure that the Administrative option is toggled on in the define behavior tab, and accept all other defaults to create the stack.

After the stack is created, trigger a run:

After reaching the unconfirmed state, you will see all the resources that it will create, in my case, there are 5 spaces, 1 stack, and 1 module.

You can easily see that all resources have been migrated successfully.

10. State Migration

If we go to our stack and select resources, we will see that there are no resources in the state:

In TFC, however, we have a couple of resources created:

In our local environment, when we've done the export, the state files were also downloaded:

To upload our state files, we must run the following command: spacemk import-state-files-to-spacelift.

Note: if you have changed the name of the stack in your terraform configuration, you will also need to modify the data.json file, to reflect the stack name, when importing the state.

After running the command, if we go back to our stack, we can see the state has been migrated successfully:

11. [Optional] Customization

While the Migration Kit does a great job when it comes to migrating to Spacelift, we understand that every migration is different and you may need to customize the functionality of the automation. To do that, you can easily implement your own commands and even your own exporter. More details about this can be found here.

Key points

This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud's new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.

To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.

This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.

Written by Flavius Dinu

DEV Community