DEV Community: Weslley Ribeiro

Renaming and Migrating Google Cloud Storage Buckets Using Storage Transfer Service

Weslley Ribeiro — Fri, 23 May 2025 14:38:02 +0000

Comparing Speed

gcloud storage cp

Average throughput: 14.1MiB/s

gcloud transfer jobs create

Average throughput: 604.96MiB/s

Google recommend the following guidelines:

Transferring less than 1 TB: Use gcloud cp.
Transferring more than 1 TB: Use Cloud Storage Transfer Service.

If you're moving more than 0.4 TB, STS is already a more efficient and safer choice.

Utilizing Storage Transfer Service (STS)

Migrating and renaming Google Cloud Storage (GCS) buckets using Storage Transfer Service is the fastest and safest strategy for large datasets.

Why STS?

Parallelization: Transfers multiple files simultaneously.
Resilience: Handles failures and retries automatically.
Scheduling: Allows for one-time or recurring transfers.
Metadata Preservation: Retains object metadata during transfers. you can learning more about this here.

In the tutorial, the challenge was to migrate a GCS bucket to a new project. However, using gcloud storage cp for large datasets can be slow and inefficient. We’ll solve this using STS.

Before Executing

Enable the Storage Transfer Service API in both projects.
The first transfer job (source → temp) is created in the source project.
Ensure the logged-in Service Account has the necessary roles assigned:

roles/storage.admin
roles/storagetransfer.admin
roles/iam.securityAdmin (optional, for IAM policy manipulation)

Executing First Script

Clone the GitHub repository to execute this tutorial. It contains 2 scripts you’ll need to fill in with your environment variables. Follow the included README.
Run the first script

bash 1_script_source_to_temp.sh

The script will:

Create a temporary bucket
Copy IAM policies from the original bucket to policies.json
Let you edit policies.json to keep existing permissions and add new ones
Apply all policies to the temporary bucket on confirmation
Assign roles to the Storage Transfer Service account to source and temp bucket
Create a job and initiate transfer from source to temporary bucket

To monitor detailed progress (percentage, transfer speed, files copied), use the Google Cloud Console:
Console: https://console.cloud.google.com/transfer/

Executing Second Script

temp and new original bucket are in the same project (new project) the job from temp to new origin will be create in the new project
You’ll need to fill in with your environment variables. Follow the included README.
Run the Second script

bash 2_script_temp_to_new_source.sh

The script will:

⚠️ After ensuring all data is transferred, delete the Original Bucket
Recreate the Bucket with the Original Name in the new project
Copy IAM policies from the temp bucket to policies.json
Let you edit policies.json to keep existing permissions and add new ones
Apply all policies to the new Original Bucket on confirmation
Assign roles to the Storage Transfer Service account to temp and new original bucket
Create a job and initiate transfer from temporary to new original bucket
⚠️ After ensuring all data is transferred, delete the temporary bucket

Conclusion

At the end, two transfer jobs will be created:

One in the source project (source → temp)
One in the destination project (temp → renamed bucket)

Both are one-time transfers using Storage Transfer Service, ensuring a high-speed, metadata-preserving, and downtime-free bucket migration.

Moving Google Cloud Scheduler between projects: A Safe Strategy

Weslley Ribeiro — Thu, 15 May 2025 17:45:03 +0000

🔄 Use Case

You have scheduled jobs using Cloud Scheduler in a project called Project A (e.g., develop), but now you want to migrate them to Project B, where you're centralizing the infrastructure resources of your application.

This approach follows Google's recommended best practices:

Centralization: makes maintenance and control easier.
Cost organization: improves billing tracking.
Automation: avoids manually recreating jobs one by one.

⚙️ Prerequisites

Clone the repo at repository it contains the script you’ll need to fill in with your environment variables. Follow the included README.

Edit script variables

At the beginning of the script, modify the values below:

SOURCE_PROJECT="source-project-id"
DESTINATION_PROJECT="destination-project-id"
SOURCE_LOCATION="source-region"         # e.g., us-central1
DESTINATION_LOCATION="destination-region"   # e.g., us-central1

Install required tools

gcloud CLI (authenticated in both projects with appropriate permissions)
jq (for JSON manipulation)

Required permissions

The account running the script must have permissions to:

List, describe, and pause jobs in the source project
Create and pause jobs in the destination project

🚀 What the script does

Lists all Cloud Scheduler jobs in the source project:

gcloud scheduler jobs list --project="$SOURCE_PROJECT" --location="$SOURCE_LOCATION"

For each job found:

Asks if you want to copy it (y/n)
If the job already exists in the destination, it notifies and skips
Otherwise:
- Exports the job data (gcloud scheduler jobs describe)
- Saves it to a file named schedulers.json (for backup/audit)
- Creates a new job in the destination with the same data (schedule, time zone, HTTP method, headers, body, retries, etc.)
- Pauses the job in the source project (to avoid duplicate execution)
- (Optional) Pauses the job in the destination if it was already paused in the source

📝 Script Structure

Fully interactive (confirmation for each job)
Saves original job JSON data in the schedulers.json file (for later review or re-execution)
Supports:
- httpTarget (including base64 body)
- Custom headers
- Retry settings (attempts, backoff, etc.)
- Paused jobs remain paused in the destination

💡 Important Tips

Carefully review each job before applying.
Use schedulers.json as a backup.
After migration, update any systems or variables that depend on job location (e.g., logging, monitoring, or IAM bindings).

📌 Practical Example

bash migrate_scheduler_jobs.sh

The script will list all available jobs.

For each one, it will ask:

wish copy job 'my-job' to project 'destination'? (y/n)

After copying, it will confirm:

✅ Job 'my-job' created successfully.

And it will pause the original job:

⏸️ Pausing job 'my-job' in source project...

📂 `schedulers.json` File

This file serves as an execution history, containing all exported jobs from the source project.

You can later use it to compare jobs between environments or even recreate them manually if needed.

✅ Conclusion

This script is a practical and safe solution to migrate Cloud Scheduler jobs between different projects in Google Cloud, maintaining configurations intact and reducing the risk of human error.

If you want to evolve this process into a CI/CD scenario, you can automate script calls using environment variables defined in your pipeline.

If you need help adapting or improving it, I can assist!

Moving and Renaming Google Cloud Storage Buckets Without Downtime: A Safe Strategy - part 1

Weslley Ribeiro — Fri, 25 Apr 2025 19:19:49 +0000

📦 My Scenario

Currently, I have a bucket located in a project called Project 1, which is serving the production environment of my application. As the project has grown, the goal is to move this bucket to Project 2, which contains all other infrastructure resources for my application in a centralized way.
This migration will follow Google's recommended best practices, centralizing all resources into a single project, which will provide:

Better cost tracking: Keeping all resources in one project allows for more efficient and precise cost control.
Easier maintenance: Consolidating infrastructure in a single project simplifies management and operations, reducing complexity.

🔍 Summary of limitations and alternatives for bucket migration as recommended by Google

Some bucket properties are permanent e cannot be changed after creation:
- Bucket name
- Geographic location
- Associated project

The process of "moving" or "renaming" a bucket involves creating a new bucket and migrating the data.

Strategy for empty buckets
- Delete the old bucket.
- Create a new bucket with the desired properties.

# Delete the empty original bucket
gcloud storage buckets delete gs://SOURCE_BUCKET

# Create the new bucket
gcloud storage buckets create gs://DESTINATION_BUCKET \
       --location=REGION \
       --project=PROJECT_ID

Strategy for buckets with data
- Create a new bucket with the desired properties.
- Copy the data from the old bucket to the new one.
- Delete the old bucket and its contents.

# Create the new bucket
gcloud storage buckets create gs://DESTINATION_BUCKET \
  --project=ID_DO_PROJETO \
  --location=LOCALIZACAO

# Recursively copy all objects
gcloud storage cp --recursive gs://SOURCE_BUCKET/* gs://DESTINATION_BUCKET

# Delete the original bucket and its contents
gcloud storage rm --recursive gs://SOURCE_BUCKET

Strategy to reuse the same bucket name
- Create a temporary bucket with a different name.
- Copy the data into this temporary bucket.
- Delete the original bucket.
- Create a new bucket with the same name as the original.
- Copy the data back from the temporary bucket to the new bucket.

# Create temporary bucket
gcloud storage buckets create gs://TEMP_BUCKET \
  --project=ID_DO_PROJETO \
  --location=LOCALIZACAO
# Copy data from original bucket to temporary
gcloud storage cp --recursive gs://SOURCE_BUCKET/* gs://TEMP_BUCKET
# Delete original bucket
gcloud storage rm --recursive gs://SOURCE_BUCKET
# Recreate the original bucket with the same name
gcloud storage buckets create gs://SOURCE_BUCKET \
  --project=ID_DO_PROJETO \
  --location=LOCALIZACAO
# Copy data back from temporary to new bucket
gcloud storage cp --recursive gs://TEMP_BUCKET/* gs://SOURCE_BUCKET
# Delete temporary bucket
gcloud storage rm --recursive gs://TEMP_BUCKET

Supporting documentation

🧪 Evaluation

Both the development and production buckets are in Project 1. I’ll start by applying the migration strategy in the development environment to fail safely before touching production.
I chose the strategy that keeps the same bucket name. Unfortunately, any option involves a tradeoff — in this case, a short downtime.
My application uses a CI/CD pipeline, and the bucket name is stored as an environment variable in GitHub Actions. So, theoretically, after making the changes in the cloud, I can just update the variable, and the application will start using the new bucket.

Before executing anything, always ask yourself: “What could go wrong?” By answering this, you’ll be more prepared to execute.

💡 A fourth custom strategy (no downtime)

While analyzing my scenario and Google's suggestion of preserving the bucket name, I identified the need for a fourth strategy that overcomes the downtime tradeoff with something more acceptable for my case.
The goal is to move the bucket while keeping the same name and avoiding downtime. Here's the proposed strategy:

Create a temporary bucket;
Copy data;
Point the application to the temporary bucket;
Delete the original bucket;
Recreate the bucket with the original name;
Copy data;
Point the application back to it;
Sync any final changes from the temporary bucket. This avoids downtime while accepting other operational tradeoffs that are easier to manage in my case.

✅ Main Advantage	⚠️ Main Tradeoff
Zero Downtime	Higher operational complexity

🚀 Executing

Map all points where the bucket is used I searched the code and found some hardcoded URLs from an old implementation. Since I’m keeping the bucket name, no changes are needed there.
Check the current bucket size to estimate the transfer duration and validate completeness

gcloud storage du -s gs://SOURCE_BUCKET

Returned:
2064031471

Unit	Value
Bytes	2.064.031.471 B
Megabytes	~1.968 MB
Gigabytes	~1,92 GB

This small size indicates that transfer won’t be a bottleneck.

Make sure to replicate configurations such as lifecycle rules, IAM policies, object visibility, ACLs, etc. Use this to inspect the original bucket:

gcloud storage buckets describe gs://SOURCE_BUCKET

Clone the repo at repository it contains 2 scripts you’ll need to fill in with your environment variables. Follow the included README.
Run the first script

The script will:
Create a temporary bucket
Copy IAM policies from the original bucket to policies.json
Let you edit policies.json to keep existing permissions and add new ones
Apply all policies to the temporary bucket on confirmation

Then it starts copying objects and folders

It was fast!

Now, it’s time to point your app to the temporary bucket while the script pauses. This avoids downtime since the temporary bucket is ready.

After updating the variables and redeploying with zero downtime:

I tested the app — the image uploaded successfully to the temporary bucket!

After confirming the application was using the temporary bucket, I resumed the script and it performed a final sync

With that, we conclude Part 1 of the strategy — successfully migrating the app to use a temporary bucket with no downtime. Go to part 2 to finish the transition: delete the old bucket, recreate it with the original name, and sync everything back.

Moving and Renaming Google Cloud Storage Buckets Without Downtime: A Safe Strategy - part 2

Weslley Ribeiro — Fri, 25 Apr 2025 19:19:11 +0000

To execute the strategy of moving a bucket without downtime, it is necessary to work with an interactive script to standardize the actions, as well as perform some application updates during the process.
In Parte 1 we understood the required steps to start the migration strategy.
After executing it, we need to analyze if any issues occurred — such as errors accessing old files, difficulties inserting new objects into the new bucket, or any unexpected application errors.
After completing all necessary validations, we can mitigate any concerns that deleting the old and temporary buckets could cause problems.

Completing the Migration Process

To execute this script, the account authenticated in the gcloud CLI must have the following permissions

Storage Admin or equivalent permissions in both the source and destination projects.

Execute Script 2 (which includes deletion actions)

Confirm the deletion of the old bucket

Notice that the script deletes the old bucket and creates a new one with the same name in the new destination project.

After validating and confirming the information inside the policies.json file, and pressing ENTER, the policies will be added one by one

The script then starts copying the files from the temporary bucket to the newly recreated bucket

Once the copy finishes, the script will prompt you to update your application to use the newly created bucket (if it is not already), and will wait until you press ENTER to proceed.

After updating the application, the script will perform a final sync to ensure consistency, and then prompt you to confirm the deletion of the temporary bucket.

Congratulations! 🎉
You have successfully completed a bucket migration with zero downtime.