DEV Community

Cover image for Moving Storage from AWS to GCP: S3, Disks and File Shares
CobuildX AI profile image Emmanuel R
Emmanuel R for CobuildX AI

Posted on Edited on Originally published at cobuildx.ai

Moving Storage from AWS to GCP: S3, Disks and File Shares

#devops #googlecloud #aws #cloud

A worked AWS to Google Cloud storage migration: 400 TB of S3 with 2 billion small objects and a Deep Archive tier, a Postgres on EBS and an EFS share. Egress and per-object cost math, Storage Transfer Service tradeoffs, storage class traps, checksum verification, and a delta-sync cutover that fits a 30-minute freeze.

You have 400 TB in S3 across four buckets, two billion of those objects are thumbnails under 20 KB, 90 TB sits in Glacier Deep Archive for compliance, a Postgres primary lives on gp3 volumes, and twelve render workers share an EFS mount. The GCP contract is signed. The first instinct is to start gcloud storage rsync on a big VM and wait.

That instinct gets the order wrong. Terabytes decide how long the bytes take to move, but they don't decide the plan. The object count decides the bill and the timeline. The archive tier decides what you must do on AWS before any copy can start. The cutover window decides how you handle deletes. This post works through one representative workload end to end and makes each of those decisions with numbers.

This guide covers storage. For EC2, EKS and GPUs, see moving compute from AWS to GCP. For Lambda, Fargate and EventBridge, see moving serverless from AWS to GCP. For Redshift, EMR and Glue, see moving big data from AWS to GCP. Going the other way? Read moving storage from GCP to AWS.

The workload we'll move

This is an illustrative workload, not a customer. The shape is common for a media-heavy SaaS product: a few large buckets, one bucket with an enormous object count, a cold archive, a database and a shared file system. Source is us-east-1, target is us-central1. All prices below are list prices as of November 2026 for those two regions.

Dataset AWS today Size Objects Notes
media-originals S3 Standard 150 TB 40 M (avg 3.75 MB) Read and written by the app; many objects uploaded with multipart
thumbs S3 Standard 30 TB 2.0 B (avg 15 KB) Served through a CDN; keys are thumbs/{userId}/{photoId}_{size}.webp
exports S3 Standard-IA 130 TB 6 M (avg 22 MB) Customer exports and backups; lifecycle deletes them at 400 days
audit-archive S3 Glacier Deep Archive 90 TB 9 M (avg 10 MB) Seven-year compliance retention; read maybe once a year
Postgres EBS gp3, primary and replica 2 x 6 TB volumes, 3.5 TB of data n/a 12,000 IOPS and 500 MiB/s provisioned per volume
Render share Amazon EFS 12 TB Tens of millions of files NFS mount on 12 workers

Constraints: four platform engineers, and the business accepts one write freeze of at most 30 minutes per application. Everything else has to happen while production runs.

The target services are not the hard part. They map closely, and the figure below is the whole mapping. The rest of this post is about the decisions the mapping doesn't make for you.

Storage service map from AWS to Google Cloud: Amazon S3 to Cloud Storage, EBS to Persistent Disk or Hyperdisk, EFS to Filestore, FSx for NetApp ONTAP to Google Cloud NetApp Volumes, FSx for Windows to NetApp Volumes SMB, and AWS Backup to Backup and DR Service, with what to watch for on each

Smaller pieces map simply: instance store becomes Local SSD, FSx for Lustre becomes Managed Lustre, and the aws s3 command line becomes gcloud storage.

Storage decision flow, one dataset at a time: if it is files in a bucket, use Cloud Storage with Autoclass or a class chosen by read frequency; if it is a disk for one machine, use Hyperdisk or Persistent Disk, whichever the machine series takes; if Linux machines share a folder, use Filestore sized from real usage; if it needs SMB or NetApp features, use NetApp Volumes with NFS, SMB and Active Directory; backups go to Backup and DR Service with locked vaults in a separate project. Stop at the first yes

Where the money goes

Most migration budgets start and stop at egress. For this workload, egress is the line you can most easily make zero, and the line nobody budgets for is per-object operations.

Egress: three ways to pay for 400 TB

AWS charges for data transfer out to the internet in us-east-1 at $0.09 per GB for the first 10 TB in a month, $0.085 for the next 40 TB, $0.07 for the next 100 TB and $0.05 beyond 150 TB, after a free 100 GB per month [19][20]. If all 400 TB leaves in one billing month, that's about $23,800. Spread the same copy over two months and the tiers reset, which pushes it to about $27,600.

Path Cost for 400 TB What it costs you instead Pick it when
Storage Transfer Service over the public internet, with the AWS exit credit $0 egress after the credit Eligibility rules and a deadline to finish You are moving all your data off AWS, or all data for a service such as S3
Storage Transfer Service over the Google-managed private network $0.03 per GiB in North America, about $11,200; no AWS internet egress Shared pooled bandwidth that can throttle at peak; console or REST API only You don't qualify for the credit, for example you keep some S3 data on AWS
Storage Transfer Service over the public internet, no credit About $23,800 to $27,600 Nothing else, but it's the most expensive Small partial moves where neither of the above applies

The exit credit is the cheapest path, and its terms are what shape the rest of the plan. AWS waives data transfer out to the internet for eligible customers moving off AWS. You request it through AWS Support before the copy starts. AWS's announcement post says the waiver applies when you move all your data from all eligible accounts, or all your data for a particular service across those accounts. It currently gives customers outside the EU 180 days to finish, after earlier versions said 60 and then 90, so get the window confirmed in writing for your account. EU and UK customers follow the AWS EU Data Act and UK switching addenda instead [18].

The Google-managed private network option is the fallback. Storage Transfer Service routes the copy over private links that Google provisions, and you pay no S3 internet egress. You do pay a Google-side fee per GiB, plus the normal S3 request charges. It only works for agentless S3 transfers from listed AWS regions (us-east-1 is one), and it can't be configured from gcloud or the client libraries [3][4].

Operations: the two billion objects

Every object Storage Transfer Service writes is at least one Class A operation on Cloud Storage. In a regional bucket, Class A costs $0.005 per 1,000 operations in Standard, $0.01 in Nearline, $0.02 in Coldline and $0.05 in Archive [1]. Run the thumbnails through that:

Landing class for thumbs Class A writes for 2.0 B objects Cost per TB moved
Standard $10,000 About $333
Nearline $20,000 About $667
Coldline $40,000 About $1,333
Archive $100,000 About $3,333

Compare that with media-originals: 150 TB in 40 million objects costs $200 to write in Standard, about $1.33 per TB. The thumbnails are 7.5% of the bytes and 93% of the Cloud Storage write bill. On the AWS side the same objects are cheap to read: GET requests cost $0.0004 per 1,000, so two billion GETs are about $800, and the two million LIST pages needed to enumerate them are about $10 [19].

That table also kills a tempting idea. A team that wants to save on storage might land cold thumbnails straight into Coldline or Archive. The write bill alone would be four to ten times the Standard figure, before the minimum storage durations even start.

The real alternative is not copying them. Regenerating every thumbnail on GCP from the originals still writes two billion objects, so it saves nothing. Regenerating them lazily, on the first request after cutover, writes only the thumbnails people actually look at. If 20% get requested in the first quarter, that's 400 million writes, about $2,000. The cost is a cache-miss storm on launch day and a thumbnail service that must already run on GCP before cutover.

Our call for this workload: copy the thumbnails. $10,000 is less than the engineering time to make lazy generation safe under launch-day traffic. The decision flips if the object count were ten times larger, or if the thumbnail service already handles misses gracefully and access is concentrated on recent uploads.

The archive: restore before you can copy

Storage Transfer Service cannot read objects that are still archived. It skips S3 Glacier Flexible Retrieval objects by default, and it transfers Deep Archive objects only after they have been restored [3]. A job pointed at audit-archive without a restore will finish quickly, report success, and copy close to nothing. Log skipped objects (--log-action-states=failed,skipped) and check the counts.

There are two honest options for the 90 TB.

Option One-time cost Ongoing cost What it really costs you
Leave it in Deep Archive until retention ends $0 About $89 per month at $0.00099 per GB-month An AWS account, IAM and billing kept alive for years, and S3 no longer fully moved, which can cost you the exit credit on the other 310 TB
Restore with Bulk retrieval and copy into Archive class About $225 retrieval ($0.0025 per GB), about $9 of Batch Operations, about $450 of Archive Class A writes About $108 per month at Archive list price Up to 48 hours per restore wave, plus paying for the temporary restored copies while they exist

We'd restore and copy. The archive costs about $700 to move, and leaving it behind puts a five-figure egress credit at risk. The decision flips if retention ends within a year and you're keeping other workloads on AWS anyway.

Mechanics matter here. Deep Archive Bulk restores typically finish within 48 hours, and S3 accepts restore requests at 1,000 per second per account, so 9 million individual RestoreObject calls from a script will throttle. S3 Batch Operations, driven by an S3 Inventory manifest, uses the full request rate for you [21][22]. Restored copies exist for the number of days you set, and you pay for them on top of the archived object [23]. Restore in waves by prefix, with ExpirationInDays set just long enough for the copy to finish.

# Restore one wave of audit-archive with Bulk retrieval, driven by an S3 Inventory manifest
aws s3control create-job --account-id 111122223333 --region us-east-1 \
  --operation '{"S3InitiateRestoreObject":{"ExpirationInDays":7,"GlacierJobTier":"BULK"}}' \
  --manifest '{"Spec":{"Format":"S3InventoryReport_CSV_20161130"},
               "Location":{"ObjectArn":"arn:aws:s3:::acme-inventory/audit-archive/2026-11-01/manifest.json",
                           "ETag":"<manifest-etag>"}}' \
  --report '{"Enabled":true,"Bucket":"arn:aws:s3:::acme-inventory","Prefix":"restore-reports",
             "Format":"Report_CSV_20180820","ReportScope":"FailedTasksOnly"}' \
  --priority 10 --no-confirmation-required \
  --role-arn arn:aws:iam::111122223333:role/batch-restore
Enter fullscreen mode Exit fullscreen mode

One more AWS-side charge: reading Standard-IA data costs $0.01 per GB, so copying the 130 TB exports bucket adds about $1,300 in retrieval fees [19].

Choosing the copy tool

How the data moves: S3 buckets are copied to Cloud Storage by Storage Transfer Service with no agent; event-driven transfers keep them in sync through S3 notifications on SQS; EFS and FSx shares are copied by Storage Transfer Service agents or NetApp SnapMirror; disks move with their VMs through Migrate to Virtual Machines; a final sync runs before the switch

Tool Cost Good at Reject it when
Storage Transfer Service, agentless No service fee for agentless transfers; you pay operations and egress Managed retries, checksum validation, skip-if-unchanged, logs, event-driven sync You need to transform objects in flight
gcloud storage rsync or rclone on VMs VMs and your time Small buckets, ad-hoc fixes, renaming or repacking during the copy Billions of objects: you now own retries, parallelism, checksums and restarts
Transfer Appliance Appliance fees and shipping Data in your own data centre with a thin network link The data is already in AWS: there's no way to plug it into an AWS region, so it doesn't apply
Cloud Storage XML API with HMAC keys Free to set up Pointing existing S3 tools at storage.googleapis.com during the move As the end state: some S3 features and SDK defaults don't carry over

Storage Transfer Service is the default for all four buckets [2][4]. Read access on AWS should be a federated IAM role, not a long-lived access key:

# aws-role.json: {"role_arn": "arn:aws:iam::111122223333:role/gcs-transfer-reader"}

# Bulk copy of the originals, skipping anything still archived
gcloud transfer jobs create s3://acme-media-originals gs://acme-media-originals \
  --source-creds-file=aws-role.json \
  --include-storage-classes=STANDARD \
  --custom-storage-class=STANDARD \
  --overwrite-when=different \
  --log-actions=copy,find --log-action-states=failed,skipped

# Exports land directly in Nearline (see the class section for why)
gcloud transfer jobs create s3://acme-exports gs://acme-exports \
  --source-creds-file=aws-role.json \
  --custom-storage-class=NEARLINE \
  --overwrite-when=different
Enter fullscreen mode Exit fullscreen mode

How long it takes: bytes versus objects

Bytes first. If the transfer sustains 10 Gbps in aggregate, that's about 4.5 TB per hour, so 400 TB is about 3.7 days of pure byte movement. Treat 10 Gbps as an assumption to test in the pilot, not a promise.

Objects second, and for thumbs this is the number that matters. Cloud Storage starts each bucket at roughly 1,000 object writes per second. It scales up automatically, but Google's guidance is to grow load no faster than doubling every 20 minutes, and sequential key names slow that scaling down. With a random prefix, a bucket can reach roughly 16,000 writes per second [5].

Sustained write rate into thumbs Time for 2.0 B objects
1,000 objects/s (a new bucket, no ramp) About 23 days
5,000 objects/s About 4.6 days
16,000 objects/s (well-spread keys, fully ramped) About 35 hours

These are Cloud Storage ceilings, not Storage Transfer Service guarantees. Google doesn't publish an object rate for the service, so measure it. Run the pilot on a single thumbs/{userId}/ range, read the objects-copied rate from the job, and multiply. Our keys start with a user ID, which spreads writes better than a timestamp prefix would. If your keys start with dates, split the job by prefix and run the parts in parallel so the writes land across the key range.

The practical result: bytes for the big buckets finish in days, and the thumbnail bucket sets the critical path. Start it first.

Storage classes: map by access, not by name

The class table is short. The traps are in what Storage Transfer Service doesn't carry over.

S3 class Cloud Storage class Minimum storage duration (S3 vs GCS) Read latency on GCS
S3 Standard Standard None vs none Milliseconds
S3 Standard-IA / One Zone-IA Nearline 30 days vs 30 days Milliseconds
S3 Glacier Instant Retrieval Coldline 90 days vs 90 days Milliseconds
S3 Glacier Flexible Retrieval Coldline or Archive 90 days vs 90 or 365 days Milliseconds, no restore step
S3 Glacier Deep Archive Archive 180 days vs 365 days Milliseconds, no restore step
S3 Intelligent-Tiering Autoclass None vs none Milliseconds

Every Cloud Storage class reads in milliseconds, including Archive. There is no restore step. And there's no 128 KB minimum billable object size like S3 Standard-IA and Glacier Instant Retrieval have [1][24].

Trap 1: lifecycle ages restart on migration day

Storage Transfer Service keeps Content-Type, user metadata and the source ETag. It doesn't keep the S3 last-modified timestamp, ACLs or object tags [6]. Every migrated object is born on the day it was copied. An age condition in a lifecycle rule counts from that birth.

For exports, that bites twice. If you land the bucket in Standard and rely on an age = 30 rule to move it to Nearline, you pay Standard for 30 days on 130 TB: about $1,300 more than landing it in Nearline directly. And the S3 rule that deletes exports at 400 days stops firing: objects that were 399 days old on AWS live another 400 days on GCP.

The fix is cheap for 6 million objects. Write the S3 last-modified time from the inventory into each object's customTime after the copy, and base lifecycle on days_since_custom_time. At Nearline Class A rates, 6 million metadata updates cost about $60. Don't do the same for thumbs: two billion updates would be another $10,000. Skip objects that are within 30 days of their deletion date entirely, because a Nearline object deleted early is still billed for 30 days.

resource "google_storage_bucket" "exports" {
  name                        = "acme-exports"
  location                    = "US-CENTRAL1"
  storage_class               = "NEARLINE"
  uniform_bucket_level_access = true
  public_access_prevention    = "enforced"

  # customTime carries the original S3 LastModified, so retention survives the move
  lifecycle_rule {
    condition { days_since_custom_time = 400 }
    action { type = "Delete" }
  }
}
Enter fullscreen mode Exit fullscreen mode

Trap 2: early deletion during the migration itself

Cloud Storage charges early deletion when an object in Nearline, Coldline or Archive is deleted, replaced or rewritten before its minimum duration. A rewrite to change storage class counts. A lifecycle class change doesn't, and Autoclass buckets never pay early deletion [1]. The migration risk is a second full run with --overwrite-when=always into a Nearline bucket, which pays 30 days of early deletion on everything it overwrites. Keep --overwrite-when=different.

Soft delete is on by default with a 7-day window, and time as a soft-deleted object counts toward the minimum. If a botched first run of thumbs has to be deleted and redone, the deleted copies stay billable for a week. Run the pilot before the full job, not after.

Trap 3: Autoclass ignores small objects

Autoclass charges a management fee of $0.0025 per 1,000 objects per month, but objects smaller than 128 KiB are not managed and not counted [1]. For media-originals, that's 40 million objects and about $100 a month, a fair price for unknown access patterns. For thumbs, Autoclass would do nothing: every thumbnail stays in Standard. Decide the class for small-object buckets yourself.

Access and events

Bucket policies don't convert line by line. Turn on uniform bucket-level access, keep public access prevention enforced, and rewrite each policy from what each principal actually does. A prefix-scoped S3 grant becomes an IAM binding with a condition:

# Before: S3 bucket policy statement
{ "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/thumb-renderer" },
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::acme-thumbs/thumbs/*" }

# After: uniform access plus a conditional IAM binding
gcloud storage buckets update gs://acme-thumbs \
  --uniform-bucket-level-access --public-access-prevention
gcloud storage buckets add-iam-policy-binding gs://acme-thumbs \
  --member=serviceAccount:thumb-renderer@acme-prod.iam.gserviceaccount.com \
  --role=roles/storage.objectViewer \
  --condition='expression=resource.name.startsWith("projects/_/buckets/acme-thumbs/objects/thumbs/"),title=thumbs-only'
Enter fullscreen mode Exit fullscreen mode

Explicit Deny statements keyed on aws:SourceVpce or source IP have no direct bucket-level equivalent. That intent usually moves to a VPC Service Controls perimeter around the project. Public website assets go behind Cloud CDN and a load balancer, not a public bucket.

Events have two gotchas, one on each side.

  • On AWS, adding the sync queue can fail. Event-driven transfers need S3 to send object-create events to an SQS queue [7]. If the bucket already sends s3:ObjectCreated:* for the whole bucket to a Lambda, S3 rejects a second configuration with an overlapping prefix for the same event type [25]. Fanning out through an SNS topic with raw message delivery is the usual workaround. Test it on the pilot bucket, because Google documents only direct S3-to-SQS delivery.
  • On GCP, don't create notifications before the bulk copy. A Pub/Sub OBJECT_FINALIZE notification on thumbs during the load fires two billion messages, and any subscriber, like a thumbnail generator or a search indexer, processes the whole history again. Create notifications as a cutover step, after the bulk copy is done. 
# Cutover step, not setup step
gcloud storage buckets notifications create gs://acme-media-originals \
  --topic=media-object-events --event-types=OBJECT_FINALIZE,OBJECT_DELETE
Enter fullscreen mode Exit fullscreen mode

The database on EBS: don't copy the disk

Disks for ordinary VMs move with their machines through Migrate to Virtual Machines, which replicates EBS volumes while the instance runs (the compute post covers this). A database is different, because what you want at cutover is a consistent database, not a consistent block device.

Option Cutover downtime What it costs you Pick it when
Block replication of the EBS volumes Stop Postgres, final sync, boot on GCP A cold start on the new disk, and the outage includes the last sync The database is small or downtime is cheap
Streaming replica on Compute Engine over HA VPN, then promote Seconds to drain lag, then promote You run the VPN, the replica build and the promotion runbook You stay on self-managed Postgres
Database Migration Service to Cloud SQL or AlloyDB Similar to a replica promotion A managed target with its own limits on extensions and settings You want to stop running Postgres yourself

For 3.5 TB, the initial copy is not the hard part. At an effective 1 Gbps over the VPN, 3.5 TB takes about 7.8 hours, and the primary keeps serving the whole time. What matters is that a replica turns the cutover into a lag check plus a promotion, which fits a 30-minute freeze with room to spare. We'd reject block replication here: it puts a full database stop inside the window.

Size the new disk from measurements, not from the gp3 settings. Pull the peak per-minute VolumeReadOps and VolumeWriteOps from CloudWatch over a busy month. Hyperdisk Balanced lets you provision IOPS and throughput separately from size, like gp3, so a 6 TB disk at 12,000 IOPS and 500 MiB/s maps directly [9]. Persistent Disk ties performance to size, so a small, fast gp3 volume can come out slower on pd-balanced unless you make it bigger. Newer machine series such as N4, C4 and C4A only take Hyperdisk, so pick the machine first.

One GCP option has no EBS match: Regional Persistent Disk and Hyperdisk Balanced High Availability keep a synchronous copy in two zones. It doesn't replace the Postgres replica, but it can replace snapshot-based zone recovery.

The EFS share: 12 TB of NFS

Filestore is the usual target for EFS, and the billing model changes. EFS bills for the bytes you store. Filestore bills for the capacity you provision, full or not [10]. With 12 TB used and 30% headroom, you pay for about 16 TB from day one. Zonal and Regional tiers can grow later, so start close to real usage and set an alert on free space.

The copy runs through Storage Transfer Service agents on EC2 instances that mount EFS. Agent-based file system transfers cost $0.0125 per GiB, about $140 for 12 TB [4]. Like S3, run a full pass while the workers keep writing, then incremental passes that move only changed files, then a final pass inside the freeze. For tens of millions of small files, the incremental pass is dominated by scanning, not copying. Time it in the pilot.

Other file systems: FSx for NetApp ONTAP moves to Google Cloud NetApp Volumes with SnapMirror, which keeps NetApp snapshots and clones. FSx for Windows File Server usually becomes NetApp Volumes SMB shares joined to Active Directory, since Filestore is NFS only. FSx for Lustre becomes Managed Lustre.

Proving the copy: checksums that actually compare

"The job succeeded" is not proof. Comparing hashes across clouds is harder than it looks, because the two sides don't store the same hash.

  • S3 ETags are MD5 only sometimes. For single PUT uploads with SSE-S3 or no encryption, the ETag is the MD5 of the data. For SSE-KMS or SSE-C, it isn't. For multipart uploads, it never is, regardless of encryption [26]. media-originals is full of multipart uploads.
  • Cloud Storage stores CRC32C on every object, and MD5 only on non-composite objects. Google recommends CRC32C for integrity checks [11].
  • Storage Transfer Service checks every object. It uses source checksums where S3 provides them. For multipart and SSE-KMS objects, it computes a checksum on the fly and fails the task with DATA_LOSS on a mismatch [8]. It also stores the source ETag on each object as x-goog-source-etag [6].

That last point gives you a cheap, independent check. S3 Inventory lists key, size and ETag for $0.0025 per million objects, about $5 for all of thumbs [19]. Load it and a listing of the Cloud Storage bucket into BigQuery and diff them:

-- s3_inv:  key (URL-decoded), size, e_tag       from S3 Inventory
-- gcs_inv: name, size, source_etag             from the bucket listing (metadata x-goog-source-etag)
SELECT
  COUNT(*)                                          AS s3_objects,
  COUNTIF(g.name IS NULL)                           AS missing_in_gcs,
  COUNTIF(g.size <> s.size)                         AS size_mismatch,
  COUNTIF(TRIM(g.source_etag, '"') <> TRIM(s.e_tag, '"')) AS etag_mismatch
FROM s3_inv AS s
LEFT JOIN gcs_inv AS g ON g.name = s.key;
Enter fullscreen mode Exit fullscreen mode

Two gotchas in that query. S3 Inventory writes keys URL-encoded in CSV, so a key with a space or a non-ASCII character won't join until you decode it. And normalise quoting on both ETag columns before comparing. On top of the diff, download a random sample, say 0.1% of multipart objects, from both sides and compare SHA-256. That's the only check that doesn't trust either provider's metadata.

The gates for each bucket: zero missing, zero size mismatches, zero ETag mismatches, zero failed objects in the transfer logs, and a clean sample. Nothing on AWS gets deleted until all five pass.

Cutover: delta sync and the deletes problem

Event-driven transfers keep creates in sync after the bulk copy: S3 sends object-create events to SQS and Storage Transfer Service copies each new or changed object. They don't propagate deletes. Deleting an object at the source doesn't delete it in the destination [7].

The textbook fix is a final full pass with --delete-from=destination-if-unique during the freeze. That needs a full listing of both sides. For thumbs that's two million LIST pages per side. If listing runs at 20 pages a second (an assumption, measure yours), one side takes about 28 hours. Even media-originals, at 40,000 pages, takes over half an hour at that rate. Neither fits in a 30-minute freeze.

So don't list during the freeze. Capture deletes as they happen instead. s3:ObjectRemoved:* is a different event type from s3:ObjectCreated:*, so a second notification to a separate deletes queue doesn't conflict with the sync queue. Start it on the same day as the bulk copy, and replay it after the final creates have drained:

import json, urllib.parse
import boto3
from botocore.exceptions import ClientError
from google.api_core.exceptions import NotFound
from google.cloud import storage

SRC = "acme-media-originals"
Q = "https://sqs.us-east-1.amazonaws.com/111122223333/originals-deletes"
s3, sqs = boto3.client("s3"), boto3.client("sqs", region_name="us-east-1")
dst = storage.Client().bucket("acme-media-originals")

while msgs := sqs.receive_message(QueueUrl=Q, MaxNumberOfMessages=10, WaitTimeSeconds=5).get("Messages"):
    for m in msgs:
        for rec in json.loads(m["Body"]).get("Records", []):
            key = urllib.parse.unquote_plus(rec["s3"]["object"]["key"])  # event keys are URL-encoded
            try:
                s3.head_object(Bucket=SRC, Key=key)
                continue  # deleted then re-created on AWS: keep the GCS copy
            except ClientError:
                pass
            try:
                dst.blob(key).delete()
            except NotFound:
                pass
        sqs.delete_message(QueueUrl=Q, ReceiptHandle=m["ReceiptHandle"])
Enter fullscreen mode Exit fullscreen mode

The head_object check matters. A key that was deleted and then re-created on AWS shows up in the deletes queue, and replaying the delete blindly would remove the good copy on GCP. For thumbs, a stale extra object is harmless because nothing lists the bucket, so we replay its deletes after cutover rather than inside the freeze.

With that in place, the freeze for each application is short and every step has a gate:

  1. Stop writes to the bucket, share or database. Put the app in read-only mode, not fully down.
  2. Drain creates. Wait until the sync queue shows zero visible and in-flight messages and the transfer job's last operation has no failures.
  3. Final file share pass. Run the last incremental EFS-to-Filestore pass, already timed in the pilot.
  4. Promote the database. Confirm replica lag is zero, then promote.
  5. Replay deletes from the deletes queue for buckets where stale objects matter.
  6. Diff the last day. Re-run the inventory diff on objects created since yesterday's inventory.
  7. Flip configuration to the GCS buckets, Filestore mount and new database endpoint, then create the Pub/Sub notifications.

Decide the rollback rule before the freeze, not during it. Storage Transfer Service writes into Cloud Storage and file systems, not into S3, so any writes that land on GCP after the flip must be copied back by your app or a tool like rclone if you roll back. Keep the rollback window short and named: for example, "roll back within the first hour on a 5xx rate above 1%, otherwise roll forward".

After cutover, watch four numbers against the pre-move baseline: Cloud Storage 429 and 5xx rates (the ramp-up limits apply to read traffic too), p99 origin latency behind the CDN, 404s on thumbs (missing objects show up here first), and daily Class A and Class B operation counts against the forecast.

Backups

AWS Backup becomes Backup and DR Service. It protects Compute Engine VMs, disks, Cloud SQL and more on a schedule, and stores copies in backup vaults with an enforced retention period, which plays the role of AWS Backup Vault Lock. Put the vault in a separate project from the workloads it protects.

Don't migrate old AWS backups as backups. Let them age out on AWS where retention allows, or restore and copy only what retention rules require you to keep, like audit-archive here. Start fresh backups on GCP before cutover day, so the first post-cutover restore test isn't the first backup.

What the plan costs, all in

Line item Basis Cost
AWS egress for 400 TB Exit credit approved $0 (about $11,200 via managed private network, or $23,800 to $27,600 over the internet without the credit)
S3 GET and LIST requests 2.06 B GETs at $0.0004 per 1,000, plus LIST pages About $830
Standard-IA retrieval 130 TB at $0.01 per GB About $1,300
Deep Archive restore 90 TB Bulk at $0.0025 per GB, plus Batch Operations About $235, plus temporary restored copies
Cloud Storage Class A writes Thumbs $10,000, originals $200, exports $60, archive $450 About $10,710
Exports customTime update 6 M Nearline metadata updates About $60
EFS agent transfer 12 TB at $0.0125 per GiB About $140
Total one-time, excluding egress About $13,300

Now the run rate. On AWS, the four buckets cost about $5,700 a month: 180 TB of S3 Standard at $0.023 then $0.022 per GB, 130 TB of Standard-IA at $0.0125, and 90 TB of Deep Archive at $0.00099. On GCP in us-central1, the same data costs about $5,000: Standard at $0.020 per GiB, Nearline at $0.010 and Archive at $0.0012 [1][19]. That's a saving of roughly $700 a month, and it's on a like-for-like basis that ignores units and discounts.

The honest reading: storage savings alone take about 19 months to pay back the one-time operations bill, and the operations bill is mostly one bucket's object count. If the move is justified, it's by the GCP commit and the rest of the platform, not by storage list prices. Know that before someone asks.

What we'd do first

  1. Turn on S3 Inventory for every bucket today. Count objects, not just bytes. The object count drives the write bill, the timeline and the verification plan.
  2. Ask AWS Support about the exit credit and get the window and scope confirmed in writing before the first byte moves.
  3. Reserve bucket names. They're global on both clouds, and the one you want may be taken.
  4. Start the archive restore early. It's the longest wait with the least effort.
  5. Pilot one prefix of the largest-object-count bucket. Measure objects per second, listing speed and the verification diff. Every other estimate in the plan comes from these numbers.
  6. Wire the deletes queue and the sync queue on the same day as the bulk copy starts.

Common questions

How do I transfer data from S3 to Google Cloud Storage?

Use Storage Transfer Service with a federated AWS IAM role. It reads S3 directly, validates checksums, skips unchanged objects on later runs and logs failures. Add an event-driven transfer through SQS to keep buckets in sync until cutover, and handle deletes separately, because event-driven transfers don't propagate them.

Does AWS charge to move data out to GCP?

Normally yes, per GB. AWS waives internet data transfer out for eligible customers moving all their data, or all data for a service, off AWS. Request it through AWS Support before you start, and confirm the completion window in writing: AWS's post currently says 180 days outside the EU. If you don't qualify, Storage Transfer Service's managed private network avoids S3 internet egress for a per-GiB fee.

Can Storage Transfer Service copy S3 Glacier objects?

Not while they're archived. Glacier Flexible Retrieval objects are skipped by default, and Deep Archive objects are copied only after a restore. Restore in waves with S3 Batch Operations, then copy.

What is the GCP equivalent of S3 Glacier?

Coldline and Archive. Both return data in milliseconds with no restore step. Coldline has a 90-day minimum storage duration and Archive has 365 days, longer than Deep Archive's 180.

Is Google Cloud Storage S3 compatible?

Partly. The XML API accepts many S3 requests with HMAC keys at storage.googleapis.com. It's a useful bridge during a move, but not every S3 feature works, so move code to the Google client libraries.

What is the GCP equivalent of EFS?

Filestore for NFS, and Google Cloud NetApp Volumes when you need SMB or NetApp features. Filestore bills for provisioned capacity, so size it from measured usage.

How do I verify an S3 to GCS migration?

Diff S3 Inventory against a Cloud Storage listing on key, size and the x-goog-source-etag metadata that Storage Transfer Service writes. Don't expect S3 ETags to match Cloud Storage MD5s, because multipart and SSE-KMS ETags aren't MD5. Sample-download a fraction of objects from both sides and compare SHA-256.

How MigrateX handles this

MigrateX

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. Automation does the repetitive work, and experienced engineers make the decisions this post walks through.

  1. Assessment. We pull inventories for every bucket, volume, share and backup, and count objects as well as bytes. We find the ties to AWS that break at cutover: hard-coded S3 URLs, notification wiring, bucket policies and long-lived access keys.
  2. Cost and timeline model. We model egress options, per-object write costs, restore costs and storage classes for your data, so the one-time bill and the run rate are known before anything moves.
  3. Migration in waves. Automation generates the Terraform for buckets, IAM and lifecycle rules, and sets up transfer jobs, sync queues and deletes queues. Our engineers own the parts that need judgement: the class decisions, the database cutover and the freeze runbook.
  4. Proof. Every bucket passes an inventory diff, a checksum sample and a failed-object check before anything on AWS is deleted.
  5. Handover. We retire AWS storage wave by wave, set up Backup and DR on GCP, and leave your team with storage they understand and own.

Planning a move from AWS to GCP? Tell us about your setup through Contact Us, and we'll start with a free conversation about where the object counts, archives and cutover windows put the real work.

References

Google Cloud

AWS

CobuildX helps teams migrate legacy applications and cloud infrastructure with MigrateX.

Originally published on the CobuildX blog.

Top comments (0)