DEV Community

Cover image for Moving Big Data from GCP to AWS: BigQuery, Dataproc and Dataflow
CobuildX AI profile image Emmanuel R
Emmanuel R for CobuildX AI

Posted on Originally published at cobuildx.ai

Moving Big Data from GCP to AWS: BigQuery, Dataproc and Dataflow

#aws #googlecloud #dataengineering #bigdata

A plain-English guide to moving big data from Google Cloud to AWS: BigQuery, Dataproc, Dataflow, Pub/Sub, Composer and the data catalog. What replaces what, what behaves differently, what changes on the bill, and a step-by-step plan.

"Big data" is the part of the cloud that stores and works on large amounts of data. It includes the data warehouse, the Spark jobs, the streaming pipelines, the message queues and the tools that run them on a schedule. For many teams on Google Cloud (GCP), this is the most valuable part of their setup. It is also the hardest part to move.

Teams move it to AWS for simple reasons. The rest of the company already runs on AWS. A company they bought came with an AWS data lake. Or their apps have already moved, and sending every event back to GCP costs too much.

On paper the move looks easy: BigQuery goes to Redshift, Dataproc goes to EMR, Pub/Sub goes to Kinesis. In real life, small things change. The SQL is a little different. Prices work differently. Messages arrive in a different order. Each of these can quietly break a report that people rely on.

"Data integration is making all the data an organization has available in all its services and systems."

— Jay Kreps, co-creator of Apache Kafka, in The Log

That is the goal of a big data move too. The data has to keep reaching every team and every report while it moves. This guide covers:

  • Which AWS service replaces each GCP big data service
  • What works differently, and what tends to break
  • How to move the data itself and prove nothing was lost
  • What changes on the bill
  • A simple step-by-step plan

The Quick Map

Here is each GCP big data service and its closest match on AWS. The rest of the post explains each row.

Big data service map from Google Cloud to AWS: BigQuery to Amazon Redshift or Athena on S3, Dataproc to Amazon EMR, Dataflow to Managed Service for Apache Flink or AWS Glue, Pub/Sub to Kinesis Data Streams or Amazon MSK, Cloud Composer to Amazon MWAA, and Dataplex to the Glue Data Catalog and Lake Formation

GCP Closest AWS match Watch out for
BigQuery Amazon Redshift (Serverless or provisioned) The SQL and the pricing are different
BigQuery over files in a lake Amazon Athena on S3, with Apache Iceberg tables You pay per TB read, so file layout matters
Dataproc Amazon EMR (on EC2, Serverless or EKS) gs:// paths and start-up scripts change
Dataproc Serverless EMR Serverless Different job settings and limits
Dataflow (Apache Beam) Managed Service for Apache Flink, AWS Glue or EMR No exact match; some jobs need a rewrite
Pub/Sub Kinesis Data Streams, Amazon MSK, or SNS + SQS Order, how long messages are kept, and push delivery differ
Cloud Composer Amazon MWAA Same Airflow, different operators and connections
Cloud Data Fusion AWS Glue (visual ETL) Pipelines are rebuilt, not converted
Datastream AWS DMS Change capture is set up differently
Dataplex / Data Catalog Glue Data Catalog, Lake Formation, SageMaker Catalog Access rules are rebuilt by hand
BigQuery ML Redshift ML or SageMaker AI Models are trained again, not copied
Looker Studio Amazon QuickSight Dashboards are rebuilt by hand
Bigtable Amazon DynamoDB or Keyspaces Different data model; plan it as its own project

The Warehouse: BigQuery to Redshift or Athena

BigQuery is Google's data warehouse: a place to keep large tables and run SQL on them. It is usually the centre of a GCP data platform, so it's the biggest decision. AWS has two good answers, and many teams use both:

  • Amazon Redshift is AWS's data warehouse. Use it for busy dashboards, many people running queries at the same time, and fast joins. Redshift Serverless feels closest to BigQuery: there are no servers to size, it grows and shrinks on its own, and you pay for the time it spends working.
  • Amazon Athena runs SQL directly on files stored in Amazon S3. Like BigQuery on-demand, you pay for the data each query reads. Use it for a data lake and for queries that run now and then. Store the tables in the Apache Iceberg format so you can update and delete rows.

A common setup: keep one copy of the data in S3 as Iceberg tables. Engineers query it with Athena and Spark. Redshift serves the dashboards that need to be fast, and it can read the same Iceberg tables, so you don't need two copies.

What changes in the SQL

Most simple queries move with small edits. The trouble is in features that only BigQuery has:

BigQuery Redshift Athena
STRUCT and ARRAY columns, UNNEST SUPER column type ROW and ARRAY; UNNEST works
Partitioned and clustered tables Sort keys and distribution keys Partitions and Iceberg sorting
SAFE_CAST, IFNULL CAST with checks, NVL TRY_CAST, COALESCE
project.dataset.table names database.schema.table catalog.database.table
Scripts and stored procedures Stored procedures Not supported; move logic to Airflow or Glue
Scheduled queries Redshift query scheduler or Airflow EventBridge or Airflow

Don't rewrite thousands of queries by hand. The AWS Schema Conversion Tool can convert BigQuery tables and much of the SQL for you. AWS also has a step-by-step BigQuery to Redshift guide. Treat the converted SQL as a first draft, and test every query behind an important report.

Rule of thumb: find the 50 queries that matter most (top dashboards, finance reports, ML features). Test those first, and compare their results row by row with BigQuery. Fix the rest as people use them.

Spark and Hadoop: Dataproc to EMR

Dataproc is Google's service for running Spark and Hadoop clusters. On AWS the same service is Amazon EMR. Both run the same open-source tools, so most jobs work on EMR with small changes:

On Dataproc On EMR
gs://bucket/path s3://bucket/path
Initialization actions (start-up scripts) Bootstrap actions
Cluster properties EMR configurations
Autoscaling policies EMR managed scaling
Spot secondary workers Spot task nodes
Dataproc Metastore Glue Data Catalog
Dataproc Serverless EMR Serverless

Pick the kind of EMR by how your jobs run. Clusters that run all day stay on EMR on EC2. Jobs that start, finish and stop are usually cheaper and simpler on EMR Serverless. If your team already runs everything on Kubernetes, EMR on EKS runs Spark on your existing cluster.

Match versions first. Use the same Spark and Python versions on EMR as on Dataproc. Otherwise small changes in behaviour will look like migration bugs.

Pipelines: Dataflow Is the Hard One

Dataflow is Google's service for running data pipelines written with Apache Beam. It handles the servers for you. AWS has no service that runs Beam in exactly the same way, so most of the engineering work goes here. There are three paths:

  • Keep the Beam code and run it on Flink. Beam pipelines can run inside Amazon Managed Service for Apache Flink. The code mostly stays the same, but your team has to learn some Flink tuning. This works well for streaming pipelines with a lot of logic.
  • Rewrite batch pipelines in Spark. Pipelines that run on a schedule often become AWS Glue jobs or Spark jobs on EMR Serverless. It's a rewrite, but Spark is widely known.
  • Write new streaming jobs for Flink directly. For simple streaming jobs, skip Beam and write them in Flink SQL.

Ready-made Dataflow templates (like "Pub/Sub to BigQuery") often need no code on AWS. Amazon Data Firehose can load a stream straight into S3, Iceberg tables or Redshift.

Count your Dataflow jobs early. They decide your timeline more than anything else. Sort them into three groups: "template, no code", "batch, rewrite in Spark" and "streaming, keep Beam or move to Flink".

Messaging: Pub/Sub to Kinesis, MSK or SQS

Pub/Sub is Google's message service: one system sends messages, and others receive them. Pub/Sub does several jobs at once. AWS splits those jobs across different services, so pick by how each topic is used:

How you use Pub/Sub Best AWS match Why
A stream of events for analytics Kinesis Data Streams Keeps order, lets you replay, feeds Firehose and Flink
Large streams, or teams that know Kafka Amazon MSK (managed Kafka) Works with Kafka tools; keeps data a long time
One message to many services SNS with SQS queues Simple fan-out
A work queue with retries Amazon SQS Simple, cheap, with a queue for failed messages
Pushing messages to a web address EventBridge or SNS Kinesis and SQS don't push

Three differences break the code that reads messages:

  • Order. Pub/Sub can keep messages in order by key. Kinesis keeps order inside each shard. SQS only keeps order in FIFO queues, which are slower.
  • How long messages are kept. Pub/Sub keeps them for up to 31 days. Kinesis keeps them for 24 hours by default and up to 365 days. SQS keeps them for up to 14 days.
  • Duplicates. On AWS, plan for the same message to arrive twice. Make sure handling a message twice does no harm.

Scheduling: Composer to MWAA

Cloud Composer runs Apache Airflow, the tool that starts jobs in the right order on a schedule. On AWS the same thing is Amazon Managed Workflows for Apache Airflow (MWAA). Your Airflow workflows (DAGs) move almost as they are. The work is in what they call:

  • Operators. Steps like BigQueryInsertJobOperator and DataprocSubmitJobOperator become the Redshift, Athena, EMR and Glue operators from the Amazon provider package.
  • Connections. The Google service account becomes an IAM role for MWAA. Passwords and keys go into AWS Secrets Manager.
  • Versions. Use the same Airflow version on MWAA as on Composer. Upgrade later. Doing both at once makes it hard to tell what broke.

During the move, one Airflow can run jobs on both clouds. That lets you move jobs one at a time instead of all at once.

Moving the Data Itself

Moving data from GCP to AWS: export BigQuery tables to Parquet in Cloud Storage, copy to S3 with AWS DataSync, load into Iceberg tables or Redshift, keep new data flowing with change data capture or dual writes, then compare row counts and checksums before switching readers

"Data outlives code."

— Martin Kleppmann, author of Designing Data-Intensive Applications

Your pipelines will be rewritten many times. Your data will stay. So move it with care. Old data and new data move in different ways:

  1. Export the old data. Export BigQuery tables to Parquet files in Cloud Storage. Parquet keeps column types and takes up little space.
  2. Copy it across. AWS DataSync can read straight from Google Cloud Storage and copy the files to S3.
  3. Load it. Register the files as Iceberg tables, or load them into Redshift with COPY.
  4. Keep new data flowing. While both clouds run, send new data to both. You can write from the apps to both sides, run the pipelines on both, or use AWS DMS to copy database changes as they happen.
  5. Check it. Every day, compare row counts and totals for every table on both sides. Switch readers only when they match.

Moving data out of GCP costs money. Google charges per GB for data leaving its network. For a large warehouse, ask Google about its free transfer offer for customers who are leaving, and ask AWS about migration credits, before you start copying.

Catalog and Access Rules

On GCP, who can see which data is set with IAM on datasets and tables, plus column and row rules in BigQuery and Dataplex. On AWS, the Glue Data Catalog lists your tables, and AWS Lake Formation holds the access rules. Athena, Redshift, EMR and Glue all follow them.

No tool converts these rules for you. Export your current rules, review them (many are old and too open), and rebuild them in Lake Formation. Use tags, so new tables get the right rules on their own.

A Simple Way to Decide

Big data decision flow: if it is a Dataflow pipeline, sort it into template, Spark rewrite or Flink; if it is Spark or Hadoop, EMR; if it is dashboards with many users, Redshift; if it is ad-hoc or lake queries, Athena on Iceberg; if it is a message stream, Kinesis or MSK; otherwise SQS or SNS

For each workload, ask these questions in order and stop at the first yes:

  1. Is it a Dataflow pipeline? Sort it first: template, Spark rewrite, or Flink.
  2. Is it a Spark or Hadoop job? EMR. Serverless for short jobs, EC2 for long-running clusters.
  3. Is it a table behind busy dashboards? Redshift. Start with Serverless.
  4. Is it lake data, or queries that run now and then? Iceberg on S3, queried with Athena.
  5. Is it a stream of events for analytics? Kinesis Data Streams, or MSK if your team knows Kafka.
  6. Is it a queue between services? SQS, with SNS to send to many.

What Changes on the Bill

Big data costs depend more on how you are charged than on list prices.

Topic GCP AWS
Pay per query BigQuery on-demand, per TiB read Athena, per TB read
Pay for capacity BigQuery editions (slots per hour) Redshift Serverless (RPU-hours) or fixed nodes
Commit for a discount Slot commitments Redshift reservations
Storage Cheaper after 90 days without changes S3 storage classes; Redshift managed storage
Spark Dataproc fee on top of VM price EMR fee on top of EC2 price; Spot for extra workers
Streaming Pub/Sub, per TiB of messages Kinesis per shard-hour or per GB; MSK per broker-hour
Moving data in or out Charged per GB leaving GCP Data coming into AWS is free

Two things catch teams out:

  • Athena charges for data read, just like BigQuery. Badly organised tables cost money in the same way. Split tables by date and merge small files before people start querying them.
  • Redshift Serverless charges for time, not data. One dashboard that refreshes every minute can keep it running all day. Set usage limits from day one.

Don't buy reservations in the first weeks. Pay as you go until usage is steady, then commit.

What It Looks Like in Code

A typical BigQuery query and the same query in Redshift:

-- BigQuery
SELECT user_id, item.sku, SAFE_CAST(item.price AS NUMERIC) AS price
FROM `shop-prod.sales.orders`, UNNEST(items) AS item
WHERE DATE(created_at) = CURRENT_DATE();

-- Redshift (items stored as a SUPER column)
SELECT o.user_id, item.sku, item.price::DECIMAL(18,2) AS price
FROM sales.orders AS o, o.items AS item
WHERE TRUNC(o.created_at) = CURRENT_DATE;
Enter fullscreen mode Exit fullscreen mode

A Dataproc cluster in Terraform, and the same cluster on EMR:

# GCP: a Spark cluster
resource "google_dataproc_cluster" "etl" {
  name   = "etl"
  region = "us-central1"
  cluster_config {
    master_config {
      num_instances = 1
      machine_type  = "n2-standard-8"
    }
    worker_config {
      num_instances = 2
      machine_type  = "n2-standard-8"
    }
    initialization_action { script = "gs://etl-scripts/setup.sh" }
  }
}

# AWS: the same cluster on EMR
resource "aws_emr_cluster" "etl" {
  name          = "etl"
  release_label = "emr-7.x.0"              # match your Spark version
  applications  = ["Spark"]
  service_role  = aws_iam_role.emr_service.arn
  ec2_attributes {
    subnet_id        = var.private_subnet_id
    instance_profile = aws_iam_instance_profile.emr_ec2.arn
  }
  master_instance_group { instance_type = "m7i.2xlarge" }
  core_instance_group {
    instance_type  = "m7i.2xlarge"
    instance_count = 2
  }
  bootstrap_action {
    name = "setup"
    path = "s3://etl-scripts/setup.sh"   # was an initialization action
  }
}
Enter fullscreen mode Exit fullscreen mode

These snippets show the shape, not every setting. Check the current docs before you copy them.

Things That Bite Later

"Failures are a given and everything will eventually fail over time."

— Werner Vogels, CTO of Amazon, in 10 Lessons from 10 Years of AWS

Plan for the small things that fail after the move, not just the big ones:

  • Time zones. BigQuery TIMESTAMP is always UTC. Redshift has two kinds of timestamp. Check every date filter in your top reports.
  • Small files. Streaming into S3 creates thousands of tiny files. That makes Athena slow and expensive. Merge them on a schedule.
  • Hidden gs:// paths. They hide in Spark settings, Airflow variables, notebooks and SQL. Search for all of them before the switch.
  • Key files. Google service account key files used by pipelines become IAM roles. Find every one, then delete the keys.
  • BI tools. Looker, Tableau and notebooks point at BigQuery. Every connection and saved query needs a new home.
  • Limits. New AWS accounts start with low limits for EMR, Kinesis and Glue. Ask for more weeks ahead.

Doing the Move, Step by Step

  1. List everything. Every dataset, table, query, pipeline, topic, Airflow workflow and dashboard, with its size, cost, owner and who reads it.
  2. Pick the AWS target for each one, using the questions above. Write down why.
  3. Build the base: an S3 data lake with Iceberg, the Glue Data Catalog, Lake Formation rules, Redshift, MWAA, a network link to GCP, and higher limits.
  4. Start with a small pilot. Move one data product end to end: its tables, pipeline, schedule and one dashboard.
  5. Copy the old data, then keep both sides in sync. Compare both sides every day.
  6. Switch readers one dashboard at a time, then switch the pipelines that write data.
  7. Clean up. Turn off GCP jobs, delete old copies once you're sure, and buy reservations only once AWS usage is steady.

"Scope it to one to two weeks, with a few tables and a few queries only."

— Saman Irfan, Solutions Architect at AWS, on the pilot for a BigQuery move, in Best practices for a BigQuery to Amazon Redshift migration proof of concept

Big data is often the hardest part of a cloud move. The services are not that different. The hard part is that so many people depend on the numbers. Move the data carefully, prove the numbers match, and switch readers one at a time.

Common Questions

What is the BigQuery equivalent in AWS?

There are two. Amazon Redshift (Serverless feels closest) is the data warehouse for busy dashboards and many users. Amazon Athena runs SQL on files in S3 and, like BigQuery on-demand, charges for the data each query reads. Many teams keep Iceberg tables in S3 and use both.

Is BigQuery similar to Redshift?

The idea is the same: large tables and SQL. The details differ. BigQuery STRUCT and ARRAY columns become SUPER, partitioning and clustering become sort and distribution keys, and functions like SAFE_CAST and IFNULL need rewriting. The AWS Schema Conversion Tool converts much of this as a first draft.

How do I migrate data from BigQuery to Redshift?

Export BigQuery tables to Parquet in Cloud Storage, copy the files to S3 with AWS DataSync, then load them with COPY or register them as Iceberg tables. Keep new data flowing to both sides, and compare row counts and totals every day before you switch readers.

What is the Dataflow equivalent in AWS?

There is no exact match. Streaming Beam pipelines can run on Amazon Managed Service for Apache Flink with little code change. Batch pipelines are usually rewritten as AWS Glue or EMR Serverless Spark jobs. Simple templates often become Amazon Data Firehose with no code.

Dataproc vs EMR: how hard is the move?

Usually easy. Both run the same Spark and Hadoop. Change gs:// paths to s3://, initialization actions to bootstrap actions, and Dataproc Metastore to the Glue Data Catalog. Keep the same Spark and Python versions so behaviour doesn't shift.

What is the Pub/Sub equivalent in AWS?

It depends on how you use it: Kinesis Data Streams or Amazon MSK for event streams, SNS with SQS for fan-out, SQS for work queues, and EventBridge or SNS for push delivery. Plan for different ordering, retention and duplicate messages.

How do I migrate Cloud Composer to MWAA?

Your DAGs move almost as they are. Swap the Google operators for the Amazon provider operators, move the service account to an IAM role and secrets to Secrets Manager, and keep the same Airflow version until the move is done.

About MigrateX

MigrateX

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. It combines automation with experienced engineers. The automation does the repetitive work quickly, and the engineers make the tricky decisions.

Here's how we run a GCP to AWS big data move with MigrateX:

  1. Assessment. We list every BigQuery dataset, query, Dataproc job, Dataflow pipeline, Pub/Sub topic and Airflow workflow, with its size, cost, owner and who reads the results. You see where the work is before anyone agrees on a timeline.
  2. Plan. We work with your team to pick the AWS target for each workload, design the data lake and access rules, and decide the order of the move.
  3. Migration, step by step. Automation does the repetitive parts: converting SQL, rewriting paths and operators, and turning Dataproc and Composer setups into EMR and MWAA. Our engineers handle the parts that need thought: Dataflow rewrites, data modelling, streaming and access rules.
  4. Proof that it works. Both clouds run side by side. We compare row counts, totals and report results every day, and only switch readers once AWS matches GCP.
  5. Handover and clean-up. We turn off GCP pipelines one at a time, set up reservations once usage is steady, and leave your team with a data platform they understand and own.

What you get along the way:

  • No frozen reports. Dashboards keep working while the data moves, one piece at a time.
  • Numbers you can trust. Daily side-by-side checks prove the data matches before anyone switches.
  • A smaller bill, not a bigger one. Good table layout, Spot, right-sized serverless and reservations are part of the move, not an afterthought.
  • One partner for the whole move. The same team and approach covers your data, your infrastructure and your apps.

Planning a move from GCP to AWS? We'd love to hear about it. Tell us a little about your setup through Contact Us, and we'll start with a free chat about where the work is likely to be.

References

AWS

Expert sources

Google Cloud

CobuildX helps teams migrate legacy applications and cloud infrastructure with MigrateX.

Originally published on the CobuildX blog.

Top comments (0)