If you run Spark ETL on AWS Glue and need to move it to Oracle Cloud, the good news is that most of it maps cleanly. The bad news is the few parts that don't, and they are exactly the parts Glue made easy. Here is the map I use.
Why teams move: cost
Most teams I talk to move for one reason: the bill. OCI list prices for compute, block storage and Autonomous Database are often lower than the AWS equivalents, and OCI includes the first 10 TB of outbound data transfer each month at no charge, which matters for pipelines that ship data out. Don't take my word for it, though: price your own workload. That is one reason the tool at the end of this post shows the monthly cost of every part before it builds anything.
The short version
| AWS | OCI | Notes |
|---|---|---|
| Glue ETL job (PySpark) | OCI Data Flow application | Managed Spark. You upload the script to Object Storage and run it. |
| S3 | Object Storage | Paths change from s3://bucket/key to oci://bucket@namespace/key. |
| Glue Data Catalog | OCI Data Catalog (as a Hive metastore) | Data Flow can use a Data Catalog metastore for your tables. |
| Glue crawlers | Data Catalog harvesting | Harvest a bucket to discover the schema. |
| Glue triggers / workflows | Airflow (or another scheduler) | Your DAGs keep working. Point the operators at Data Flow. |
| Lambda glue code around the job | OCI Functions | Small handlers move over with little change. |
What changes in the script
1. Drop the Glue wrappers. GlueContext, DynamicFrame and job.commit() are Glue-only. Data Flow runs plain Spark, so use a normal SparkSession and DataFrames.
# Before (Glue)
from awsglue.context import GlueContext
glue = GlueContext(SparkContext.getOrCreate())
df = glue.create_dynamic_frame.from_catalog(database="raw", table_name="events").toDF()
# After (Data Flow)
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("events").getOrCreate()
df = spark.read.parquet("oci://raw@mynamespace/events/")
2. Change the paths. Every s3:// path becomes oci://bucket@namespace/. Put the bucket and namespace in job arguments, not in the code, so the same script runs in dev and prod.
3. Replace job bookmarks. Glue bookmarks remember what was already processed. Data Flow has no built-in equivalent, so keep a small watermark yourself: a table or file with the last processed date or file name, read at the start of the run and written at the end.
4. Iceberg. If Glue was writing Iceberg tables, Spark on Data Flow can write them too. Add the Iceberg runtime to the application and point the catalog at Object Storage.
What stays the same
Your Spark logic. Joins, window functions, UDFs, partitioning: all of it is Spark, not Glue, and runs unchanged. In my experience the transformation code is the smallest part of the move. The paths, the catalog and the scheduling are where the work is.
Try the move before you commit to it
The hardest part of a migration is not the code, it is getting a place to try it. A ticket, a week, a compartment, and then nobody deletes it.
So I built a chat for that on OCI. You paste a mapping like the table above, or a Git folder with your DAG and Spark job. It builds the bucket, the Data Flow application, the Data Catalog, Airflow with your DAG loaded, and an Autonomous Database for the gold tables, shows you the monthly price of each part first, and deletes all of it when the sandbox expires (1 to 30 days).
- 3-minute demo: https://youtu.be/PGnDhkchqUY
- Free one-click install in your own tenancy: https://github.com/ashishsinha1602/oci-sandbox-factory
- Worked example (Airflow + Spark, "move this to OCI"): https://github.com/ashishsinha1602/oci-sandbox-factory/tree/main/examples/telemetry-pipeline
If you have moved Glue jobs to OCI and hit something this map misses, tell me in the comments. I'll add it.
Top comments (0)