Our pipeline ran Mongo → S3 parquet → Glue/Spark → Iceberg → Oracle ADB. Autonomous Database can read parquet from S3 natively, so the Spark step was mostly paying to move files.
We replaced it with this:
S3 parquet → DBMS_CLOUD_PIPELINE → landing table (all VARCHAR2) → PL/SQL MERGE every 5 min → curated table
It now covers 13 collections and 24 feeds, with tables up to ~30M rows. Row counts match Iceberg, freshness is about 10 minutes, and no Spark is involved.
Things that caught us, in case they save someone time:
The pipeline load is positional, with an exact column count. Schemaless sources produce files with different column sets, which fail with ORA-00913 or ORA-00947. We land every column as VARCHAR2(4000) and apply types by name in the MERGE.
TO_DATE('15-SEP-26','YYYY-MM-DD') returns year 0015 with no error. Classify each value's shape with a regex first, then parse it with FX.
A pipeline's location can't change while it runs, and a reset forgets which files were loaded. Regex locations loaded 0 files. Wildcards picked up every older matching folder. What worked was creating one pipeline per date folder ahead of time (tomorrow's at 22:00) and dropping old ones after a grace period.
Check completeness by outcome. Every 15 minutes, list the S3 files, compare them with the files the pipeline recorded as loaded, and load any missing file by name.
Full re-pulls re-stamp unchanged records. Guard updates and deletes on version, and reconcile on content rather than timestamps.
Is anyone else using DBMS_CLOUD_PIPELINE for date-partitioned S3 folders? I'm curious whether there's a cleaner pattern than one pipeline per day.
Top comments (0)