The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told that if you just pick the right tool—usually the one with the most GitHub stars—your medallion architecture will magically materialize into a self-healing, performant pipeline.
I’ve spent six years cleaning up the aftermath of this delusion. I’ve seen healthcare pipelines stall because a Postgres metadata database hit its connection limit, and financial reconciliation jobs fail because a Lambda function timed out during a state transition. You don't need a "universal" tool; you need the right tool for the specific failure mode you’re willing to debug. Here is the field guide to keeping your bronze, silver, and gold layers moving without losing your mind.
1. Airflow is for people who love debugging Python environments
If your organization has a massive infrastructure team dedicated solely to keeping the Airflow scheduler alive, fine. Use it. But for the rest of us, Airflow is a dependency hell machine. Between pip conflicts in your worker nodes and the inherent fragility of the scheduler’s heartbeat, you are spending 40% of your time managing the orchestrator instead of the data.
In a medallion pipeline, you want atomicity. Airflow doesn’t give you that; it gives you a task-based graph that fails halfway through a Bronze-to-Silver merge, leaving you to write custom cleanup logic that inevitably fails as well.
# The classic Airflow "oops" - dependency bloat
from airflow.operators.python import PythonOperator
# Oh, your task needs pandas 1.5 but the DAG next door needs 2.0?
# Good luck with your virtualenv hell.
Photo by Manuel Luikenga on Unsplash
2. Step Functions are the ultimate "Set and Forget" for AWS shops
If your Medallion architecture lives in S3 and interacts with Glue or EMR, AWS Step Functions are the only orchestrator that doesn't feel like a side project. You aren't managing a server; you're managing a state machine. The beauty here is Wait for Callback. When I trigger a long-running Spark job, the state machine enters a paused state until the job reports back via an API call.
Failure mode: You will eventually hit the execution history limit. If you have a pipeline that iterates through 10,000 files, you will blow past the 25,000-event limit for a single execution. Chunk your data, or you’ll be staring at a cryptic ExecutionLimitExceeded error in the middle of a Friday deployment.
{
"Type": "Task",
"Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
"Parameters": {
"ClusterId.$": "$.ClusterId",
"Step": { ... }
},
"Retry": [{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 30,
"MaxAttempts": 3
}]
}
3. Databricks Workflows is the only "Native" choice
If you are running a Medallion pipeline on Delta Lake, stop using an external orchestrator. Databricks Workflows (the Jobs API) is built for this. It handles the cluster lifecycle, the notebook/JAR execution, and—most importantly—the underlying Delta commits.
When you use an external tool like Airflow, you are disconnected from the cluster's health. When you use Databricks Workflows, the orchestration is aware of the cluster status. If a node goes down, the job retries efficiently. You avoid the "orphan cluster" problem where your orchestrator thinks a job succeeded, but the cluster died during the final VACUUM command.
4. Don't build "God DAGs"
The biggest mistake I see in medallion architectures is the monolithic pipeline. A single DAG that runs Bronze, Silver, and Gold in one long serial chain is a ticking time bomb. If your Gold transformation fails, you have to restart the whole damn thing, effectively re-processing Bronze and Silver again.
Use decoupled triggers. Bronze completes, it emits a FileArrived event (or a Databricks Job completion signal), and that triggers Silver. Use a simple event-driven model. If Silver fails, you fix the logic and trigger Silver again. You don't touch the Bronze layer.
5. The Metadata Store is your source of truth, not the Orchestrator
Never use your orchestrator’s database to track your data quality. I’ve seen people write custom Airflow variables to track last_processed_timestamp. This is a disaster waiting to happen when you need to backfill or re-run a day of data.
Instead, keep your state in the data itself. Use a _metadata table in your Delta lake.
-- Pattern: Watermarking in the Medallion layer
SELECT * FROM bronze.events
WHERE ingestion_timestamp > (
SELECT MAX(processed_until) FROM silver.watermark_table
);
By keeping the state in the table, the orchestrator becomes "dumb." It just says "Run the job," and the job asks the metadata table "Where did we leave off?" This makes your pipeline orchestrator-agnostic. You could rip out Airflow and replace it with a Cron job, and your data wouldn't care.
Photo by Fahmi Anwar on Unsplash
6. Observability is not "Success/Failure"
Stop measuring success by the green checkmark in the UI. A job can finish "successfully" while loading empty data or corrupting a partition. Your orchestrator needs to gate-keep based on Data Quality (DQ) thresholds.
In Databricks, use dbt or Great Expectations as a task in your workflow. If the DQ check fails, the job fails. Don't let the pipeline proceed to the Gold layer if the Silver layer has null IDs.
# Databricks Workflows JSON config
tasks:
- task_key: "dq_check"
notebook_task:
notebook_path: "/Tests/Silver_Validation"
- task_key: "gold_load"
depends_on:
- task_key: "dq_check" # This is your primary circuit breaker
Conclusion
The orchestrator you choose matters less than how you decouple your logic. If you are deeply invested in Databricks, use their native Workflows—it’s the path of least resistance. If you are doing multi-cloud, multi-service orchestration, use Step Functions for their durability. If you are in a massive Python-heavy org that already has an Airflow platform team, stay there, but keep your pipelines granular and stateless.
The real question isn't "Which tool is best?" but rather: how much of your pipeline logic is stuck in your orchestrator, and how fast can you delete it when the tool inevitably goes out of fashion? Are you building a data platform, or are you just building a very expensive Airflow configuration file?
Cover photo by K. Mitch Hodge on Unsplash.
Top comments (0)