Roughly 70% of production data pipelines in the healthcare and fintech sectors I’ve audited are effectively "zombie orchestrators"—they are running, but they are technically insolvent. They suffer from massive technical debt, opaque failure states, and cost structures that make your CFO weep. Most engineers pick an orchestrator based on what they read in a blog post from 2019, ignoring that the tooling landscape has fundamentally shifted.
I’ve spent the last six years debugging distributed deadlocks at 3 AM. I’ve watched multi-million dollar Spark jobs hang indefinitely because a dependency was misconfigured in Airflow. I’m here to tell you that the orchestrator you choose for your Medallion architecture (Bronze/Silver/Gold) isn't just a "preference"—it dictates your incident response time and your monthly cloud bill. Stop treating orchestration like a commodity. It’s the engine of your data product.
1. Don't use Airflow if you aren't paying for Astronomer
Apache Airflow is a fantastic framework for Python developers, but it is a bottomless pit of operational toil if you host it yourself. If your team is spending more time managing airflow-scheduler pods in Kubernetes than writing transformation logic, you have already lost. In fintech, we don't have the luxury of "debugging the scheduler."
If you insist on open-source Airflow, you’re stuck managing celery_worker concurrency and PostgreSQL metadata bloat. If you haven't manually purged the task_instance table because your metadata database hit 50GB, you haven't really lived. If you have the budget, use a managed service. If you don't, avoid Airflow entirely.
# The classic airflow trap: Task concurrency limits
# If you don't tune these, your workers will starve
[celery]
worker_concurrency = 16
task_instance_max_active_tasks_per_dag = 4
Photo by CHUTTERSNAP on Unsplash
2. Step Functions are for event-driven, not ETL
AWS Step Functions are beautiful for state machines. They are perfect for handling user onboarding flows or microservices orchestration where you need sub-second state transitions. They are terrible for heavy-duty Medallion pipelines.
Why? Because Step Functions were never built for the idiosyncratic retry logic of a Spark job. When you wrap a databricks-submit-run call in a Step Function, you lose the native integration with the Spark UI. You end up with a state machine that just waits for a cluster to finish, providing zero visibility into the actual data movement. Use Step Functions to trigger the ingestion, but get out of the way for the transformation.
3. Databricks Workflows is the "Goldilocks" choice for Medallion
If your Medallion architecture lives on Databricks, Databricks Workflows is now the default winner. The tight coupling between the job scheduler and the cluster lifecycle (especially with Serverless Compute) eliminates 90% of the cold-start and orchestration overhead Airflow forces on you.
The key benefit here is "Job Clusters." By using Job Clusters instead of All-Purpose clusters, you’re paying significantly less per DBU. Integrating this into your CI/CD pipeline via databricks-cli or terraform is trivial.
# Terraform snippet for a Databricks Job
resource "databricks_job" "medallion_silver_layer" {
name = "silver-layer-transformation"
job_cluster {
job_cluster_key = "shared_job_cluster"
new_cluster {
spark_version = "13.3.x-scala2.12"
node_type_id = "i3.xlarge"
num_workers = 4
}
}
task {
task_key = "transform"
notebook_task {
notebook_path = "/pipelines/silver_transform"
}
}
}
4. The "Bronze-to-Silver" dependency trap
The biggest mistake I see in Medallion architectures is over-orchestrating. You don't need a complex DAG to move data from Bronze to Silver if your transformation is idempotent.
If your Silver layer is just a Spark job reading from a Delta table, use dbt models run via Databricks Workflows. Do not write complex Airflow logic to check if data arrived. Use Delta Live Tables (DLT) declarative syntax. DLT handles the dependency graph internally. When you let the framework handle the graph, you stop writing glue code that eventually breaks.
5. Failure modes define your sanity
Airflow’s Sensor pattern is the silent killer. If you have a sensor checking for a file in S3 every 60 seconds, you are burning money. It’s a classic "distributed systems smell."
Contrast this with Databricks Workflows: when a job fails, the cluster terminates, you get an email, and the state is cleaned up. With Airflow, a worker node might get "zombie" status, holding onto resources while the UI says the task is running. If you are in healthcare, auditability is king. Databricks Workflows provides a native audit log of who ran what and when, which is far easier to present to compliance officers than a tangled collection of Airflow logs.
6. Version control is not optional
Whatever you choose, it must live in Git. If I see a team manually editing DAGs in an Airflow UI or clicking "Run Now" in the Databricks console without a corresponding PR, I know the pipeline is broken.
The move toward "Orchestration as Code" (using dbt-databricks or Terraform) is the only way to scale. If your orchestrator configuration isn't version-controlled, you don't have a pipeline; you have a collection of brittle scripts that will break the moment the senior engineer who wrote them goes on vacation.
Photo by Haberdoedas on Unsplash
7. The cost of visibility
You will eventually have a "Gold" table that is wrong. When that happens, you need to trace the provenance of the data back to the Bronze ingestion.
In Airflow, this requires complex XCom tracking and log scraping. In Databricks Workflows, Unity Catalog handles the lineage for you. The orchestrator is becoming less about "when to run" and more about "what did I run and where did it go." Unity Catalog integration with your workflows is non-negotiable in 2024.
Conclusion
Stop treating your orchestrator like a generic task runner. If you’re already in the Databricks ecosystem, stop fighting the platform and use Databricks Workflows. Airflow is a Ferrari: it’s beautiful, fast, and will cost you a fortune in maintenance if you aren't an expert mechanic. Step Functions are a reliable utility vehicle, but they aren't meant to race on the data track.
Pick the path of least resistance for your infrastructure team. Medallion architectures are complex enough without adding an unnecessary layer of "orchestration glue."
Are you managing your pipeline, or is your pipeline managing you?
Top comments (0)