Azure Data Factory is a strong data integration service. Within a single factory you get visual pipelines, a large connector library, parameterised datasets and linked services, schedule and tumbling-window triggers, and an integration runtime that can reach hybrid sources. For a team whose data lives inside Azure, ADF is often the right primary tool, and nothing here argues for replacing it.
The conversation gets interesting when Data Factory stops being the only thing in the room.
Where ADF's orchestration model runs out
ADF's scheduling and dependency primitives are built around the assumption that the work being coordinated is also in ADF. Once that assumption breaks, three gaps appear quickly.
-
Dependencies stop at the factory boundary. ADF activity dependencies (
dependsOn) resolve between activities inside the same pipeline. You can invoke one pipeline from another with the Execute Pipeline activity, but a Databricks job kicked off elsewhere, a Cloud Function on Google Cloud, or an external arrival event cannot participate as a first-class node in an ADF dependency graph. - Triggers fire from a clock, not from a graph. Schedule and tumbling-window triggers know when to run, but they do not natively express "run this pipeline only after that unrelated workload finishes successfully on another platform". Teams bridge the gap with custom code, a queue, or a third system that polls.
- Concurrency control is per pipeline, not per platform. ADF lets you cap a single pipeline's concurrent runs. It does not give you a global budget that says "no more than eight things running against this Databricks workspace, across all sources". When the same cluster is driven by ADF, a notebook job and an ad-hoc workflow, that budget has to live somewhere outside ADF.
Changing a schedule should not be a deployment
In ADF, a trigger is part of the factory's definition. Changing when a pipeline runs means editing the trigger and publishing the factory, the same path a pipeline change takes, and publishing restarts the trigger with the new definition. With Git integration and CI/CD, that is a pull request, a review and a deploy window. A reasonable amount of ceremony for a change to pipeline logic, and a lot for "move this job fifteen minutes earlier".
When schedules and dependencies are data rather than artifacts, the change is an edit. An operator moves the time in the UI and the scheduler picks up the new definition on its next cycle. No branch, no review, no deploy, and the audit trail is the record of the change itself rather than a commit that has to be released before it counts.
Monitoring stops at the factory door
ADF Monitor shows ADF runs, which is exactly the right scope when ADF is the only thing running. The moment a run depends on a Databricks job, a Logic App, a Fabric pipeline or a Cloud Function, "did the whole run succeed?" is a question no single console answers. Azure Monitor and Log Analytics can collect the pieces once diagnostic settings are wired for every resource, and each service still keeps its own notion of a run. The result is a status question answered by opening four monitors and comparing timestamps.
A control plane keeps one run history. Every task, whatever platform executes it, records a status, a duration and its place in the dependency graph, and a retry is traced back to the original attempt.
You pay for orchestration, including the waiting
ADF bills per activity run, an ordinary way for a consumption service to price coordination. The consequence worth understanding is that an activity which only orchestrates is an activity run like any other. A Lookup, a Wait, a Web call, each pass through an Until loop: they all count, whether or not a byte of data moves.
That makes the polling pattern expensive in a specific way. Using ADF to watch for something that happens elsewhere, such as a file landing or another platform's job finishing, means repeated activities that do no data work. The pipeline is billed for waiting. A control plane that waits on the other platforms' own tasks does so at the queue: a dependency resolves when the upstream run reports completion, and no platform activity is created in the meantime.
Metadata-driven pipelines and the glue-code bill
ADF can be made metadata-driven, and experienced teams do it: a control table is read by a Lookup, the rows drive a ForEach, an If or Switch chooses the activity, and the outcome is written back. It is a sound pattern and it is also a framework: a parent pipeline, a table schema, error handling and parameter contracts that someone owns and maintains, and that every organisation builds a slightly different version of.
The cost is not the first pipeline. It is the second platform, where the framework has to be rebuilt because ADF cannot iterate over a Databricks job or a Cloud Function as a first-class node. Then the third platform, and then the person who leaves.
What a control plane adds around ADF
Polysync treats ADF pipelines as first-class nodes in a larger dependency graph. It does not run the transformations; ADF still does that. It decides when ADF runs, what runs alongside it, and what waits for it.
- Cross-platform DAG. An ADF pipeline can depend on a Databricks job, which can depend on a Synapse pipeline, which can depend on a Google Cloud Function. Trigger conditions (on success, on failure, on completion, on skipped) are resolved recursively across the whole graph.
- Auto-discovery of pipelines. Polysync connects to the factory and discovers existing pipelines, exposing them as Jobs you configure into Tasks. No retyping names or parameters.
- Parameter mapping across platforms. A row count, a file path or a status produced by an ADF pipeline becomes an input to a downstream task on another platform, without a custom passthrough.
- Concurrency and rate limits across platforms. Concurrency profiles apply two kinds of control: a concurrency limit on how many runs may be in flight, and a rate limit (token bucket, leaky bucket or sliding window) on how many may start per period. A profile can be scoped to a single platform or shared across several, so an ADF run, a Databricks job and a Logic App invocation can all count against the same budget for the same downstream resource.
- One monitoring view. Every run across every connected platform appears in a single dashboard with status, duration and history.
- Standard cron with a visual builder. Schedules attach to root tasks in standard cron syntax with timezone awareness, and a task can carry more than one schedule.
- Changes without a deployment. Schedules, dependencies and parameters are configuration, so changing them is an edit rather than a release.
- Waiting without activity cost. A task waits for its upstream dependencies at the queue; the platform is only called once they are satisfied.
- No orchestration code to maintain. There is no parent pipeline or control-table framework to build, own and hand over.
Side by side
| ADF on its own | ADF with a cross-platform control plane |
|---|---|
| Dependencies inside a pipeline, not across platforms | ADF pipelines as nodes in a cross-platform DAG |
| Per-pipeline concurrency, no global budget | Concurrency budgets that span connected platforms |
| Triggers driven by clock or window, not graph state | Triggers driven by upstream task state, with trigger conditions |
| Monitoring scoped to the factory | One dashboard across the platforms in the flow |
| Hand-rolled glue when work spans services | Parameters mapped between tasks without bespoke code |
| A schedule change is a release | Schedules and dependencies change in the UI |
| Orchestration and polling are billable activity runs | Waiting happens at the queue, not in billable activities |
| Metadata-driven work needs a framework you maintain | The graph, parameters and schedules are the configuration |
Will I still need ADF? Yes
Polysync does not move bytes. If you use ADF today to copy from on-premises SQL into ADLS, that activity is still in ADF and still uses your integration runtime. Polysync calls the pipeline, waits for it, captures its result, and decides what runs next.
When ADF alone is still the right answer
If every pipeline you operate lives inside one ADF instance, your dependencies are simple, and you have no plans to bring Databricks, Functions, Logic Apps, Synapse, Fabric or Google Cloud services into the same flow, ADF's built-in scheduling is fine. Adding a separate orchestrator before you need one is overhead, not insurance.
The point at which most teams cross the line is the second platform: the first Databricks job, the first Cloud Function, the first workload that has to coordinate with an ADF pipeline. The full comparison, with the figures, is in the original article on our website: https://www.polysync.com.au/articles/adf-vs-polysync?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=adf-vs-polysync
About Polysync
Polysync Data Solutions is an Australian company that builds cross-platform orchestration for data, integration and AI workloads. It coordinates the platforms you already run rather than replacing them.
Microsoft, Azure, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Functions, Logic Apps and Microsoft Fabric are trademarks of the Microsoft group of companies. Other product names, logos and brands are the property of their respective owners.
Top comments (0)