DEV Community: Polysync

When native cloud schedulers are enough, and when not

Samir Elmallah — Fri, 16 Oct 2026 10:40:31 +0000

Every cloud platform ships with a scheduler. For work that stays inside that platform it is usually the right tool, and adding an orchestrator on top buys you a hop, a credential and a failure mode.

The trouble starts at the boundary, where one platform's scheduler has to wait on another platform's work and no single graph covers the run. Here are five checks for working out which side of that line you are on, and three ways across it if you are on the wrong side.

The five checks

You are still comfortably inside one platform when:

every step is a first-class job in the same platform, not an HTTP call;
a single dependency graph covers the whole run;
monitoring shows the whole run, not one platform's slice of it;
retries and idempotency are handled inside the platform;
adding a step does not mean inventing a new integration pattern.

You are past the edge when:

steps are chained with storage events, webhooks or polling timers;
a step is a service call rather than a job the scheduler understands;
"did the other platform's job finish?" is answered by a person, or by a table someone maintains;
lineage and run history stop at the platform boundary;
adding the next platform means building another bespoke bridge.

All five hold and the native scheduler is enough. Any one fails and you are about to start building glue. That is worth saying plainly: a scheduler that ships with the platform is not a compromise. There is nothing extra to run, retries happen where the failures happen, and the platform's own team keeps it working.

What glue actually looks like

Glue is not a product. It is the work that exists because one platform has to wait on another: a storage event that fires a function, a webhook that calls the next platform's API, a polling timer that asks whether the other job finished, or a lookup table somebody keeps current by hand.

Two properties matter before you build more of it:

Glue has no graph. The dependency lives in the event subscription or the timer, not in something anyone can draw.
Glue makes the run invisible at the edges. Each platform sees only its own slice, so retries, lineage and error handling end at the boundary, and it breaks quietly.

Scenario 1: one platform does the whole job

Picture a pipeline where every stage lives in Databricks. Data is ingested from cloud storage and message buses by the platform's declarative pipelines, transformations run in the same pipelines, and unstructured documents are parsed with an AI function:

SELECT ai_parse_document(content) FROM documents

The parsed output is indexed for search with Databricks AI Search, and analysts read the result in AI/BI dashboards. Lakeflow Jobs schedules the lot with one dependency graph, and the platform's monitoring shows every task in one place.

Run the five checks: all five hold. In this shape an external orchestrator removes nothing.

Scenario 2: the same pipeline, two platforms

Now mirror it. Ingestion in Azure Data Factory, document extraction on Azure Document Intelligence, transformation on Databricks, the search index in Azure AI Search, dashboards in Power BI.

Logically identical. Physically it crosses three or four boundaries. Data Factory can invoke a Databricks job directly and has a connector that writes to an Azure AI Search index, so those hops are expressible inside the platform. Document extraction has no first-class activity, so it becomes a Web call, a place where the dependency no longer resolves on its own. The Databricks job schedules its own tasks on its own clock, and Power BI refreshes on a third.

Run the five checks again. The graph no longer covers the run. Monitoring is split across four consoles. None of this is a defect in any one product. It is what a boundary does.

Three ways across, and when each is right

Event glue. Storage events, webhooks or a polling timer pick up where the last platform left off. Cheapest to start, and the cost shows up later as silent gaps and manual reconciliation.
A scheduler of schedulers. One job whose only task is to start other schedulers and wait. It works, but each platform still sees only its own slice, and the outer graph is a script one person owns.
A control plane that spans the platforms. Each platform stays the executor; a separate layer holds the dependency graph across them, maps parameters between tasks and keeps one run history.

A simple test settles it: try to answer, for one run, what ran, in what order, and what waited on what. If the honest answer needs four browser tabs, the boundary is costing more than an orchestrator would.

Where Polysync fits

Polysync is a control plane rather than a replacement for the platforms you already run. It treats pipelines, jobs and functions across Azure, Databricks, Fabric, AWS and Google Cloud as first-class nodes in one dependency graph, and decides what runs, when, and what waits. It does not move your data and does not change how any platform executes its work.

The article this is based on is the canonical home for this piece, with the figures: https://www.polysync.com.au/articles/native-cloud-schedulers?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=native-cloud-schedulers

Microsoft, Azure, Azure AI Search, Azure Data Factory, Azure Document Intelligence, Microsoft Fabric and Power BI are trademarks of the Microsoft group of companies. Other trademarks are the property of their respective owners.

Automating cross-platform data pipelines: a practical walkthrough

Samir Elmallah — Fri, 16 Oct 2026 10:39:33 +0000

Most teams do not have a transformation problem. Data Factory, Databricks, Functions, Logic Apps, Synapse, Fabric and Cloud Composer all do their respective jobs well. The pain shows up between them: the pipeline that finishes in the morning, the job that should follow it but instead runs on a fixed schedule and hopes the other one is done, the Cloud Function that nobody is sure ran at all.

Automating a cross-platform pipeline comes down to one question: what is ready to run now? Seven things have to be in place to answer it reliably, and this walkthrough sets them out in the order you would build them. The pattern is not tied to one orchestrator; each step ends with how it looks in Polysync, so the walkthrough doubles as a tour of the product.

One example runs throughout: a nightly customer refresh. An Azure Data Factory pipeline copies new customer records into Azure storage. A Databricks notebook builds a gold table from them. A Google Cloud Function enriches the new rows. A Power BI dataset publishes the result. Four services across three clouds, and before any of this is automated, three of them run on separate clocks and hope the others have finished.

Step 1. Connect each platform

Before anything can run, the orchestrator has to be able to call each platform for you. Two choices are involved, and they are easy to confuse.

How does the orchestrator authenticate? Every connection picks an authentication method. For the Azure services, one option is the Polysync Service Principal: a multi-tenant application that Polysync owns, which you grant a role to in your own tenant. There is no long-lived secret for you to hold or rotate. The other choices are your own service principal (with a secret or a certificate) and, on the other clouds, the platform's own methods such as an AWS access key or assume-role, or a Google service account key, application default credentials, workload identity federation or an impersonated service account.

Where does a supplied credential live? If you use your own credential, it is stored in a secret vault and referenced by name, rather than pasted into a configuration screen. Polysync reads from Azure Key Vault, AWS Secrets Manager, Google Cloud Secret Manager and HashiCorp Vault, and can also hold the credential in its own managed vault if you would rather not link one of yours. The value is read when a task is dispatched, not copied into the task's configuration.

Polysync connects to more than 20 services across Azure, AWS and Google Cloud, including Azure Data Factory, Microsoft Fabric, Synapse Pipelines, Databricks and Power BI, AWS Glue and Step Functions, and Google Cloud Composer, Dataflow and Vertex AI.

Step 2. Discover jobs instead of redefining them

Once a platform is connected, you should not have to retype what already lives there. Polysync reads the platform's own resources and imports them as Jobs: the ADF pipeline, the Databricks notebook, the Cloud Function, the Power BI dataset. The parameters each resource accepts come with it.

Keep two words apart. A Job is the resource on the platform. A Task is a configured run of that Job: parameters bound, dependencies attached, a schedule or trigger. The same Job can back several Tasks, so one notebook can serve both the nightly refresh and a one-off backfill without being copied.

In the example, the four platforms are connected and Polysync imports the copy pipeline, the notebook, the Cloud Function and the dataset refresh. Nothing is rewritten; the resources stay where they are.

Step 3. Wire the dependencies, not the clock

A pipeline is a set of relationships, not a set of times. "Run the notebook at 06:15 because the copy usually finishes at 06:10" encodes a guess that will be wrong on the one night it matters. Tell the orchestrator the relationship and let it work out the time.

Each edge in the graph carries a trigger condition:

On Success. Run only if the upstream task succeeded. This is the chain in the example: copy, then transform, then enrich, then publish.
On Failure. Run only if the upstream failed, which is where an alert or a compensation step belongs.
On Completion. Run whatever the outcome, which is where cleanup belongs.
On Skipped. Run only if the upstream was skipped by an earlier condition, which keeps branch-and-merge graphs correct.

Conditions are resolved across the whole graph, not just the immediate parent, so a skip deep in a branch behaves the way you would expect. A cycle is rejected before a run starts: an accidental loop between two tasks is reported as a configuration error, not discovered as a run that never finishes.

Step 4. Pass values between tasks

Useful pipelines hand values along. The copy pipeline knows how many rows it moved, and the notebook needs that count as a parameter. The Cloud Function writes a file and returns its path, and the Power BI refresh needs to know which partition changed. Without parameter mapping, that value travels through a shared file or a table that somebody maintains by hand.

Polysync maps an output of one task into a parameter of another, across platforms: the row count from the ADF pipeline becomes an input to the Databricks notebook, and the notebook's partition name becomes an input to the Cloud Function. The mapping is part of the task configuration, so it is visible and changeable without touching transformation code.

Step 5. Put governance on the queue

Every managed service has hard limits: Databricks job concurrency per workspace, Logic Apps action throughput, Cloud Functions concurrent executions. Parallel triggers turn those limits into HTTP 429 responses, dropped runs and unpredictable spend.

Two different controls put a ceiling on that, and they are worth keeping apart:

A concurrency limit caps how many tasks are in flight against the platform at once. New work waits until a slot opens.
A rate limit caps how many tasks are dispatched in a period of time (per second, minute or hour), using a token bucket, a leaky bucket or a sliding window.

A profile carries one or both, and because it is attached to a platform or a set of platforms you choose the scope: a separate budget per platform, or one budget shared across several. A busy Databricks workspace therefore cannot hold up work aimed at an unrelated Data Factory factory, and several services driving the same downstream database can be held to a single shared ceiling.

The queue is also where waiting happens: a task whose dependencies are not yet met waits there, rather than polling the platform and piling up activity.

Step 6. Schedule the roots

Schedules attach to root tasks, the ones with nothing upstream. They use standard cron with a timezone, and there is a visual builder if you would rather not count asterisks. A task can carry more than one schedule.

Everything downstream runs because the graph says it is ready, not because it was given its own clock. In the example, only the ADF copy is scheduled. The transform, the enrichment and the publish have no times of their own.

Step 7. Watch it in one place

The point of all of the above is to answer "what is happening right now" on one screen. Polysync shows queue depth, the tasks currently dispatched, recent run history, durations and status breakdowns across every connected platform, so a run that spans four services reads as one run instead of four consoles.

What you end up with

A pipeline that no longer depends on someone remembering which portal to open first, and a schedule that states a relationship rather than a hope. The graph triggers itself, the queue applies the limits, and one view tells the truth about what ran. The orchestrator becomes the system of record for what your data platform actually did last night.

The figures that go with each step, and the full walkthrough, are in the original article on our website: https://www.polysync.com.au/articles/data-pipeline-automation?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=data-pipeline-automation

About Polysync

Polysync Data Solutions is an Australian company that builds cross-platform orchestration for data, integration and AI workloads. It coordinates the platforms you already run rather than replacing them.

Microsoft, Azure, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Functions, Logic Apps and Microsoft Fabric are trademarks of the Microsoft group of companies. Other product names, logos and brands are the property of their respective owners.

ADF vs Polysync: when Azure Data Factory stops being enough

Samir Elmallah — Fri, 16 Oct 2026 10:39:20 +0000

Azure Data Factory is a strong data integration service. Within a single factory you get visual pipelines, a large connector library, parameterised datasets and linked services, schedule and tumbling-window triggers, and an integration runtime that can reach hybrid sources. For a team whose data lives inside Azure, ADF is often the right primary tool, and nothing here argues for replacing it.

The conversation gets interesting when Data Factory stops being the only thing in the room.

Where ADF's orchestration model runs out

ADF's scheduling and dependency primitives are built around the assumption that the work being coordinated is also in ADF. Once that assumption breaks, three gaps appear quickly.

Dependencies stop at the factory boundary. ADF activity dependencies (dependsOn) resolve between activities inside the same pipeline. You can invoke one pipeline from another with the Execute Pipeline activity, but a Databricks job kicked off elsewhere, a Cloud Function on Google Cloud, or an external arrival event cannot participate as a first-class node in an ADF dependency graph.
Triggers fire from a clock, not from a graph. Schedule and tumbling-window triggers know when to run, but they do not natively express "run this pipeline only after that unrelated workload finishes successfully on another platform". Teams bridge the gap with custom code, a queue, or a third system that polls.
Concurrency control is per pipeline, not per platform. ADF lets you cap a single pipeline's concurrent runs. It does not give you a global budget that says "no more than eight things running against this Databricks workspace, across all sources". When the same cluster is driven by ADF, a notebook job and an ad-hoc workflow, that budget has to live somewhere outside ADF.

Changing a schedule should not be a deployment

In ADF, a trigger is part of the factory's definition. Changing when a pipeline runs means editing the trigger and publishing the factory, the same path a pipeline change takes, and publishing restarts the trigger with the new definition. With Git integration and CI/CD, that is a pull request, a review and a deploy window. A reasonable amount of ceremony for a change to pipeline logic, and a lot for "move this job fifteen minutes earlier".

When schedules and dependencies are data rather than artifacts, the change is an edit. An operator moves the time in the UI and the scheduler picks up the new definition on its next cycle. No branch, no review, no deploy, and the audit trail is the record of the change itself rather than a commit that has to be released before it counts.

Monitoring stops at the factory door

ADF Monitor shows ADF runs, which is exactly the right scope when ADF is the only thing running. The moment a run depends on a Databricks job, a Logic App, a Fabric pipeline or a Cloud Function, "did the whole run succeed?" is a question no single console answers. Azure Monitor and Log Analytics can collect the pieces once diagnostic settings are wired for every resource, and each service still keeps its own notion of a run. The result is a status question answered by opening four monitors and comparing timestamps.

A control plane keeps one run history. Every task, whatever platform executes it, records a status, a duration and its place in the dependency graph, and a retry is traced back to the original attempt.

You pay for orchestration, including the waiting

ADF bills per activity run, an ordinary way for a consumption service to price coordination. The consequence worth understanding is that an activity which only orchestrates is an activity run like any other. A Lookup, a Wait, a Web call, each pass through an Until loop: they all count, whether or not a byte of data moves.

That makes the polling pattern expensive in a specific way. Using ADF to watch for something that happens elsewhere, such as a file landing or another platform's job finishing, means repeated activities that do no data work. The pipeline is billed for waiting. A control plane that waits on the other platforms' own tasks does so at the queue: a dependency resolves when the upstream run reports completion, and no platform activity is created in the meantime.

Metadata-driven pipelines and the glue-code bill

ADF can be made metadata-driven, and experienced teams do it: a control table is read by a Lookup, the rows drive a ForEach, an If or Switch chooses the activity, and the outcome is written back. It is a sound pattern and it is also a framework: a parent pipeline, a table schema, error handling and parameter contracts that someone owns and maintains, and that every organisation builds a slightly different version of.

The cost is not the first pipeline. It is the second platform, where the framework has to be rebuilt because ADF cannot iterate over a Databricks job or a Cloud Function as a first-class node. Then the third platform, and then the person who leaves.

What a control plane adds around ADF

Polysync treats ADF pipelines as first-class nodes in a larger dependency graph. It does not run the transformations; ADF still does that. It decides when ADF runs, what runs alongside it, and what waits for it.

Cross-platform DAG. An ADF pipeline can depend on a Databricks job, which can depend on a Synapse pipeline, which can depend on a Google Cloud Function. Trigger conditions (on success, on failure, on completion, on skipped) are resolved recursively across the whole graph.
Auto-discovery of pipelines. Polysync connects to the factory and discovers existing pipelines, exposing them as Jobs you configure into Tasks. No retyping names or parameters.
Parameter mapping across platforms. A row count, a file path or a status produced by an ADF pipeline becomes an input to a downstream task on another platform, without a custom passthrough.
Concurrency and rate limits across platforms. Concurrency profiles apply two kinds of control: a concurrency limit on how many runs may be in flight, and a rate limit (token bucket, leaky bucket or sliding window) on how many may start per period. A profile can be scoped to a single platform or shared across several, so an ADF run, a Databricks job and a Logic App invocation can all count against the same budget for the same downstream resource.
One monitoring view. Every run across every connected platform appears in a single dashboard with status, duration and history.
Standard cron with a visual builder. Schedules attach to root tasks in standard cron syntax with timezone awareness, and a task can carry more than one schedule.
Changes without a deployment. Schedules, dependencies and parameters are configuration, so changing them is an edit rather than a release.
Waiting without activity cost. A task waits for its upstream dependencies at the queue; the platform is only called once they are satisfied.
No orchestration code to maintain. There is no parent pipeline or control-table framework to build, own and hand over.

Side by side

ADF on its own	ADF with a cross-platform control plane
Dependencies inside a pipeline, not across platforms	ADF pipelines as nodes in a cross-platform DAG
Per-pipeline concurrency, no global budget	Concurrency budgets that span connected platforms
Triggers driven by clock or window, not graph state	Triggers driven by upstream task state, with trigger conditions
Monitoring scoped to the factory	One dashboard across the platforms in the flow
Hand-rolled glue when work spans services	Parameters mapped between tasks without bespoke code
A schedule change is a release	Schedules and dependencies change in the UI
Orchestration and polling are billable activity runs	Waiting happens at the queue, not in billable activities
Metadata-driven work needs a framework you maintain	The graph, parameters and schedules are the configuration

Will I still need ADF? Yes

Polysync does not move bytes. If you use ADF today to copy from on-premises SQL into ADLS, that activity is still in ADF and still uses your integration runtime. Polysync calls the pipeline, waits for it, captures its result, and decides what runs next.

When ADF alone is still the right answer

If every pipeline you operate lives inside one ADF instance, your dependencies are simple, and you have no plans to bring Databricks, Functions, Logic Apps, Synapse, Fabric or Google Cloud services into the same flow, ADF's built-in scheduling is fine. Adding a separate orchestrator before you need one is overhead, not insurance.

The point at which most teams cross the line is the second platform: the first Databricks job, the first Cloud Function, the first workload that has to coordinate with an ADF pipeline. The full comparison, with the figures, is in the original article on our website: https://www.polysync.com.au/articles/adf-vs-polysync?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=adf-vs-polysync

About Polysync

What is ETL orchestration? A technical guide

Samir Elmallah — Fri, 16 Oct 2026 10:38:49 +0000

ETL orchestration is the discipline of running data pipelines in the right order, at the right time, under the right resource constraints, with the right response when something fails. It is the layer between your scheduler and the cloud services that actually move and transform data.

A useful working definition: orchestration is the runtime that turns a graph of work into a sequence of platform API calls, applies governance to those calls, and records what happened.

Scheduling is not orchestration

The two terms get used interchangeably. They are not the same thing.

Scheduling answers a single question: when should this start? A cron expression and a clock are enough.
Orchestration answers four: when does it start, what has to finish first, how many copies can run at once, and what happens on failure?

Every scheduler is a tiny orchestrator, and every orchestrator contains a scheduler. The interesting design choices live in the other three questions.

The DAG is the core data structure

Workflows are modelled as a directed acyclic graph (DAG). Each node is a task. Each edge declares that one task must wait for another before it runs. The "acyclic" part is non-negotiable: if task A waits for B and B waits for A, neither can ever start. A real orchestrator detects that loop before it executes anything and refuses the run.

Edges usually carry a trigger condition:

On Success. Run the downstream task only if the upstream task finished successfully.
On Failure. Run it only if the upstream failed, which is where alerting or compensation belongs.
On Completion. Run regardless of outcome, which is where cleanup belongs.
On Skipped. Run only if the upstream task was skipped by an earlier condition.

A solid orchestrator resolves these conditions recursively: the status of a node is a function of the resolved status of its ancestors, not just its immediate parent. That is what allows complex branches and skip paths to behave correctly without manual bookkeeping.

Dispatchers, queues, and why they run separately

The component that actually calls the cloud platform API is the dispatcher. In any non-trivial orchestrator it runs as a separate process from the scheduler and from the UI, for three reasons:

Throughput. Dispatching is bursty. Decoupling lets you scale dispatchers without scaling the UI.
Isolation. A misbehaving platform connection (slow API, throttling, auth failure) should not freeze unrelated work.
Durability. If the dispatcher restarts, work in the queue is still there, and the scheduler keeps producing work even if a dispatcher is busy.

The channel between them is typically a queue table or a message broker, and the queue is where governance lives.

Concurrency limits and rate limits are different controls

Both protect a downstream service from being overwhelmed, and they are not the same thing.

A concurrency limit caps how many tasks may be in flight against a platform at any moment. New work waits until a slot opens.
A rate limit caps how many tasks may start in a period of time (per second, minute or hour). A token bucket starts full and admits a burst before settling to the refill rate, a leaky bucket drains at a fixed rate, and a sliding window counts starts inside a rolling period.

Cloud services publish limits that map onto one or both: Databricks job concurrency per workspace, Data Factory pipeline concurrency, Logic Apps action throughput, Cloud Functions concurrent executions. Hitting them shows up as HTTP 429 responses, dropped runs and retry storms.

A mature orchestrator expresses both, and lets you choose the scope. A separate budget per platform keeps one busy workspace from starving unrelated work; a single shared budget is what you want when different services all drive the same downstream database.

State, retries and idempotency

Distributed work fails for boring reasons: network blips, transient throttling, expired tokens, mid-flight deployments. A retry policy is not optional; it is part of the contract.

The non-obvious part is idempotency. A retry is safe only if the underlying platform call is idempotent, or the orchestrator tracks the original run identifier. Good orchestrators capture the platform-side run ID on first dispatch and reconcile state from it on retry, rather than blindly issuing a second call.

Metadata-driven vs code-driven

Two design schools dominate the market.

Code-driven orchestrators (Airflow, Dagster, Prefect) express workflows as code: the graph is built by evaluating that definition each time a run starts.
Metadata-driven orchestrators express workflows as configuration: the graph is stored as data and read at run time, so it can change without a code deployment.

Both are valid. Code-driven gives you arbitrary logic and version-control diffs. Metadata-driven gives you a UI non-engineers can use, hot reload without redeployment, and a cleanly auditable configuration store. The right answer depends on who owns the pipelines.

Where Polysync fits

Polysync is a metadata-driven orchestrator for work that spans platforms. Jobs are discovered from each connected platform rather than retyped, the dependency graph is resolved recursively with cycle detection before a run starts, the dispatcher is a separate process with state and locking that survive restarts, and concurrency and rate limits are applied as profiles you can scope per platform or share across several. Credentials live in a vault you control (Azure Key Vault, AWS Secrets Manager, Google Cloud Secret Manager, HashiCorp Vault), or in Polysync's managed vault if you would rather not link one of yours.

For the worked examples and the diagrams, read the full article on our website: https://www.polysync.com.au/articles/what-is-etl-orchestration?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=what-is-etl-orchestration

About Polysync

Microsoft, Azure, Azure Data Factory, Azure Databricks, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Other product names, logos and brands are the property of their respective owners.