When native cloud schedulers are enough, and when not

Every cloud platform ships with a scheduler. For work that stays inside that platform it is usually the right tool, and adding an orchestrator on top buys you a hop, a credential and a failure mode.

The trouble starts at the boundary, where one platform's scheduler has to wait on another platform's work and no single graph covers the run. Here are five checks for working out which side of that line you are on, and three ways across it if you are on the wrong side.

The five checks

You are still comfortably inside one platform when:

• every step is a first-class job in the same platform, not an HTTP call;
• a single dependency graph covers the whole run;
• monitoring shows the whole run, not one platform's slice of it;
• retries and idempotency are handled inside the platform;
• adding a step does not mean inventing a new integration pattern.

You are past the edge when:

• steps are chained with storage events, webhooks or polling timers;
• a step is a service call rather than a job the scheduler understands;
• "did the other platform's job finish?" is answered by a person, or by a table someone maintains;
• lineage and run history stop at the platform boundary;
• adding the next platform means building another bespoke bridge.

All five hold and the native scheduler is enough. Any one fails and you are about to start building glue. That is worth saying plainly: a scheduler that ships with the platform is not a compromise. There is nothing extra to run, retries happen where the failures happen, and the platform's own team keeps it working.

What glue actually looks like

Glue is not a product. It is the work that exists because one platform has to wait on another: a storage event that fires a function, a webhook that calls the next platform's API, a polling timer that asks whether the other job finished, or a lookup table somebody keeps current by hand.

Two properties matter before you build more of it:

• Glue has no graph. The dependency lives in the event subscription or the timer, not in something anyone can draw.
• Glue makes the run invisible at the edges. Each platform sees only its own slice, so retries, lineage and error handling end at the boundary, and it breaks quietly.

Scenario 1: one platform does the whole job

Picture a pipeline where every stage lives in Databricks. Data is ingested from cloud storage and message buses by the platform's declarative pipelines, transformations run in the same pipelines, and unstructured documents are parsed with an AI function:

SELECT ai_parse_document(content) FROM documents

The parsed output is indexed for search with Databricks AI Search, and analysts read the result in AI/BI dashboards. Lakeflow Jobs schedules the lot with one dependency graph, and the platform's monitoring shows every task in one place.

Run the five checks: all five hold. In this shape an external orchestrator removes nothing.

Scenario 2: the same pipeline, two platforms

Now mirror it. Ingestion in Azure Data Factory, document extraction on Azure Document Intelligence, transformation on Databricks, the search index in Azure AI Search, dashboards in Power BI.

Logically identical. Physically it crosses three or four boundaries. Data Factory can invoke a Databricks job directly and has a connector that writes to an Azure AI Search index, so those hops are expressible inside the platform. Document extraction has no first-class activity, so it becomes a Web call, a place where the dependency no longer resolves on its own. The Databricks job schedules its own tasks on its own clock, and Power BI refreshes on a third.

Run the five checks again. The graph no longer covers the run. Monitoring is split across four consoles. None of this is a defect in any one product. It is what a boundary does.

Three ways across, and when each is right

1. Event glue. Storage events, webhooks or a polling timer pick up where the last platform left off. Cheapest to start, and the cost shows up later as silent gaps and manual reconciliation.
2. A scheduler of schedulers. One job whose only task is to start other schedulers and wait. It works, but each platform still sees only its own slice, and the outer graph is a script one person owns.
3. A control plane that spans the platforms. Each platform stays the executor; a separate layer holds the dependency graph across them, maps parameters between tasks and keeps one run history.

A simple test settles it: try to answer, for one run, what ran, in what order, and what waited on what. If the honest answer needs four browser tabs, the boundary is costing more than an orchestrator would.

Where Polysync fits

Polysync is a control plane rather than a replacement for the platforms you already run. It treats pipelines, jobs and functions across Azure, Databricks, Fabric, AWS and Google Cloud as first-class nodes in one dependency graph, and decides what runs, when, and what waits. It does not move your data and does not change how any platform executes its work.

The article this is based on is the canonical home for this piece, with the figures: https://www.polysync.com.au/articles/native-cloud-schedulers?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=native-cloud-schedulers

Microsoft, Azure, Azure AI Search, Azure Data Factory, Azure Document Intelligence, Microsoft Fabric and Power BI are trademarks of the Microsoft group of companies. Other trademarks are the property of their respective owners.