ETL orchestration is the discipline of running data pipelines in the right order, at the right time, under the right resource constraints, with the right response when something fails. It is the layer between your scheduler and the cloud services that actually move and transform data.
A useful working definition: orchestration is the runtime that turns a graph of work into a sequence of platform API calls, applies governance to those calls, and records what happened.
Scheduling is not orchestration
The two terms get used interchangeably. They are not the same thing.
- Scheduling answers a single question: when should this start? A cron expression and a clock are enough.
- Orchestration answers four: when does it start, what has to finish first, how many copies can run at once, and what happens on failure?
Every scheduler is a tiny orchestrator, and every orchestrator contains a scheduler. The interesting design choices live in the other three questions.
The DAG is the core data structure
Workflows are modelled as a directed acyclic graph (DAG). Each node is a task. Each edge declares that one task must wait for another before it runs. The "acyclic" part is non-negotiable: if task A waits for B and B waits for A, neither can ever start. A real orchestrator detects that loop before it executes anything and refuses the run.
Edges usually carry a trigger condition:
- On Success. Run the downstream task only if the upstream task finished successfully.
- On Failure. Run it only if the upstream failed, which is where alerting or compensation belongs.
- On Completion. Run regardless of outcome, which is where cleanup belongs.
- On Skipped. Run only if the upstream task was skipped by an earlier condition.
A solid orchestrator resolves these conditions recursively: the status of a node is a function of the resolved status of its ancestors, not just its immediate parent. That is what allows complex branches and skip paths to behave correctly without manual bookkeeping.
Dispatchers, queues, and why they run separately
The component that actually calls the cloud platform API is the dispatcher. In any non-trivial orchestrator it runs as a separate process from the scheduler and from the UI, for three reasons:
- Throughput. Dispatching is bursty. Decoupling lets you scale dispatchers without scaling the UI.
- Isolation. A misbehaving platform connection (slow API, throttling, auth failure) should not freeze unrelated work.
- Durability. If the dispatcher restarts, work in the queue is still there, and the scheduler keeps producing work even if a dispatcher is busy.
The channel between them is typically a queue table or a message broker, and the queue is where governance lives.
Concurrency limits and rate limits are different controls
Both protect a downstream service from being overwhelmed, and they are not the same thing.
- A concurrency limit caps how many tasks may be in flight against a platform at any moment. New work waits until a slot opens.
- A rate limit caps how many tasks may start in a period of time (per second, minute or hour). A token bucket starts full and admits a burst before settling to the refill rate, a leaky bucket drains at a fixed rate, and a sliding window counts starts inside a rolling period.
Cloud services publish limits that map onto one or both: Databricks job concurrency per workspace, Data Factory pipeline concurrency, Logic Apps action throughput, Cloud Functions concurrent executions. Hitting them shows up as HTTP 429 responses, dropped runs and retry storms.
A mature orchestrator expresses both, and lets you choose the scope. A separate budget per platform keeps one busy workspace from starving unrelated work; a single shared budget is what you want when different services all drive the same downstream database.
State, retries and idempotency
Distributed work fails for boring reasons: network blips, transient throttling, expired tokens, mid-flight deployments. A retry policy is not optional; it is part of the contract.
The non-obvious part is idempotency. A retry is safe only if the underlying platform call is idempotent, or the orchestrator tracks the original run identifier. Good orchestrators capture the platform-side run ID on first dispatch and reconcile state from it on retry, rather than blindly issuing a second call.
Metadata-driven vs code-driven
Two design schools dominate the market.
- Code-driven orchestrators (Airflow, Dagster, Prefect) express workflows as code: the graph is built by evaluating that definition each time a run starts.
- Metadata-driven orchestrators express workflows as configuration: the graph is stored as data and read at run time, so it can change without a code deployment.
Both are valid. Code-driven gives you arbitrary logic and version-control diffs. Metadata-driven gives you a UI non-engineers can use, hot reload without redeployment, and a cleanly auditable configuration store. The right answer depends on who owns the pipelines.
Where Polysync fits
Polysync is a metadata-driven orchestrator for work that spans platforms. Jobs are discovered from each connected platform rather than retyped, the dependency graph is resolved recursively with cycle detection before a run starts, the dispatcher is a separate process with state and locking that survive restarts, and concurrency and rate limits are applied as profiles you can scope per platform or share across several. Credentials live in a vault you control (Azure Key Vault, AWS Secrets Manager, Google Cloud Secret Manager, HashiCorp Vault), or in Polysync's managed vault if you would rather not link one of yours.
For the worked examples and the diagrams, read the full article on our website: https://www.polysync.com.au/articles/what-is-etl-orchestration?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=what-is-etl-orchestration
About Polysync
Polysync Data Solutions is an Australian company that builds cross-platform orchestration for data, integration and AI workloads. It coordinates the platforms you already run rather than replacing them.
Microsoft, Azure, Azure Data Factory, Azure Databricks, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Other product names, logos and brands are the property of their respective owners.
Top comments (0)