Most teams do not have a transformation problem. Data Factory, Databricks, Functions, Logic Apps, Synapse, Fabric and Cloud Composer all do their respective jobs well. The pain shows up between them: the pipeline that finishes in the morning, the job that should follow it but instead runs on a fixed schedule and hopes the other one is done, the Cloud Function that nobody is sure ran at all.
Automating a cross-platform pipeline comes down to one question: what is ready to run now? Seven things have to be in place to answer it reliably, and this walkthrough sets them out in the order you would build them. The pattern is not tied to one orchestrator; each step ends with how it looks in Polysync, so the walkthrough doubles as a tour of the product.
One example runs throughout: a nightly customer refresh. An Azure Data Factory pipeline copies new customer records into Azure storage. A Databricks notebook builds a gold table from them. A Google Cloud Function enriches the new rows. A Power BI dataset publishes the result. Four services across three clouds, and before any of this is automated, three of them run on separate clocks and hope the others have finished.
Step 1. Connect each platform
Before anything can run, the orchestrator has to be able to call each platform for you. Two choices are involved, and they are easy to confuse.
How does the orchestrator authenticate? Every connection picks an authentication method. For the Azure services, one option is the Polysync Service Principal: a multi-tenant application that Polysync owns, which you grant a role to in your own tenant. There is no long-lived secret for you to hold or rotate. The other choices are your own service principal (with a secret or a certificate) and, on the other clouds, the platform's own methods such as an AWS access key or assume-role, or a Google service account key, application default credentials, workload identity federation or an impersonated service account.
Where does a supplied credential live? If you use your own credential, it is stored in a secret vault and referenced by name, rather than pasted into a configuration screen. Polysync reads from Azure Key Vault, AWS Secrets Manager, Google Cloud Secret Manager and HashiCorp Vault, and can also hold the credential in its own managed vault if you would rather not link one of yours. The value is read when a task is dispatched, not copied into the task's configuration.
Polysync connects to more than 20 services across Azure, AWS and Google Cloud, including Azure Data Factory, Microsoft Fabric, Synapse Pipelines, Databricks and Power BI, AWS Glue and Step Functions, and Google Cloud Composer, Dataflow and Vertex AI.
Step 2. Discover jobs instead of redefining them
Once a platform is connected, you should not have to retype what already lives there. Polysync reads the platform's own resources and imports them as Jobs: the ADF pipeline, the Databricks notebook, the Cloud Function, the Power BI dataset. The parameters each resource accepts come with it.
Keep two words apart. A Job is the resource on the platform. A Task is a configured run of that Job: parameters bound, dependencies attached, a schedule or trigger. The same Job can back several Tasks, so one notebook can serve both the nightly refresh and a one-off backfill without being copied.
In the example, the four platforms are connected and Polysync imports the copy pipeline, the notebook, the Cloud Function and the dataset refresh. Nothing is rewritten; the resources stay where they are.
Step 3. Wire the dependencies, not the clock
A pipeline is a set of relationships, not a set of times. "Run the notebook at 06:15 because the copy usually finishes at 06:10" encodes a guess that will be wrong on the one night it matters. Tell the orchestrator the relationship and let it work out the time.
Each edge in the graph carries a trigger condition:
- On Success. Run only if the upstream task succeeded. This is the chain in the example: copy, then transform, then enrich, then publish.
- On Failure. Run only if the upstream failed, which is where an alert or a compensation step belongs.
- On Completion. Run whatever the outcome, which is where cleanup belongs.
- On Skipped. Run only if the upstream was skipped by an earlier condition, which keeps branch-and-merge graphs correct.
Conditions are resolved across the whole graph, not just the immediate parent, so a skip deep in a branch behaves the way you would expect. A cycle is rejected before a run starts: an accidental loop between two tasks is reported as a configuration error, not discovered as a run that never finishes.
Step 4. Pass values between tasks
Useful pipelines hand values along. The copy pipeline knows how many rows it moved, and the notebook needs that count as a parameter. The Cloud Function writes a file and returns its path, and the Power BI refresh needs to know which partition changed. Without parameter mapping, that value travels through a shared file or a table that somebody maintains by hand.
Polysync maps an output of one task into a parameter of another, across platforms: the row count from the ADF pipeline becomes an input to the Databricks notebook, and the notebook's partition name becomes an input to the Cloud Function. The mapping is part of the task configuration, so it is visible and changeable without touching transformation code.
Step 5. Put governance on the queue
Every managed service has hard limits: Databricks job concurrency per workspace, Logic Apps action throughput, Cloud Functions concurrent executions. Parallel triggers turn those limits into HTTP 429 responses, dropped runs and unpredictable spend.
Two different controls put a ceiling on that, and they are worth keeping apart:
- A concurrency limit caps how many tasks are in flight against the platform at once. New work waits until a slot opens.
- A rate limit caps how many tasks are dispatched in a period of time (per second, minute or hour), using a token bucket, a leaky bucket or a sliding window.
A profile carries one or both, and because it is attached to a platform or a set of platforms you choose the scope: a separate budget per platform, or one budget shared across several. A busy Databricks workspace therefore cannot hold up work aimed at an unrelated Data Factory factory, and several services driving the same downstream database can be held to a single shared ceiling.
The queue is also where waiting happens: a task whose dependencies are not yet met waits there, rather than polling the platform and piling up activity.
Step 6. Schedule the roots
Schedules attach to root tasks, the ones with nothing upstream. They use standard cron with a timezone, and there is a visual builder if you would rather not count asterisks. A task can carry more than one schedule.
Everything downstream runs because the graph says it is ready, not because it was given its own clock. In the example, only the ADF copy is scheduled. The transform, the enrichment and the publish have no times of their own.
Step 7. Watch it in one place
The point of all of the above is to answer "what is happening right now" on one screen. Polysync shows queue depth, the tasks currently dispatched, recent run history, durations and status breakdowns across every connected platform, so a run that spans four services reads as one run instead of four consoles.
What you end up with
A pipeline that no longer depends on someone remembering which portal to open first, and a schedule that states a relationship rather than a hope. The graph triggers itself, the queue applies the limits, and one view tells the truth about what ran. The orchestrator becomes the system of record for what your data platform actually did last night.
The figures that go with each step, and the full walkthrough, are in the original article on our website: https://www.polysync.com.au/articles/data-pipeline-automation?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=data-pipeline-automation
About Polysync
Polysync Data Solutions is an Australian company that builds cross-platform orchestration for data, integration and AI workloads. It coordinates the platforms you already run rather than replacing them.
Microsoft, Azure, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Functions, Logic Apps and Microsoft Fabric are trademarks of the Microsoft group of companies. Other product names, logos and brands are the property of their respective owners.
Top comments (0)