I accidentally ended up in the process automation and orchestration space when I started my career, and somehow even more accidentally I'm still there. As Taylor Swift says, "help, I'm still at the restaurant."
Being in the enterprise software space this long, I feel like the pendulum has swung back and forth between "buy a full suite platform, it has everything you need" and "avoid vendor lock-in" and something in between. AI has certainly left it's mark, but hasn't changed things all that dramatically.
You'll find people and companies saying you need a specialized purpose-built platform with a very specific set of use cases or personas or one that's agnostic and fits any and all needs.
It's no different in the process automation and orchestration space.
All of this to say, we still have the same problem - this separation in ownership and responsibilities from the thing we want to improve and how it's actually going to be improved. You often here this positioned as "the business" and "the development team". "The business" gets pitched as this non-technical group of diagram drawing, process optimizing suits while "the development team" adds the code and system integrations to make the diagram actually work, but doesn't own the process.
I still look back on my early career days and remember our business partner asking if she could just give me a Visio diagram and could I turn that into BPMN for her because the standards were too technical. BPMN... as in the standards that were supposed to be the literal language that filled this "gap" between "the business" and "the development team"... was too technical. Interesting.
I have this hypothesis that people in tech generally understand process automation and why it's beneficial, but only certain areas and roles within tech seem to really resonate with the term "orchestration". So maybe let's talk about that a bit and give you a few examples of where orchestration really shines. Then you can tell me your definition.
Ok, so what is orchestration?
You would think it's overdone, but the analogy of a conductor with an orchestra really fits here. You have a bunch of different instruments kind of doing their own thing, but together, with the help of a conductor, making a beautiful coherent piece of music happen. They get clues on when to speed up, slow down, or when to play with more or less intensity.
But I can see how that doesn't translate super well to tech.
Let's get some help from the internets.
Orchestration is the coordinated execution of multiple IT automation tasks or processes. Orchestration is usually applied across multiple computer systems, applications, and services to ensure that deployment, configuration management, and other processes are performed in the proper sequence.
Via RedHat
Orchestration is the coordination and management of multiple computer systems, applications and/or services, stringing together multiple tasks in order to execute a larger workflow or process. More specifically, orchestration is the coordinated execution of multiple IT automation tasks or processes across multiple computer systems, applications and services. These processes can consist of multiple tasks that are automated and can involve multiple systems.
Via Databricks
Orchestration is the process of coordinating distributed software components and systems so that they execute seamlessly as an automated, repeatable process.
Via Orkes
These definitions read a lot like the DevOps definitions - multiple systems, coordination, working together with automation, etc.
Basically, anytime you want to do something complex and chain together a sequence of tasks or coordinate and manage multiple systems, applications, or services either ad-hoc, scheduled, or when something happens somewhere you probably want to do it in a clear and reusable way with orchestration. Got it? So clear!
You'll see orchestration and automation together quite often, but it's typically positioned around the involvement of humans where automation is helping humans, without intervention, complete a single task. Orchestration tends to be more complex, but may involve humans (human-in-the-loop). To what extent humans participate depends.
I think I get it, but I need examples...
Let's try to make this a little more tangible.
Data Orchestration
You have data everywhere, in multiple disparate systems, and now you want to do something with it. Very everything, everywhere, all at once. This is where you see data pipelines, ETL, etc. come into play. You don't necessarily want to merge all the data together into one big database, but you do want to coordinate across the systems to, say, use the data to make informed decisions.
Business Orchestration
This tends to be where process automation and orchestration live in the traditional IT, HR, and finance workflows. My favorite example in this space is a two-step approval process, because it involved a human-in-the-loop aspect (maybe even two!), but many people have seen this at work or in their personal life.
If you know what "signature authority" is you've definitely been a part of a process like this, and the larger the organization, the more systems might be involved to make decisions.
You may encounter a set of rules that fast track you through the process (like my "signature authority" was once $100, which was basically desk trinkets) and anything more than that was escalated to my manager until his "signature authority" limit was hit and then it escalated to his manager. So this system needs to know the org chart, the individuals' "signature authority", the price of the items to be purchased, etc. All of that info is definitely in different databases, if not completely different systems.
AI Orchestration
This is relatively new and gets us into murky territory as well. Agent orchestration, multi-agent systems (MAS), AI pipelines can all fall into this bucket of AI orchestration. MAS for me is really the foundation for where we are at now with AI agents and potentially having them do coordinated work together.
The agents need data, then with that data they do something with other agents, and something complex happens.
I'm intentionally being a bit handwavy here because I don't have a great real world example of this yet, just a lot of marketing blogs around specialized agents working together to complete a common goal.
But when do you need an orchestrator?
It depends, of course.
Some organizations will start building some stitched together, homegrown solution and ultimately hit a point where there is a maintenance or scale issue. Others will realize they want to do something beyond CRON and scheduled jobs, something more real-time and event-based where they need listeners or webhooks.
Others still will realize they need to do a mix of scheduled, event-based, and manual executions with responsibilities spanning across technical and non-technical roles, and this is usually when they start investigating platforms to handle some of this.
So, how do you define orchestration?
Hopefully, I've given you a bit to think about and start to formulate your own opinion on orchestration.
What's your definition of orchestration? Are you doing orchestration within your organization? And how did you evaluate your options?
Top comments (2)
Yeah a former colleague of mine is also in this space, he's a "systems integration expert" - he's probably doing very much the same thing as you!
It's an interesting "specialization", and because it'a a bit 'niche' and specialist I think you're running a lower risk of being replaced/outsourced/"AI-ized" or whatever (commands a nice salary too, I believe) ...
I think that's the advantage of this kind of thing (specialization) versus generic "web developers", which are starting to be seen a bit as 'replaceable commodities' (whether that's true or not, but that's the feeling you get these days) ...
There will always be processes that need to be better. That's what I tell myself at least!