DEV Community: Tihomir Surdilovic

Defining Workflows

Tihomir Surdilovic — Mon, 11 Oct 2021 15:31:57 +0000

Introduction

Workflows are based on the concept of capturing and expressing our business processes in a structured way. Workflow technologies are software components that promote the use of workflows for building understandable, explainable applications.

All workflow technologies provide two core capabilities:

Ways to define your workflows
A runtime that executes them

Additionally, most provide integrations with different frameworks, testing and debugging capabilities, visibility and observability features, etc.

In the first of two articles, we will give a general introduction to workflows and their importance. We will also look into different, currently-relevant approaches of defining workflows and provide their pros and cons.
In the subsequent article, we will dive deeper into this subject and provide more insight into developing a real-life workflow solution using these different approaches.

Defining workflows

Workflows can be defined in many different ways. One way that we are probably most familiar with is by just using a pen and paper. As we are planning out our day, we can write all of our tasks as a list with some defined order. This list then represents our workflow, or our structured business plan for the day. Let’s say that we write the following initial outline of our morning tasks. Note this is actually our first workflow definition.

Figure 1. Initial workflow

Workflows typically have a set goal, the result we are trying to achieve. In our case, the goal is to get to work on time. Each task in our workflow can be seen as a sub-goal or task that is important to achieve in order to reach our goal.

Our initial workflow is not perfect, and can be improved. For example, we notice that in order to give us more time to eat breakfast, we could shower and brush our teeth at the same time. Furthermore, we can make our breakfast while making coffee and lunch, as well as read our newspaper while eating. We can update our workflow to look like this:

Figure 2. Revised workflow

Next thing we need to realize is that this plan represents a “perfect” scenario. It does not for example deal with failures that we know could happen. We need to update our workflow to deal with predictable failures. Updating workflows is an iterative process. Updates can be based on plain common sense, but also from experience gained over time. One update we want to add to our current workflow is to check the traffic report before driving to work. If the traffic is bad, we will need to notify our boss that we will be late:
re 3. Further updates

Figure 3. Further updates

Notice that we used the adjectives “ok” and “bad” to describe traffic conditions. These adjectives correspond to what we think is ok or bad depending on things like the distance from our home to work, our current car condition, our driving abilities, etc. An important part of the way we define and think about workflows is very domain-specific, meaning that it is closely related to our own reasoning, how we process information, and our current circumstances.

One thing we realize is that no matter how bullet-proof we think our workflow is, it often is not. Uncertainty and having to deal with unexpected issues is part of life.
For example, let’s say we are making our breakfast burrito, it falls on the floor and our dog eats it. We can recover from this failure by making another one, while still maintaining the successful completion of our main goal. In the case we don’t have the ingredients any more to make another burrito, we can recover from this intermittent failure by making a sandwich instead.

Adding all possible intermittent failures to our workflow definitions is not feasible. There are just too many of them, some of which we cannot predict. Our workflow needs to have the capability however to recover from any such intermittent failures. One possible way to recover is to retry a task that failed, another is to be able to compensate for a task that failed with one or more other ones, all while still being able to reach our goal.

Defining workflows in our tech world is based on the same principles we used to describe using our simple morning routine workflow written on a piece of paper.
All the strategies we were able to apply to our morning routine workflow so far, such as
Iteration-based improvements, domain-specific terminology , dealing with known and unknown failures, recovering from failure, and the ability to adapt to changes, all apply to workflows targeting any domain.

Why workflows, really

Over the last decade and more that I’ve been involved with workflow technologies, people often asked me to define what is a workflow, and why to use workflows.

The answer to “What is a workflow?” is kind of a tricky one. If you put ten people in the same room, you will get ten different answers and each one is going to be correct.
One answer that resonates to me more than others is that workflows are means to define a structured set of objectives or tasks needed to achieve a desired goal.

The answer to “Why use workflows?” (especially in the tech field), is very important to me on the other hand, and is the reason why I have been a workflow enthusiast for many years.
It’s true that we can write any sort of application without using workflows. However, by doing so, we are missing out. Workflows provide the most natural way to describe how our processes should work to both humans as well as computers.
Our workflow definition maps to how we think our applications should behave and give instructions to computers on how to execute them.

Unfortunately we cannot just scan our workflow definition that we wrote on a piece of paper and execute it. Workflow technologies, over the last years, have developed many different approaches to defining executable workflows. In the next sections we will talk about the most widely used ones.

Common approaches for defining workflows

The two most widely used approaches for defining workflows are

Domain-specific languages
General-purpose programming languages

Domain-specific languages

Domain-specific languages (DSLs) provide a higher level of abstraction compared to general-purpose programming languages. They also tend to target a very specific class of problems. The idea of domain-specific languages is to expose a smaller subset of functionality that can be used to write our workflows. This subset should be very specific to a single technology domain, and thus can be more efficient for defining workflows in that domain.
DSLs can thus be seen as specialized languages. Trying to solve a problem in one domain using a DSL that specializes in another is often inefficient or even impossible. Similarly, DSLs that attempt to specialize in multiple technology domains often lack functionality of those that specialize solely on one.

Domain-specific languages are typically exposed in three ways:

Flowcharts
Forms
Markup

Both Flowchart-based and Form-based DSLs allow us to visually create our workflow definitions. They target non-technical businesses to allow them to share and describe workflows to other non-technical, as well as technical parties.

Flowchart and Form-based workflow solutions have been around for many years and are in most cases suited to represent workflows of small to medium complexities.

Figure 4. Sample visual workflow

Markup-based DSLs allow us to use light-weight formats such as JSON or YAML to define our workflows. They too target specific technology domains so are well suited for workflows of small to medium complexities. Markup-based DSLs do not rely on a visual representation; however many tools exist to provide this feature. In addition, they can be embedded inside resource definitions of many different frameworks and container-based solutions which use the same markup, making them attractive for integration into existing tech solutions.

Figure 5. Sample markup-based DSL workflow

It is important to mention that whether we use flowchart, form, or markup-based DSLs, these definitions are not executable by computers and require an additional translation step which converts them into executable programming language code.

Pros and Cons of defining workflows with DSLs

We already touched upon some benefits of defining workflows with DSLs such as domain-specialization, visualization, ability to be understood by both technical and non-technical users, as well as easy integration with existing frameworks and tools for markup-based DSLs.

Flowchart-based DSLs typically provide a drag-and-drop experience that is intuitive to many users. Markup-based DSLs can benefit from being easily integrated into existing tooling, such as many different editors and can benefit from their Intellisense features, as well have the ability to be visualized.
Markup-based DSLs also have the benefit of being fairly easy to create and can be customized to fit the end-users specific needs.

As mentioned, when writing workflows using DSLs we should focus on solving problems of small to medium-sized complexities.

DSLs often run into issues when trying to use them for complex workflow definitions.
For flowchart-based workflows, the visualization aspect can quickly lose its benefits and become hard to understand and reason over.

Figure 6. Example large flowchart-based workflow

Another issue we can run into is the need to overuse programming language code that is embeddable into the visual workflow representations. Flowchart-based DSLs allow for embedding code as non-visual parameters of certain control-flow logic blocks. This code performs some aspects of the control-flow logic that cannot be expressed by visualization alone. For complex workflow definitions, we often need to add large amounts of code to define certain parts of our workflow logic that can lead to our visual model becoming misleading as to what our workflow is actually supposed to do.

Figure 7. Code inside flow-chart-based DSL

Markup-based DSLs experience similar issues when dealing with complex workflow definitions.
As they heavily rely on string-based references, defining complex workflow control-flow logic can become very hard to follow and understand. The same applies again for possible over-use of code, in the case of Markup-based DSLs, expression language syntax inside complex workflow definitions that can become hard to grasp.

One more thing that’s important to mention about DSLs is their benefit of being very domain-specific and ability to be used by both technical and non-technical users comes at a price.
DSLs often demand custom ecosystem support, namely tooling that has to be specifically created for that DSL (which are oftentimes proprietary). This is especially the case for flowchart-based DSLs.
Switching from one DSL to another to define our workflow solutions for different technology domains often leads us to having to not only learn a new DSL, but also having to get familiar with vastly different ways to test, debug, deploy, and interact with our workflows.

General-purpose programming languages

General-purpose programming language workflow models target technical users (developers).
As the name implies, they also target a generic class of problems and can be leveraged to define workflows targeting a broad set of technology domains. For this reason they include a much wider set of features compared to those typically exposed by DSLs.

When writing our workflow definitions using code, we can take advantage of many and often all of the programming languages features, and just like generic code that we write, our workflow definitions can scale along with their complexity.

Figure 8. Sample programming-language based workflow

Pros and Cons of defining workflows using programming languages

Programming languages are one of the most widely used and popular technology standards out there. Because of this popularity they have very large ecosystems and mature tooling support for things like documentation, testing, debugging, deployment and framework integration. Many IDEs and frameworks have support for multiple programming languages which makes switching out workflow definitions from one language to another a lot easier. Popular programming languages often have huge communities which leads to stable continuous language innovation, as well as quick turnarounds on improvements and bug fixes.

For developers, learning to write workflow definitions using programming languages comes naturally, as they are already familiar with coding and can easily integrate workflows as part of their existing and new development/architecture efforts.

Code-based workflow definitions do not appeal to non-technical users and cannot be easily used as means to describe business logic across different teams with wide ranges of technical knowledge levels.

Given that they are based on general-purpose programming languages, code-based workflow definitions can become complex to understand. Defining a common set of libraries and solutions for specific control-flow logic implementations, alongside the language's features such as reusability, orthogonality, and abstraction amongst many others, can often aid in lowering this complexity.

Conclusion

I hope that this article has given you a good introduction to what workflows are and why you should use them. In addition, we went through the most common approaches to defining workflows and mentioned their pros and cons.

In the next article, we will dive deeper into this subject and provide more insight into developing a real-life workflow solution using these different approaches.

Stay tuned :)

My journey to Temporal

Tihomir Surdilovic — Thu, 03 Jun 2021 17:09:01 +0000

My 12-year journey with workflows has finally led me to Temporal, and I do not believe it was by chance.
For a long time I was trying to fit in with different workflow ecosystems, without much success.
I have been involved in writing workflow runtimes such as jBPM, Web and Editor tools, libraries, as well as
defining workflow language specifications, but often felt as something was missing.

Looking back

I have been involved with open-source workflow technologies since 2009, right when they started taking off. Back then we were enthusiastic about it but more importantly, the work was done with the right mindset while focusing on the community.
Fast-forward 12 years and I find myself in a situation where things seem to be heading in a significantly different direction.
The ecosystem I was involved with had stagnated, mostly due to its inability to adapt and keep up with the increasing demand for workflow technologies in modern architectures.
Innovation took a back seat to pulling wool over people's eyes, luring them into vendor locks and outdated technologies all while using buzz words like “standards” and “best practices” as bait.

The situation made me want to try my luck in a different tech field, but I never gave up hope that there are open source companies still doing it right. My search ended at Temporal, and it could not have been a better fit.

On the technical side of things, Temporal is no joke. When they say they enable developers to build resilient apps at a massive scale, they really mean it. There are no fake promises, no switch and bait tactics, no hidden agendas.
Temporal has a strong focus on the community. If you have any questions, just reach out on our Community Website or Slack.

Control

Temporal is very un-intrusive which really stood out to me, given my past experiences. Typically, when you want to adopt a new workflow technology, you’re forced to change the way you code and architect applications. You sacrifice a lot of control and have to conform to the rules of the workflow engines you adopt.

With Temporal, the story is completely different. It doesn’t matter if you have an existing application or are developing a new one; you can easily integrate it with a lot of different architectures and programming models. You pick and choose which parts of your code (typically your core business logic) you want to turn into resilient, transactional, and stateful Workflows.

Programming model

Another thing I want to mention is how workflows are defined in Temporal. My past experience revolved around representing workflows via some domain-specific language defined with XML, JSON, or YAML. DSLs can be great and often very useful, assuming they target a specific business domain and do that well.
There are currently very few workflow DSLs that are open source and vendor neutral that focus on this important fact.

With Temporal, users write workflow definitions with standard programming languages. This is a much more powerful approach to writing workflows. Not only are programming languages better at representing control flow logic (compared to DSLs for example), but along with the Temporal SDKs it’s easy to implement very complex things like:

Sync/async execution
Compensation (SAGA)
Retries
Periodic execution (cron)
Event handling
True parallel execution
Cancellations
Complex error handling

These are things that are very hard to define right or at all using any DSL. Temporal also solves a very profound problem with DSLs and workflows in general - testing. Temporal SDKs provide a very sophisticated testing API which can easily be integrated into existing testing/mocking frameworks and CI/CD pipelines.

If you are interested in using any DSL, you can write any DSL interpreter on top of Temporal.
This instantly brings all the Temporal benefits to your DSL of choice.

There are many more technical benefits I’d love to talk about, but will leave that for future posts. After a month and a half I am still learning cool new things every day.
Check out the Temporal intro video I made which shows off some of the benefits of Temporal we touched upon in this blog:

Joining Temporal

Being able to work on amazing technology is great, but what about the people inTemporal? From the beginning of my interview process until now, I still cannot believe how great the people are that work here. There hasn't been a single person I have interacted with that I didn't feel comfortable with. From the start, my co-workers went above and beyond to try to help me; from setting up health benefits, to signing up into the payment systems, and getting my laptop shipped.
Shortly after I started, I received a box full of Temporal swag in the mail. Everyone has been patient with me while trying to learn as much as possible to start being productive. And if it’s not enough to be surrounded by nice people, this team is also super smart and are leaders in the open source community. You can learn so much from them!
The team atmosphere at Temporal is real, and I think it’s something that a lot of us long for. I sure have.

If you have any questions or would like to share your thoughts, feel free to reach out to me at
tihomir@temporal.io.

Also don't forget, Temporal is hiring.