Create your first Serverless workflow with Durable functions

Follow me on Twitter, happy to take your suggestions on topics or improvements /Chris

Durable Functions are an extension of Azure Functions that lets you write stateful functions in a serverless environment. Durable functions manages state, checkpoints, and restarts for you.

What does that even mean you ask?

It means you can have long-running functions like really long running functions. It also has a state which means it remembers where it is, like a workflow.

Ok?

How about this. Imagine you have a situation where
you need to manage something by dividing it up in different checkpoints. Each checkpoint is one step closer for it to be considered handled. More concretely imagine a game for example where you need to load a bunch of different resources and only when everything is loaded and ready are you able to play the game.

Oh, ok, so it's like a workflow framework

Yea exactly, it enables you to specify how something should be executed through a flow. There are even different architectural patterns that is recommended for different flows.

Sounds like that could be expensive, is it?

No not really, the payment model is very close to the one Azure Functions use, only pay for when the functions/ workflow is actually executing.

Sounds great, tell me more

In this article we will cover:

• What are Durable functions, let's talk through what it is and what the central concepts are
• How it works, we will explain a little bit of how it works Resources, we will give out some resources so you can delve further in 
• Lab, we will code through an example so you see the major concepts in use and what happens when

 Concepts and high-level explanation

There are some concepts we need to know about when dealing with durable functions. All concepts play a role that together enables us to run our durable function.

• Orchestrator function, this is a function where we define the workflow, we set up what should happen in the workflow, what activities to carry out and what happens when it's done
• Activity function, Activity functions are the basic unit of work in a durable function orchestration. Activity functions are the functions and tasks that are orchestrated in the process. You can have as many activity functions you want. Make sure to give them descriptive names that represent steps in your flow
• Client functions, Client functions are the triggered functions that create new instances of an orchestration. Client functions are the entry point for creating an instance of a Durable Functions orchestration

Ok, I think I got but can you maybe explain it a bit more?

Sure, the best way to explain it is through a realistic example and an image. So let's talk about order processing. In order processing we imagine we have the following tasks to carry out:

• Checking the inventory,
• Charging the customer,
• Creating a shipment

Given that we know how an order is processed let's show you that image so you get a feeling for the workflow:

Ok, above we see how a client function is being invoked. In the case of an order being created, this typically is an HTTP endpoint we hit from an application. Next thing to happen is that the client function starts an instance of an orchestration. This means we will get an instance id, our unique reference to that specific flow. Next thing to happen is that we try to carry out everything inside of the orchestration like checking the inventory, charging the customer and creating a shipment.

How it works

Let's talk a bit more of how this technically works. The thing with the orchestration is that what it orchestrates usually is asynchronous which means we don't know exactly when something finishes. To avoid that you pay running costs for it durable functions powers down and saves the state.

When an orchestration function is given more work to do (for example, a response message is received or a durable timer expires), the orchestrator wakes up and re-executes the entire function from the start to rebuild the local state.

wait, re-running everything?

No worries, during the replay, if the code tries to call a function (or do any other async work), the Durable Task Framework consults the execution history of the current orchestration. If it finds that the activity function has already executed and yielded a result, it replays that function's result and the orchestrator code continues to run.

Oh ok, that sounds better

Replay continues until the function code is finished or until it has scheduled new async work

Resources

• Free account Azure account You will need to sign up on Azure to use Durable Functions
• Creating your first durable function with JavaScript Quickstart that takes you through creating a durable function
• Durable functions concepts Read more here on concepts and patterns and how to implement said patterns.
• Orchestrator function constraints Constraints you need to be aware of.

 Lab - simple activity flow

We believe the best way to learn is to build something with it. So how do we do it? Well, it's quite simple. Using VS Code we can install a plugin making this process really easy.

Creating our project

Open up the command palette or type COMMAND + SHIFT + P.

Then we select the following, to create a new project

This is followed by us selecting a language, let's take javascript. Then we are faced with this list of choices:

As you can see above we have highlighted three options, cause those are the ones we are going to be using throughout this lab. We need an entry point Durable Functions HTTP Start so let's select that first:

Next up let's create our orchestrator function os again hit COMMAND + SHIFT + P and select Azure Functions: Create Function and select Durable Functions Orchestrator, let's give it the name Orchestrator.

Upon doing so you will be asked to select a storage account:

You will need to select Subscription, Storage account and Resource group. The reason for this is that when you save the state of your function it needs to be saved somewhere for later recovery.

We're almost there, just one more thing to create Activity function. We can create that as well with COMMAND+SHIFT+P and Azure Functions: Create Function and Durable functions activity, let's give it the name Hello when it asks us for a name.

If you followed along it should look like this:

Explaining the artifacts

Ok, so we created three different artifacts, an orchestrator function, a HTTP start/client function and an activity function. How does this all work?

Well, it all starts with HttpStart function that kicks everything off. Then said function kicks off the Orchestrator that in turn starts the Activity functions specified in the orchestrator. Sounds a bit theoretical but let's dive into each of these artifacts and see what happens in the code.

HttpStart

As mentioned above, this is the function that starts it all. Let's have a look at its source code and discuss what's happening:

// HttpStart/index.js

const df = require("durable-functions");

module.exports = async function (context, req) {
    const client = df.getClient(context);
    const instanceId = await client.startNew(req.params.functionName, undefined, req.body);

    context.log(`Started orchestration with ID = '${instanceId}'.`);

    return client.createCheckStatusResponse(context.bindingData.req, instanceId);
};

Above we can see that we get a reference to a client by calling getClient on a durable functions instance df that we import from the library durable-functions.
Next up our client instance calls startNew which produces an instanceId. The instanceId is a reference or handler to this specific function invocation. It doesn't matter so much for this demo but the for the second one we will use that information.
The last thing to happen is that we create an HTTP response.

Let's have a look at function.json, our config file where we set up inputs and outputs for our function:

HttpStart/function.json

{
  "bindings": [
    {
      "authLevel": "function",
      "name": "req",
      "type": "httpTrigger",
      "direction": "in",
      "route": "orchestrators/{functionName}",
      "methods": [
        "post",
        "get"
      ]
    },
    {
      "name": "$return",
      "type": "http",
      "direction": "out"
    },
    {
      "name": "starter",
      "type": "orchestrationClient",
      "direction": "in"
    }
  ]
}

We have two pieces of interesting information. The first is that we have a httpTrigger, i.e we can reach this function through an HTTP call, specifically through a route called orchestrators/{functionName}. The other piece of interesting information is the last entry with type orchestrationClient. This one enables us to get a client reference in our code, without it we wouldn't be able to. So remember to include this config if you need a client instance.

Orchestrator

Let's look at the Orchestrator next. This is where all the interesting things are happening, this is where we set up our flow, what function to be called when and why. Let's look at the code:

// Orchestrator/index.js

const df = require("durable-functions");

module.exports = df.orchestrator(function* (context) {
    const outputs = [];

    // Replace "Hello" with the name of your Durable Activity Function.
    outputs.push(yield context.df.callActivity("Hello", "Tokyo"));
    outputs.push(yield context.df.callActivity("Hello", "Seattle"));
    outputs.push(yield context.df.callActivity("Hello", "London"));

    // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
    return outputs;
});

The first thing we see that we have a method orchestrator that takes a generator function. What are generators? They are a very similar concept to async/await and allows you to carry out asynchronous code in a synchronous-looking way. You recognize them by them having the * as part of the function declaration like so:

function*() {}

and also that the keyword yield is used. yield is used in the same way as an await and means we should stay here and wait in the code until our asynchronous operation has concluded.

So what does that mean for our code above? Let's look more closely:

 outputs.push(yield context.df.callActivity("Hello", "Tokyo"));

Here we can see that we are calling context.df.callActivity() with the args Hello and Tokyo with the keyword yield. This simply means we are invoking an activity function Hello with the argument Tokyo. We can see that there are two more calls to callActivity() that won't be carried out until our activity function has concluded.

Hello

Next up we have an activity function. This is where we carry out all the heavy lifting. Looking at the index.js for the Hello directory we see the following code:

// Hello/index.js

module.exports = async function (context) {
    return `Hello ${context.bindings.name}!`;
};

We see that it returns straight away but it could definitely be a long-running activity. The point is whether it runs in a millisecond or takes some time, it doesn't matter, the orchestration function still have to wait for it to conclude.

Debugging

You may think you understand everything up to this point but it really clicks when you see a debugging flow happening. So that's what we are going to be doing next, we are going to start up our durable function from VS Code and you will be able to see how the breakpoints hit and when.

One of the first things we need to do is to install the durable-functions NPM library that we keep referring to, so let's do that:

npm install durable-functions

Now we are ready to debug so let's hit the debug button.

We should be getting something like this printed in the terminal

Next thing we need to do is to kick everything off by hitting our client function route as indicated above orchestrators/{functionName}. Because we only have one such function called Orchestrator we need to start the whole thing by calling the following URL in the browser:

http://localhost:7071/api/orchestrators/Orchestrator

The first thing to happen is our HttpStart function and its index.js being hit like so:

We let the debugger advance:

Above we see how the Orchestrator and its index.js is being hit.

Next up we advance to the next breakpoint and we see that our activity function Hello and its index.js is being hit next.

We advanced the breakpoint and we find ourselves being back in the orchestration function:

This will lead to the activity function being hit again, this time with the argument Seattle, like so:

As you can see it will keep going like this between activity function and orchestrator until orchestrator is done.

Let's advance through all the breakpoints.

We end up coming to a page like this which is the HTTP response from the method called HttpStart

What's interesting for us to know at this point is what did we end up producing? The answer lies in the URL called statusQueryGetUri. Let's follow that link:

As you can see above the response from our Orchestration function is an array consisting of the responses from all activity functions, like so:

"output": [
  "Hello Tokyo!",
  "Hello Seattle!",
  "Hello London"
]

It ended up that way because of the way we constructed the code, we wrote this:

outputs.push(yield context.df.callActivity("Hello", "Tokyo"));

outputs.push(yield context.df.callActivity("Hello", "Seattle"));

outputs.push(yield context.df.callActivity("Hello", "London"));

// returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]

return outputs;

Summary

There is so much more to learn about Durable Functions but I can already hear some of you snore at this point which is why we will save topics such as Application Patterns and implementation of a pattern Fan-out/fan-in for the next part.

So I hope you are excited about that one.

Acknowledgements

I wouldn't be writing this if it wasn't for your guidance in how Durable Functions work. You both are amazing human beings.

So go give them a follow, they really know their stuff on Serverless

• Anthony Chu dev.to, twitter
• Jeremy Likness dev.to twitter

Comments

[MattC]

Hi Chris.. just following your guide but foiled at the get-go.. I selected 'Create New Project > JavaScript, but then the list doesn't include the 3x Durable Function templates shown in your screenshot :(

[Chris Noring]

Hi Matt. Did you install e Azure Functions VS Code extension? if you did, what version is your install on?

[MattC]

Yes.. it's on version 0.19.1. I'm wondering if it might be because I'm on a corporate proxy & maybe the newer templates haven't synchronised into Visual Code?

[Chris Noring]

checking with my colleagues, thanks for raising Matt

[MattC]

Hey Chris, I've gigured it out.. Visual Code was set to use the v1 runtime for Function Apps.. once I'd flipped it to v2 in the settings it's showing the Durable Function templates :)

[Chris Noring]

Thanks for letting me know, glad it works now :)