DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

MJ
MJ

Posted on • Updated on

Brief Guide to AWS Step Functions with Sample Workflow

AWS Step Function Background

AWS Step Functions is one of the AWS services which allows us to define workflow by integrating the existing AWS services such as Lambda, Fargate, SNS and etc. Each workflow is based on the concept of a state machine and it is defined using Amazon States Language (ASL) (JSON-based).

Workflow on Step Functions is made up of series of steps, with the output of one step acting as input into the next step. Thus, you can perform a series of actions to the workflow input. Due to the limitation of data passed between each steps, it is best to use a separate service like AWS S3 to transfer the large files (Limit).

The benefit of using Step Function is that the application logic is implemented within the workflow and it is separated from the implementation of individual component (step). This actually promotes the reusability of each component for other workflows.

A simple state machine is defined as below. StartAt and States are the compulsory building blocks. StartAt defines the entry point of your workflow and States defines the steps for your customised workflow. There are other optional components that you can have within the state machine.

{
  "Comment": "A Hello World example of the ASL.", // Optional
  "StartAt": "HelloWorld", // Compulsory
  "States": {              // Compulsory
    "HelloWorld": {
      "Type": "Pass",
      "Result": "Hello World!",
      "End": true
    }
  }
}

To define the steps within the workflow, these are the available types of states provided by ACL now:

State Purpose
Task A Task state represents a single unit of work performed by a state machine.
Pass A Pass state passes its input to its output, without performing work.
Choice A Choice state adds branching logic to a state machine.
Wait A Wait state delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end time, specified as a timestamp.
Succeed A Succeed state stops an execution successfully. The Succeed state is a useful target for Choice state branches that don't do anything but stop the execution.
Fail A Fail state stops the execution of the state machine and marks it as a failure.
Parallel The Parallel state can be used to create parallel branches of execution in your state machine.

Sample Step Functions Workflow

This article is going to cover some useful features of ASL and how to create a simple workflow by using AWS Lambda functions. Sample AWS Lambda functions are available here. Before starting, you will need to deploy the AWS Lambda functions from the sample code. Do check out here if you need some help on the Lambda deployment.

The sample Step Functions workflow diagram is as below:

Sample AWS Step Function Workflow

The basic idea of this workflow is to:
1) Configure the random-number-generator-lambda to generate a random number from a number range.
2) Then, trigger the random-number-generator-lambda to generate a random number.
3) Based on the output of the random-number-generator-lambda, the next step is to check if the output is above or below number 5.
4) If it is above 5, SNS notification will be triggered to send out SMS.
5) If it is below 5, another Lambda power-of-number-lambda will be triggered to raise the previous input to the power of configured exponents.
6) Then, the workflow will end after that.

Here is the explanation of ACL features being used for this sample workflow:
1) random-number-generator-lambda-config - To configure the random-number-generator-lambda by using the Pass state.

"random-number-generator-lambda-config": {
  "Comment": "To configure the random-number-generator-lambda.",
  "Type": "Pass",
  "Result": {
      "min": 1,
      "max": 10
    },
  "ResultPath": "$",
  "Next": "random-number-generator-lambda"
},

2) random-number-generator-lambda - To trigger a Lambda function by using the Task state.

"random-number-generator-lambda": {
  "Comment": "Generate a number based on input.",
  "Type": "Task",
  "Resource": "${random-number-generator-lambda-aws-arn}",
  "Next": "send-notification-if-less-than-5"
},

3) send-notification-if-less-than-5 - To check if the output from previous step is less than 5 using the Choice state.

"send-notification-if-less-than-5": {
  "Comment": "A choice state to decide to send out notification for <5 or trigger power of three lambda for >5.",
  "Type": "Choice",
  "Choices": [
    {
        "Variable": "$",
        "NumericGreaterThanEquals": 5,
        "Next": "power-of-three-lambda"
    },
    {
      "Variable": "$",
      "NumericLessThan": 5,
      "Next": "send-multiple-notification"
    }
  ]
},

4) power-of-number-lambda - If output is more than 5, trigger another Lambda function using Parameters to customise the input JSON object.

"power-of-three-lambda": {
  "Comment": "Increase the input to power of 3 with customized input.",
  "Type": "Task",
  "Parameters" : {
    "base.$": "$",
    "exponent": 3
  },
  "Resource": "${power-of-number-lambda-aws-arn}",
  "End": true
},

5) send-multiple-notification - A Parallel state to trigger multiple actions.
6) send-sms-notification - To trigger SNS notification to send SMS.
7) send-email-notification - To trigger SNS notification to send Email. (Not include as part of the sample)

"send-multiple-notification": {
  "Comment": "Trigger multiple notification using AWS SNS",
  "Type": "Parallel",
  "End": true,
  "Branches": [
    {
     "StartAt": "send-sms-notification",
     "States": {
        "send-sms-notification": {
          "Type": "Task",
          "Resource": "arn:aws:states:::sns:publish",
          "Parameters": {
            "Message": "SMS: Random number is less than 5 $",
            "PhoneNumber": "${valid-handphone-number}"
          },
          "End": true
        }
     }
   },
   {
    "StartAt": "send-sns-topic",
     "States": {
       "send-sns-topic": {
          "Type": "Task",
          "Resource": "arn:aws:states:::sns:publish",
          "Parameters": {
            "Message": "Email: Random number is less than 5: $",
            "TopicArn": "${aws-sns-topic-to-send-out-email}"
          },
          "End": true
        }
     }
   }  
  ]
}

You can trigger the workflow by using the Start Execution button within the Step Functions console. Each invocation allows JSON object as input to the workflow, but it is not compulsory. Or Cloudwatch Schedule event can be configured to trigger the workflow.

Before that, it is also important to ensure that you have granted the appropriate permission to the IAM role for the Step Function execution. For this sample workflow, you will need the following

  • AWS Managed Policy - AmazonSNSFullAccess - To trigger SNS notification SMS or Email.
  • AWS Lambda Invocation Permission.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "*"
        }
    ]
}

You should see something as below if the execution is successful. Do try a few more rounds to see you get a SMS from the SNS notification!

Sample Invocation

Not all states are being covered in the above sample workflow, more can be found from official doc on the different ASL features.

Hope this is useful for your exploration on Step Functions.

mengjiann / aws-step-functions-sample

Sample Workflow for AWS Step Function (with terraform)

Sample Project for AWS Step Functions

This is to demonstrate the usage of several AWS Lambda functions to create a workflow on AWS Step Function. Terraform configuration files are also provided to simplify the process of setting up the required AWS resources.

AWS Lambda Function

  • power-of-number-lambda - To calculate the power of a number.
  • random-number-generator-lambda - To generate a random number between MIN and MAX.

Terraform - Prerequisite

  • To use terraform, go to here to install and learn Terraform by Hashicorp
    • For homebrew user: brew install terraform
  • AWS user (programmatic access) with AdministratorAccess. Copy the access key ID and secret access key to setup the awscli later.
  • Install awscli from official guide
    • For homebrew user: brew install awscli
  • Using the access key ID and secret access key, follow this guide to step a new profile for your awscli.

Terraform - Guide

  • After git clone, change directory to…

Reference:

Top comments (0)

Timeless DEV post...

Git Concepts I Wish I Knew Years Ago

The most used technology by developers is not Javascript.

It's not Python or HTML.

It hardly even gets mentioned in interviews or listed as a pre-requisite for jobs.

I'm talking about Git and version control of course.

One does not simply learn git