Step Functions with Localstack

AWS State Machines are a great tool to co-ordinate the components of a distributed applications and microservices using visual workflows. Using the aws console to build and deploy step functions is great and easy. But it makes local development somewhat slow. So, as usual, I invested some time in trying to make this whole setup work with localstack.

Table of Contents

• Tools
• Functions
• State Machine
• Run everything

Tools

For this you will need to have,

• Docker
• Node12.X
• awscli-local

Functions

We are not talking calculus

By functions, we are talking lambda functions. For our case, we will have 2 functions: createMatches and calculateRunRate. Both are pretty simple functions. createMatches takes in the following input

{
  teamA: string; // Team playing the match
  teamB: string; // 2nd team playing the match
  numMatches: number; // How many matches they are playing?
  matchType: MatchType; // Is it a T20 or ODI?
}

and it outputs an array of match results in the form,

{
  matchType: MatchType; // Is it a T20 or ODI?
  teamA: MatchScore; // How many runs scored, wickets lost, overs played?
  teamB: MatchScore;
}

calculateRunRate then takes in the output of createMatches function as input and calculates the run rate for each team and returns it as,

{
  teamA: number; // signed floating number
  teamB: number; // signed floating number
}

Run rates can be heart breaking

State Machine

Now that we have our lambda functions in place, lets create the state machine. This is what our machine will look like,

{
  "Comment": "State Machine to calculate net run rates for given teams.",
  "StartAt": "CreateMatches",
  "States": {
    "CreateMatches": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:CreateMatchesFunction",
      "Next": "CalculateRunRate"
    },
    "CalculateRunRate": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:CalculateRunRateFunction",
      "End": true
    }
  }
}

This is a pretty straight-forward json file. The specs for each property can be read here. The state machine needs to specify a starting point, in this case it is the CreateMatches step. And then it also needs to be specify the end step.

The bowling machine

docker-compose

Now we will write the docker-compose.yml. This is pretty straight-forward,

version: '3.7'

services:
  localstack:
    image: localstack/localstack
    environment:
      - DATA_DIR=/tmp/localstack/data
      - SERVICES=lambda
      - TZ=America/New_York
      - LAMBDA_EXECUTOR=docker
    ports:
      - '4574:4574'
    volumes:
      - /tmp/localstack:/tmp/localstack
      - /var/run/docker.sock:/var/run/docker.sock

  stepfunctions:
    container_name: stepfunctions
    image: amazon/aws-stepfunctions-local:latest
    environment:
      - LAMBDA_ENDPOINT=http://localstack:4574
    ports:
      - '8083:8083'

Things to note here that we are starting lambda service on localstack. More information on localstack can be found here. The service will be available on port 4574. Also pay attention to the LAMBDA_EXECUTOR and shared volumes. We need to do that coz with localstack the lambda executor needs docker to run. Next we have the stepfunctions service, which is built using the amazon/aws-stepfunctions-local:latest docker image. Here we need to specify the lambda endpoint. This is endpoint where the step functions will look for lambda functions to execute.

Where is the ball?

Run everything

In order to run this, we will have to run a few commands which can be put together is a script. To deploy the lambda functions to localstack, we will first need to zip them. You can do that by any zipping tool. I did it via zip command line utility.

zip -r ./createMatches.zip ./createMatchesHandler.js
zip -r ./calculateRunRate.zip ./calculateRunRateHandler.js

Next start the containers by running docker-compose up -d. Once the containers are up, you can deploy the lambda functions to localstack like this:

awslocal lambda create-function \
  --function-name CreateMatchesFunction \
  --runtime nodejs12.x \
  --handler createMatchesHandler.handler \
  --role arn:aws:iam::012345678901:role/DummyRole \
  --zip-file fileb://dist/createMatches.zip

awslocal lambda create-function \
  --function-name CalculateRunRateFunction \
  --runtime nodejs12.x \
  --handler calculateRunRateHandler.handler \
  --role arn:aws:iam::012345678901:role/DummyRole \
  --zip-file fileb://dist/calculateRunRate.zip

The output of these commands will look something like this,

{
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "CodeSha256": "iA5Rqi3x8PvmLWPZBvWXYrpOQC9tx6zi6TjpU9K8uKI=",
    "FunctionName": "CreateMatchesFunction",
    "LastModified": "2020-03-30T21:17:30.828+0000",
    "RevisionId": "70a91d5c-c42d-422e-9e93-fbc8d6ef638c",
    "CodeSize": 995,
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:CreateMatchesFunction",
    "Version": "$LATEST",
    "Role": "arn:aws:iam::012345678901:role/DummyRole",
    "Timeout": 3,
    "Handler": "createMatchesHandler.handler",
    "Runtime": "nodejs12.x",
    "Description": ""
}
{
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "CodeSha256": "GF2chlh/6N9mcPAyg6ysY537GsN+fff6zAuNeQUaz3Y=",
    "FunctionName": "CalculateRunRateFunction",
    "LastModified": "2020-03-30T21:17:31.469+0000",
    "RevisionId": "e1bfe0ab-bce5-4e21-b7c4-8a2bfa980a22",
    "CodeSize": 815,
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:CalculateRunRateFunction",
    "Version": "$LATEST",
    "Role": "arn:aws:iam::012345678901:role/DummyRole",
    "Timeout": 3,
    "Handler": "calculateRunRateHandler.handler",
    "Runtime": "nodejs12.x",
    "Description": ""
}

Ok, so now we have successfully deployed our lambda functions to localstack. Now we just need to create our state machine and execute the functions. For that, we can run the following commands,

aws stepfunctions \
  --endpoint http://localhost:8083 \
  create-state-machine \
  --definition file://state-machine.json \
  --name "NetRunRateCalculatorStateMachine" \
  --role-arn "arn:aws:iam::012345678901:role/DummyRole"

aws stepfunctions \
  --endpoint http://localhost:8083 \
  start-execution \
  --state-machine arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine \
  --name test
  --input "{\"teamA\":\"India\", \"teamB\": \"Australia\", \"numMatches\": 3, \"matchType\": \"t20\"}"

This will give you the following output,

{
    "creationDate": 1585603052.224,
    "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine"
}
{
    "startDate": 1585603052.766,
    "executionArn": "arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test"
}

Now to check the status of the execution you can run the describe-execution utility on step functions, like this,

aws stepfunctions \
  --endpoint http://localhost:8083 \
  describe-execution \
  --execution-arn arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test

If the machine already completed execution you will get the following output,

{
  "status": "SUCCEEDED",
  "startDate": 1585603052.766,
  "name": "test",
  "executionArn": "arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test",
  "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine",
  "stopDate": 1585603055.931,
  "output": "{\"teamA\":-3.4393976210462283,\"teamB\":3.4393976210462283}
",
  "input": "{\"teamA\":\"India\", \"teamB\": \"Australia\", \"numMatches\": 3, \"matchType\": \"t20\"}"
}

That's it. You have successfully created a state machine and run it against the lambdas in localstack. You can find the full code here.

Success...