DEV Community: MJ

Using Parameter Store for Workflow on AWS Step Functions

MJ — Sat, 08 Jun 2019 09:10:21 +0000

Workflow in AWS Step Functions is represented by a state machine which consists of a number of states (steps), their relationships and their input and output. When we are constructing the state machine using Amazon States Language, we might define the application secrets as part of the state machine code.

One of the possible situations is that we are using a Pass state to prepare an input JSON for the next state. In the following example, we are passing the remote FTP username and password to the next state, which is a Lambda function to download files from remote FTP server.

"ftp-download-lambda-config": {
      "Type": "Pass",
      "Result": {
          "ftpHost" : "ftp-server.com",
          "ftpUsername" : "username",
          "ftpPassword" : "password",
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},

This will raise a concern if we are to version control the state machine code into any version control system like Git. It is because the application secrets(the FTP username and password) will be exposed to the users who have access to the codebase. As a good security practise, it is advisable to store the secrets in a separate environment, which will ensure that it is safe with limited access and possible encrypted where they are stored.

There are few prominent services in market: Vault(from Hashicorp), Secrets Manager and Parameter Store (from AWS). Since a managed service is preferred, Vault will not be taken into consideration. Comparing Secrets Manager with Parameter Store, Secrets Manager costs $0.40 per secret per month $0.05 per 10,000 API calls and Parameter Store costs nothing for usage. Although Secrets Manager comes with a secret rotation feature which allows you to automatically rotate API keys, passwords and more, it is not required for our requirement. Therefore, AWS Parameter Store is more preferable for this use case.

So, once we have decided to store our secrets in the AWS Parameter Store, we have to modify our application(Lambda function in this case) to accept the reference of secrets stored in Parameter Store and retrieve the secrets from Lambda function. The change in the state machine code is showed as below:

"ftp-download-lambda-config": {
      "Type": "Pass",
      "Result": {
          "ftpHost" : "ftp-server.com",
          "ftpUsername" : "<Reference to username in Parameter Store>",
          "ftpPassword" : "<Reference to password in Parameter Store>",
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},

Or we can have additional Lambda function to retrieve the secrets from Parameters Store and pass the output to the next state in the state machine. Thus, the existing Lambda functions/tasks can remain as it is with the usage of new Lambda functions. The usage of new Lambda function is showed as below:

"ssm-parameters-getter-lambda-config": {
      "Type": "Pass",
      "Result": {
          "param": [
            "parameters-store-ftpUsername",
            "parameters-store-ftpPassword"
          ],
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ssm-parameters-getter-lambda"
},
"ssm-parameters-getter-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ssm-parameters-getter-lambda",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Parameters" : {
        "ftpHost": "ftp-server.com",
        "ftpUsername.$": "$.parameters-store-ftpUsername",
        "ftpPassword.$": "$.parameters-store-ftpPassword",
      },
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},

The ssm-parameters-getter-lambda accepts the following as input:

{
  "param": [
    "random-input-a",
    "random-input-b"
  ],
  "withDecryption": true
}

And produce an object with param name as attribute and value from parameter store. The value can be

{
  "random-input-a": "<Value from Parameter Store>"
  "random-input-b": "<Value from Parameter Store>"
}

The Lambda function is open sourced on Github. It can be deployed manually to your AWS account using the source code: index.js or deploy directly from the AWS Serverless Application Repository with name: ssm-parameters-getter-lambda. Remember to check Show apps that create custom IAM roles or resource policies as if this Lambda requires additional role.

mengjiann / ssm-parameters-getter-lambda

To retrieve parameters from AWS Systems Manager's Parameter Store.

ssm-parameters-getter-lambda

To retrieve parameters from AWS Systems Manager's Parameter Store.

Use case

The best use case is to place this before the state in a Step Function's workflow that accepts configuration as input such as username and password. As such, the application secrets are kept safely in AWS Parameter Store rather than in the Step Function's state machine code.

Parameters Access Restriction

Following the principle of least privilege, there are two options that allows restriction on the AWS Lambda function to access certain parameters from the Parameters Store.

ParametersPrefix - Only allow access to parameters with certain prefix in the Parameters Store.
Tag Keys List - Only allow access to parameters with certain tags for the parameters. Allows comma-delimited list of tag keys.

Sample Input and Output

Input

{
  "param": [
    "random-input-a",
    "random-input-b"
  ],
  "withDecryption": true
}

Output

{
  "random-input-a": "<Value from Parameter Store>"
  "random-input-b": "<Value from Parameter Store>"
}

…

View on GitHub

References:

Key Takeaway of Publishing Lambda Function using Serverless Application Repository (SAR)

MJ — Sun, 26 May 2019 08:03:06 +0000

The Serverless Application Repository (SAR) is an easy way for the developers to find and deploy existing application in the AWS Cloud. For more info, refer to official guide.

One of more popular use cases now is to share the Lambda function with the community. After being published on the AWS SAR, anyone can just deploy the Lambda function directly into their account with just a few clicks.

There are few ways of publishing on SAR:

AWS Management Console
AWS SAM command line interface (AWS SAM CLI)
AWS SDKs to upload the code.

The easy way is to do it through the Management Console and here are some key take-aways:

Only applications created in us-east-1 or us-east-2 allows sharing/visible to the public. Applications created in other regions can be shared privately using accounts number. Semantic version is also required for publicly published application.
Ensure that you have created a bucket to host your Lambda code artifact (archive.zip) and amended the bucket policy with the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service":  "serverlessrepo.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-bucket-name>/*"
        }
    ]
}

Description and License info on the SAR is compulsory.
The most important one is to create a valid AWS Serverless Application Model (AWS SAM) that define the AWS resources. The full documentation is here. But, these are some samples on my Github:
-- Example 1 - This contains addition of multiple role policies required for the Lambda. Parameters are also used in this examples to take in inputs for the SAR application.
-- Example 2 - This contains creation of a S3 Bucket and adds listened Events to the objects created in the S3 Bucket.
-- Example 3 - This contains creation of Role and RolePolicy resources for attached to the Lambda function. DependsOn is also used to ensure the resources are created in correct order.
Once your have published a new version of the application on SAR, the users need to redeploy the application with latest version. Otherwise, they will remain on the older version.
Once an application is deployed using SAR, you can check the status from the Cloud Formation's Stacks console. You can remove the application from there and check if there is any issues with the deployment.

Reference:

What is AWS SAR: https://docs.aws.amazon.com/serverlessrepo/latest/devguide/what-is-serverlessrepo.html
How to publish on SAR: https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-how-to-publish.html

Brief Guide to AWS Step Functions with Sample Workflow

MJ — Wed, 10 Apr 2019 13:33:57 +0000

AWS Step Function Background

AWS Step Functions is one of the AWS services which allows us to define workflow by integrating the existing AWS services such as Lambda, Fargate, SNS and etc. Each workflow is based on the concept of a state machine and it is defined using Amazon States Language (ASL) (JSON-based).

Workflow on Step Functions is made up of series of steps, with the output of one step acting as input into the next step. Thus, you can perform a series of actions to the workflow input. Due to the limitation of data passed between each steps, it is best to use a separate service like AWS S3 to transfer the large files (Limit).

The benefit of using Step Function is that the application logic is implemented within the workflow and it is separated from the implementation of individual component (step). This actually promotes the reusability of each component for other workflows.

A simple state machine is defined as below. StartAt and States are the compulsory building blocks. StartAt defines the entry point of your workflow and States defines the steps for your customised workflow. There are other optional components that you can have within the state machine.

{
  "Comment": "A Hello World example of the ASL.", // Optional
  "StartAt": "HelloWorld", // Compulsory
  "States": {              // Compulsory
    "HelloWorld": {
      "Type": "Pass",
      "Result": "Hello World!",
      "End": true
    }
  }
}

To define the steps within the workflow, these are the available types of states provided by ACL now:

State	Purpose
Task	A `Task` state represents a single unit of work performed by a state machine.
Pass	A `Pass` state passes its input to its output, without performing work.
Choice	A `Choice` state adds branching logic to a state machine.
Wait	A `Wait` state delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end time, specified as a timestamp.
Succeed	A `Succeed` state stops an execution successfully. The Succeed state is a useful target for Choice state branches that don't do anything but stop the execution.
Fail	A `Fail` state stops the execution of the state machine and marks it as a failure.
Parallel	The `Parallel` state can be used to create parallel branches of execution in your state machine.

Sample Step Functions Workflow

This article is going to cover some useful features of ASL and how to create a simple workflow by using AWS Lambda functions. Sample AWS Lambda functions are available here. Before starting, you will need to deploy the AWS Lambda functions from the sample code. Do check out here if you need some help on the Lambda deployment.

The sample Step Functions workflow diagram is as below:

The basic idea of this workflow is to:
1) Configure the random-number-generator-lambda to generate a random number from a number range.
2) Then, trigger the random-number-generator-lambda to generate a random number.
3) Based on the output of the random-number-generator-lambda, the next step is to check if the output is above or below number 5.
4) If it is above 5, SNS notification will be triggered to send out SMS.
5) If it is below 5, another Lambda power-of-number-lambda will be triggered to raise the previous input to the power of configured exponents.
6) Then, the workflow will end after that.

Here is the explanation of ACL features being used for this sample workflow:
1) random-number-generator-lambda-config - To configure the random-number-generator-lambda by using the Pass state.

"random-number-generator-lambda-config": {
  "Comment": "To configure the random-number-generator-lambda.",
  "Type": "Pass",
  "Result": {
      "min": 1,
      "max": 10
    },
  "ResultPath": "$",
  "Next": "random-number-generator-lambda"
},

2) random-number-generator-lambda - To trigger a Lambda function by using the Task state.

"random-number-generator-lambda": {
  "Comment": "Generate a number based on input.",
  "Type": "Task",
  "Resource": "${random-number-generator-lambda-aws-arn}",
  "Next": "send-notification-if-less-than-5"
},

3) send-notification-if-less-than-5 - To check if the output from previous step is less than 5 using the Choice state.

"send-notification-if-less-than-5": {
  "Comment": "A choice state to decide to send out notification for <5 or trigger power of three lambda for >5.",
  "Type": "Choice",
  "Choices": [
    {
        "Variable": "$",
        "NumericGreaterThanEquals": 5,
        "Next": "power-of-three-lambda"
    },
    {
      "Variable": "$",
      "NumericLessThan": 5,
      "Next": "send-multiple-notification"
    }
  ]
},

4) power-of-number-lambda - If output is more than 5, trigger another Lambda function using Parameters to customise the input JSON object.

"power-of-three-lambda": {
  "Comment": "Increase the input to power of 3 with customized input.",
  "Type": "Task",
  "Parameters" : {
    "base.$": "$",
    "exponent": 3
  },
  "Resource": "${power-of-number-lambda-aws-arn}",
  "End": true
},

5) send-multiple-notification - A Parallel state to trigger multiple actions.
6) send-sms-notification - To trigger SNS notification to send SMS.
7) send-email-notification - To trigger SNS notification to send Email. (Not include as part of the sample)

"send-multiple-notification": {
  "Comment": "Trigger multiple notification using AWS SNS",
  "Type": "Parallel",
  "End": true,
  "Branches": [
    {
     "StartAt": "send-sms-notification",
     "States": {
        "send-sms-notification": {
          "Type": "Task",
          "Resource": "arn:aws:states:::sns:publish",
          "Parameters": {
            "Message": "SMS: Random number is less than 5 $",
            "PhoneNumber": "${valid-handphone-number}"
          },
          "End": true
        }
     }
   },
   {
    "StartAt": "send-sns-topic",
     "States": {
       "send-sns-topic": {
          "Type": "Task",
          "Resource": "arn:aws:states:::sns:publish",
          "Parameters": {
            "Message": "Email: Random number is less than 5: $",
            "TopicArn": "${aws-sns-topic-to-send-out-email}"
          },
          "End": true
        }
     }
   }  
  ]
}

You can trigger the workflow by using the Start Execution button within the Step Functions console. Each invocation allows JSON object as input to the workflow, but it is not compulsory. Or Cloudwatch Schedule event can be configured to trigger the workflow.

Before that, it is also important to ensure that you have granted the appropriate permission to the IAM role for the Step Function execution. For this sample workflow, you will need the following

AWS Managed Policy - AmazonSNSFullAccess - To trigger SNS notification SMS or Email.
AWS Lambda Invocation Permission.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "*"
        }
    ]
}

You should see something as below if the execution is successful. Do try a few more rounds to see you get a SMS from the SNS notification!

Not all states are being covered in the above sample workflow, more can be found from official doc on the different ASL features.

Hope this is useful for your exploration on Step Functions.

mengjiann / aws-step-functions-sample

Sample Workflow for AWS Step Function (with terraform)

Sample Project for AWS Step Functions

This is to demonstrate the usage of several AWS Lambda functions to create a workflow on AWS Step Function. Terraform configuration files are also provided to simplify the process of setting up the required AWS resources.

AWS Lambda Function

power-of-number-lambda - To calculate the power of a number.
random-number-generator-lambda - To generate a random number between MIN and MAX.

Terraform - Prerequisite

To use terraform, go to here to install and learn Terraform by Hashicorp.
- For homebrew user: brew install terraform
AWS user (programmatic access) with AdministratorAccess. Copy the access key ID and secret access key to setup the awscli later.
Install awscli from official guide
- For homebrew user: brew install awscli
Using the access key ID and secret access key, follow this guide to step a new profile for your awscli.

Terraform - Guide

After git clone, change directory to…

View on GitHub

Reference:

Apple Magic Keyboard Multi-Device* trick

MJ — Sat, 06 Apr 2019 16:06:49 +0000

I'm a big fan of . It has everything* that I need for a slim profile Bluetooth keyboard except its ability to connect to few different devices (one at a time). Occasionally, I would want to use my iMac (for its larger screen), but the action to switch/reconnect to another device is so painful and troublesome.

But, I have discovered a trick recently, which is to make use the lightning port on the Apple Magic Keyboard to connect to different macs. Once the keyboard is connected to device through cable, it will be connected to it after that.

Now, it is effortless* to switch the keyboard to another device (compared to disconnecting and connecting to the new device)

Let me know if it works for you too.

Using Lambda@Edge for Single Page Application(SPA) on AWS Cloudfront

MJ — Tue, 02 Apr 2019 15:53:06 +0000

Recently, there is a new requirement on the project that I have been working on at work, which is to enable social sharing. It is for a stock photography website which allows users to buy photos at different resolutions, similar to Shutterstock or GettyImages. The new feature requires sharing of individual photo URL on the social sites.

Implementing such feature is trivial for normal websites where basic metadata can be added to each HTML pages. Since the front-end app is implemented using Vuejs, it complicates. It is due to the fact that social sites like Facebook or Twitter do not executed the javascript on the crawled page for sharing on their sites. Therefore, it is not possible to use library to update the HTML page metadata dynamically.

One of the ways to resolve this is by implementing server side rendering for the pages. But, this will require additional hosting for the front-end app as the current setup leverages AWS S3 and AWS Cloudfront for static web hosting. This setup is considered cost effective and has well taken care of the scalability of the front-end app.

A couple of hours researching on the web led me to discover that Lambda@Edge can be deployed to the AWS Cloudfront to intercept and customize the HTTP request. Depending on the source of request by inspecting the User-Agent header, we can modify the returned response body in the Lambda@Edge function.

So, this is how it works.

The green line represents the request flow for normal user access from browser. Then, the red line represents the request coming from social sites.

There are a few limitations on AWS Lambda@Edge as I have written it:

The supported runtime for Lambda@Edge for Cloudfront is only Nodejs.
The Lambda@Edge function can only be deployed onto the North Virginia region.
The Lambda@Edge function timeout can only be configured to 5 secs.
The Lambda@Edge function does not support environment variables and I have handled using Gulp script. Or you can try AWS System Manager Parameter Store.
The Lambda@Edge function cant be deployed into VPC.

For deployment, you will have to:

Publish a new version of your Lambda@Edge function.
Update the Cloudfront distribution with the new Lambda ARN.

And you will need to add the required permission to the Lambda role. Refer to here

For sample AWS Lambda@Edge code, please refer to the following Github repo:

mengjiann / aws-lambda-edge-cloudfront

AWS Lambda Edge for Cloudfront

This is a Lambda@Edge function that is used to customize response body for Single Page Application (SPA) served through Cloudfront.

It allows returning static html for social sites (including Facebook, WhatsApp, Twitter) for rich social sharing (due to the nature of Single Page Application - SPA).

Pre-requisite

npm - can be installed via Homebrew brew install node
gulp-cli - npm install gulp-cli -g

Guide to prepare zip file for deployment

Update the externalApi in the environment-config.json.
npm install
gulp default --env {environment} Environment: dev/ prod
archive.zip will be created in the root folder and can be deployed to AWS Lambda

Things to note

Environment variables are not supported for the Lambda@Edge deployed to Cloudfront. Therefore, gulp is used to automate the task of replacing the env variables and also create the artifact for uploading. Or AWS System Manager Parameter Store can be used.
AWS Lambda…

View on GitHub

Reference:

Open Graph Protocol: http://ogp.me/
Vue Social Sharing Plugin: https://github.com/nicolasbeauvais/vue-social-sharing
Vue Server Side Rendering: https://vuejs.org/v2/guide/ssr.html
Official AWS Lambda@Edge doc: https://docs.aws.amazon.com/lambda/latest/dg/lambda-edge.html
Official AWS Lambda@Edge Limitation: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html#lambda-requirements-lambda-function-configuration

Shutdown Spring Boot command-line application

MJ — Mon, 25 Mar 2019 02:49:05 +0000

If you have a SpringBootApplication which implements the CommandLineRunner interface and wish to terminate it after running the override run method, here is it:

@SpringBootApplication
public class DemoApplication implements CommandLineRunner {

  SpringApplication app = new SpringApplicationBuilder()
                .sources(DemoApplication.class)
                .web(false).build();

  springApplication.run(args).close();

  @Overrride
  public void run(String... strings) throws Exception {
    // Code
  }

}