DEV Community: raisel melian

Trigger step functions from lambda after uploading files to S3 using Serverless Framework and Python

raisel melian — Fri, 14 Aug 2020 23:07:48 +0000

In this post, I'm gonna show you how to execute a step function from a lambda function after a file is uploaded to an S3 bucket.

You can watch the video version here:

Intro

In the image, we can see we have three AWS services, S3, Lambda Functions, and Step Functions.

After a file is uploaded to an S3 bucket, AWS will send an event to a lambda function passing the bucket and file name as part of the event.

Steps

Let's start by creating a serverless application:

serverless create --template aws-python3 --path my-state-machine

Now let's go inside the created folder my-state-machine

cd my-state-machine

And open it up in your favorite IDE. I'm using vscode
With vscode you can open your project directly from the terminal with the command:

code .

Now let's open the serverless.yml file and remove all comments for clarity.
After removing all comments we'll have a file similar to this:

service: my-state-machine

provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    handler: handler.hello

Our next step is to create and configure our Lambda function.
Let's replace the functions section with the following snippet:

functions:
  StateMachineTrigger:
    handler: handler.upload
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-upload-bucket
          event: s3:ObjectCreated:*
    environment:
      MY_STATE_MACHINE_ARN: ${self:resources.Outputs.MyStateMachine.Value}          
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - states:*
        Resource: "*"

The events section is declaring that every time a file is uploaded (ObjectCreated) into our bucket ${self:service}-${self:provider.stage}-upload-bucket AWS will send a message to our lambda function lib/trigger.handler with the bucket and file information.
You can see we are declaring an environment variable MY_STATE_MACHINE_ARN. This is for our lambda function to know what is the step function's ARN.

Now we need to create the lambda handler we just declared so lets
replace the file handler.py with the following:

import os
import json
import boto3

my_state_machine_arn = os.environ['MY_STATE_MACHINE_ARN']
client = boto3.client('stepfunctions')


def upload(event, context):
    print(event)
    for record in event['Records']:
        response = client.start_execution(
            stateMachineArn=my_state_machine_arn,
            input=json.dumps(record['s3'])
        )

We are using AWS boto3 to trigger our step function that we are going to declare in just a second, and so we are creating a client for that. We are also declaring a variable my_state_machine_arn to access the environment variable we declared in our serverless.yml. And finally, we are using the start_execution method to execute our lambda function passing the event's s3 (event_record['s3']) object as argument.

Our final step is to declare the step function.
To do that we are going to use a plugin named serverless step functions so let's add the following to the end of our serverless.yml:

stepFunctions:
  stateMachines:
    MyStateMachine:
      id: MyStateMachine
      definition:
        StartAt: ValidateImage
        States:
          ValidateImage:
            Type: Pass
            Result: "Valid"
            Next: GenerateThumbnail
          GenerateThumbnail:
            Type: Pass
            Result: "Thumbnail created"
            Next: NotifyUser
          NotifyUser:
            Type: Pass
            Result: "Notification sent"
            End: true

resources:
  Outputs:
    MyStateMachine:
      Description: The ARN of the example state machine
      Value:
        Ref: MyStateMachine            


plugins:
  - serverless-step-functions
  - serverless-iam-roles-per-function

package:
  individually: true
  exclude:
    - '**/*'
  include:
    - handler.py

This is just a dummy state machine to simulate a use case when we have do the following tasks

ValidateImage: you could validate if the image is from a celebrity, or cats, dogs, etc
GenerateThumbnail: generate a thumbnail of the image
NotifyUser: send an email to the user with validation feedback

The Outputs section is to get the step function's ARN we are passing in the lambda environment variable

It is recommended and a best practice to always use the plugin serverless-iam-roles-per-function for all your serverless applications

Before we deploy our application we need to install our serverless plugins and for that, we need to create a package.json file first, so let's do:

npm init -f

And now we install the plugins:

npm install serverless-step-functions
npm install serverless-iam-roles-per-function

After that, we are ready to deploy our application so let's go ahead and run:

serverless deploy

Now let's go to the AWS console, and upload a file in the bucket created and you will see that your state machine is executed

if you liked this post don't forget to click on ❤️

About me

Raisel Melian’s Twitter
Raisel Melian’s Linkedin
Raisel Melian’s YouTube channel

Manage dependencies like a pro when using Serverless Framework, serverless-python-requirements and Pipenv

raisel melian — Thu, 13 Aug 2020 14:35:51 +0000

In this post, I'm gonna show you how to manage python dependencies when using the Serverless Framework. We'll use Pipenv but you can also apply a similar workflow if using other tools like poetry.

I'm assuming you have already installed serverless framework and have set up your credentials.

Setup

Let's start by setting up our environment.

If you are on a mac(you should):

brew install pipenv

if you are using Windows(why?) or another OS you can check here

For the purpose of demonstration lets create a new project:

serverless create --template aws-python3 --path manage-dep-workflow

This generates a new python v3 project inside a folder named manage-dep-workflow

Development workflow

The first time you navigate to your project's folder, in this case:

cd manage-dep-workflow

you need to activate the pipenv shell:

pipenv shell

Now you can start adding dependencies to your project with:

pipenv install <package>

so let's add pytest as a dependency

pipenv install pytest

Because we didn't specify a version for the dependency the latest is installed. If we wanted to do that we could do:

pipenv install pytest~=6.0.0

After doing this, pipenv will create two files: one named Pipfile and another Pipfile.lock. If you are familiar with NPM these are similar to package.json and package-lock.json

Let's take a look at Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pytest = "~=6.0.0"

[requires]
python_version = "3.8"

we can see the pytest library installed under the [packages] section. There is another section [dev-packages] intended for development only libraries like pytest for example, so if we want to move it we can do it manually or use the command:

pipenv install pytest~=6.0.0 --dev

Integrate Pipenv and Serverless Framework

Now that we know how to install libraries into our project the final step is to integrate Pipenv with the Serverless Framework so that when we deploy our application our dependencies are bundled into the lambda packages.

For this, we are going to use the plugin serverless-python-requirements

This plugin will automatically detect Pipfile and pipenv installed and generate a requirements.txt file.

Let's go ahead and install it:

sls plugin install -n serverless-python-requirements

This will add a new entry in your serverless.yml plugin section like this:

plugins:
  - serverless-python-requirements

To verify everything is working together we can run:

sls deploy

sls package

In your terminal you will get an output like this one:

Serverless: Generating requirements.txt from Pipfile...
Serverless: Parsed requirements.txt from Pipfile in /Users/melianr/dev/python/manage-dep-workflow/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/melianr/Library/Caches/serverless-python-requirements/24891d0bfa66a0144c82d36d710b97b82ea9498d08e4b45957ab78e085a1fdf7_slspyc/requirements.txt ...

As you can see the Serverless Framework is effectively creating a requirements.txt file on the fly and making it available for packaging.

Conclusion

To get more information about Pipenv commands you can read the official doc and to see more configuration options for the Serverless Python Requirements you can read more here

About me

Raisel Melian’s Twitter
Raisel Melian’s Linkedin
Raisel Melian’s YouTube channel