DEV Community

Cover image for Migrate python async worker to asynchrounous Lambda
Jinwook Baek
Jinwook Baek

Posted on

Migrate python async worker to asynchrounous Lambda

Original blog post

Intro

When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).

Alt Text

Advantage

  • FasS advantage
    • scaling benefit - cost, ops, etc
    • Resiliency
    • faster development
  • No need to maintain Broker (e.g elastic search redis)
  • Smaller side effect on code change
  • Fully-managed

Disadvantage

  • FasS disadvantage
    • limited state
  • Hard to reuse app code
    • Django ORM or settings module
    • You can still use them with django package installed but inefficient
    • can't reuse code from django app
    • Django package is too heavy to use on lambda (still usable but too complex to setup
  • Latency on startup
  • Json formatted Invocation parameter
    • If you need to pass binary or big size paras, need to consider S3 or other medium

Limitation

Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

AWS Lambda quotas

  • timeout - 15min
    • If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
  • Invocation payload
    • asynchronous - 256KB

Consideration

  • VPC, Subnet, Security Group
    • If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
    • If you are lambda needs to talk to outside world, private subnets needs NAT gateway
  • Maintenance

Setup

AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.

Github

refer to this repo for details

kokospapa8/async-lambda-sample-app

Prerequisite

You need following AWS resource created in order to invoke asynchronous lambda.

DLQ

In order use asynchronous lambda, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.

Alt Text

Asynchronous invocation

SECRET

Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use aws secret manger , ssm parameterstore or vault . I will use ssm parameterstore for this post. Refer to previous post for creating secrets.(parameter store section)

How to deploy django app to ECS Fargate part3

arn:aws:ssm:<region>:<account_id>:parameter/SAMPLE/BUCKET

Alt Text

IAM ROLE

You need lambda execution role with following permission and polcies attached.

Policies
- AWSLambdaExecute
Permission 
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups

SAM

What is the AWS Serverless Application Model (AWS SAM)?

Install (on macOS)

Installing the AWS SAM CLI on macOS

$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]: 
Default output format [None]:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew tap aws/tap
$ brew install aws-sam-cli

Development

$ sam init

Alt Text

This code will create following structure

sam-app/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py

template.yaml

Update template file if you need more info on syntax, refer to the link.

AWS::Serverless::Function

  • template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sample-app
  Sample SAM Template for sample-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  SampleAppdFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: sample_app/
      DeadLetterQueue:
        Type: SNS
        TargetArn: arn:aws:sns:<region>:<account_id>:sample-dlq
      Handler: app.lambda_handler
      Runtime: python3.8
      Description: sample lambda
      EventInvokeConfig:
        MaximumEventAgeInSeconds: 60
        MaximumRetryAttempts: 2
      FunctionName: SampleApp
      Role: arn:aws:iam::<account_id>:role/sample_lambda_execution_role
      Environment:
        Variables:
          S3_URL: ""

Outputs:
  SampleAppdFunction:
    Description: "Sample app Function ARN"
    Value: !GetAtt SampleAppdFunction.Arn
  • requirements.txt
boto3
  • app.py This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)
import boto3
import os
import requests

def lambda_handler(event, context):
    image_url = event['image_url']
    S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']

    # download

    image = requests.get(image_url).content


    ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
    response = ssm_client.get_parameter(
        Name=S3_BUCKET_PARAM,
        WithDecryption=True
    )
    bucket_name = response['Parameter']['Value']

    s3_client = boto3.client('s3', region_name="ap-northeast-2")
    response = s3_client.put_object(
        Bucket=bucket_name,
        Key="image.png",
        Body=image
    )

    return response

Build

Once your logic code is ready, you can build it using following command

sam build -t template.yaml --region <region_name>

You can test your function with following command

$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json
#env.json
{
  "SampleAppdFunction": {
        "S3_BUCKET_PARAM_ARN": "parameterstore_arn",
  }
}
#event.json
{
  "image_url": "https://picsum.photos/200/300"
}

Deploy

Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt

$ sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Found
    Reading default arguments  :  Success

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sample-app]: y
    AWS Region [ap-northeast-2]: y
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [Y/n]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    Save arguments to samconfig.toml [Y/n]: y

output should look like this


    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
        A different default S3 bucket can be set in samconfig.toml

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                    LogicalResourceId                                            ResourceType                                               
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                        SampleAppdFunctionEventInvokeConfig                          AWS::Lambda::EventInvokeConfig                             
+ Add                                                        SampleAppdFunction                                           AWS::Lambda::Function                                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-10-09 18:29:04 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                ResourceType                                  LogicalResourceId                             ResourceStatusReason                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_COMPLETE                               AWS::CloudFormation::Stack                    sample-app                                    -                                           
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                              
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 SampleAppdFunction                                                                                                                                               
Description         Hello World Lambda Function ARN                                                                                                                                  
Value               arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp                                                                                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sample-app in ap-northeast-2

Check cloudformation and lambda console for successful deployment.
Alt Text
Alt Text

Invoke function in lambda console

Once Lambda is deployed, you can create test event in console.
Alt Text
You will get successful log data.
Alt Text

Invoke from python app

Now you need to invoke lambda function from your app code. Here are examples you can use.

# invoke_from_app.py
import boto3
import json

SETTINGS = {
    "AWS_DEFAULT_REGION": "ap-northeast-2",
    "ENV": "prod"
}
def main():
    lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])

    payload = {
        "image_url": "https://picsum.photos/200/300"
    }

    ret = lambda_client.invoke(
        FunctionName="SampleApp",
        InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",
        Payload=json.dumps(payload)
    )
    print(ret)

if __name__ == "__main__":
    main()

Monitoring

Celery or rq provides native or 3rd party too for monitoring such as sentry . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring environment. Go to lambda service and application menu. Select Monitoring tab to dashboard and cloudwatch logs. You can also configure x-ray for tracing.
Alt Text
Alt Text

CD

Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the

Github workflow

following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.
Alt Text

on:
  push:
    branches:
      - master

name: Lambda deployment
jobs:
  deploy:
    name: Lambda deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-northeast-2

    - name: Build, and deploy
      run: |
        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
        test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
        test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
        test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
        brew --version
        brew tap aws/tap
        brew install aws-sam-cli
        sam --version

        sam build
        sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1

If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.

Deploying serverless applications gradually

Conclusion

Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.

Top comments (3)

Collapse
 
jaradc profile image
jaradc

Good stuff. Thanks for illustrating how to invoke a Lambda function from Python code. This was my starting question. I am trying to remove celery as a dependency which would also remove the need for Redis (in my case) which is a nice decrease in project complexity and long-term cost (if I can do it). What is FasS and RQ? I figured out that DLQ meant "Dead Letter Queue" but was unsure about some of these acronyms you are using.

Collapse
 
ramsescoraspe profile image
Ramses Alexander Coraspe

Rabbit Queue?

Collapse
 
ramsescoraspe profile image
Ramses Alexander Coraspe

what is RQ?