Design the producer consumer problem using AWS

hoan-pham-duy — Tue, 12 Jul 2022 10:52:20 +0000

Hi everyone, today I would like to share how I'm using AWS to solve the producer consumer problem.
I got this case when I have to run a heavy background task that take a lot of time to run.
For example: User will send a request to generate images with filters the user choose and the tasks generate images with filters will take a lot of time to run.

Here is my explanation about my architecture:
Firstly, I need to build an API Gateway that can receive the user's request.
Secondly,every request from the user will be forwarded to Lambda Producer and the lambda will response that it received the user requests successfully
Thirstly, the producer will send message to SQS queue.
Finally, the SQS queue will trigger the Lambda Consumer each time it had new message. The Lambda Consumer will run background task and scale automatically based on number of message in the queue.

In my case, the Lambda Consumer will generate images with filters then uploads images back to S3 + send an email to user that it's done via AWS SNS.
Hope for your comments to see is there anything that I can do better.
Thank you!

Bitbucket Pipeline build and deploy Lambda Container Image in AWS

hoan-pham-duy — Tue, 12 Jul 2022 09:10:19 +0000

Summary
In this post, I want to share how I build CI/CD system that Bibucket Pipeline can build and deploy Lambda Container Image in AWS.
The architecture of this post:

Steps:
Step 1: Create a repository at Elastic Container Registry

I set it as private repository, Linux Operating systems, Architecture x86
After create the ECR repository, we can see the image URI and the push command to know how we upload our container to ECR.

The push commands:

Step 2: Push a container from your local machine to ECR.
Currently, I use sample python lambda docker from https://github.com/lambci/docker-lambda.git

Step 3: Create Lambda Container Image that using the Image URI
Choose the container tab

Choose the repository and the container we've just pushed

Step 4: Create Bitbucket repository

Step 5: Add bitbucket-pipelines.yml file to the projects to run pipeline and push it to Bitbucket Pipeline
We need to fill some information from AWS to bitbucket-pipelines.yml like IMAGE_NAME, FUNCTION_NAME, the link in "docker login"

Step 6: Set up your AWS credentials in repository's variable

Step 7: Run the pipeline in Bitbucket and wait for the pipeline completes build Lambda container images, push to ECR and Update Lambda Image URI.

Step 8: Check the image we pushed in ECR from Bitbucket pipeline, check the Lambda's Image URI with the name convention: TAG=$BITBUCKET_BUILD_NUMBER-$BITBUCKET_BRANCH-$BITBUCKET_TAG
In my case, the BITBUCKET_BUILD_NUMBER = 2 and run in master branch with empty BITBUCKET_TAG

That's all of what I want to share in this post.
Please let me know if you have other questions!

DEV Community: hoan-pham-duy

Design the producer consumer problem using AWS

Bitbucket Pipeline build and deploy Lambda Container Image in AWS