AWS Lambda
AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications and only pay for what you use.
The Problem
We want some work to process at background (asynchronously) in Lambda even if the Lambda function returns. But when the lambda function returns, all the background services automatically stop.
The Solution
So here we are going to use Amazon SQS for this purpose. We will send a message to a queue, and it will trigger another lambda function that will run in the background.
What is Amazon SQS?
Amazon Simple Queue Service (SQS) is a managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Using SQS, you can send, store, and receive messages asynchronously between software components at any volume, without losing messages or requiring other services to be available.
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.
Golang Example to Run Background Processes in Lambda Using SQS
Prerequisites
- For building and deploying your functions, you’ll be using the
Serverless Framework, which is the most widely used tool for the job. Assuming you have a recent version of Node.js installed, you can install theServerless CLIwith the following npm command
$ npm install -g serverless
Once you have the Serverless CLI installed, you must configure it to use the AWS access keys of your account
$ serverless config credentials --provider aws --key <access key ID> --secret <secret access key>
You can get the access key and secret key from the My Security Credentials option. Use Create New Access Key if you do not have one already
- If you don’t have Go installed yet, you can either download an installer from the official website or use your favorite package manager to install it
Create a Messaging Queue
Let's create a queue. Go to Simple Queue Service (SQS) in your AWS account. You will see the following interface there
Create queue from here and select type FIFO and let's name it test.fifo as we want to process exactly once message, in the exact order that they are sent. you can also use the standard queue according to your need. And we'll keep all other settings to default for now.
And now let's code.
Sending Message on the SQS Queue
Now that you have everything you need, let’s install the AWS SDK for Go library.
$ go get github.com/aws/aws-sdk-go
After this, we can proceed to write the code to send a message on our SQS queue to trigger the lambda function that will perform the background task.
package handlers
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sqs"
"os"
"encoding/json"
"log"
"net/http"
)
func yourHttpHandlerFunction(w http.ResponseWrite, r *http.Request) {
// tasks you need to perform before the background task
...
// sending details to other lambda function to perform the background task
message := SqsTriggerMessage{
Message: "You can add different fields here according to your data requirements in the background task.",
}
err := sendMessage(message)
if err != nil {
log.Println(err)
w.WriteHeader(http.StatusInternalServerError)
return
}
// other tasks you need to perform
...
// sending user an immediate response
w.WriteHeader(http.StatusOK)
}
type SqsTriggerMessage struct {
Message string
}
func sendMessage(data interface{}) error {
b, err := json.Marshal(data)
if err != nil {
return err
}
svc := sqs.New(
session.Must(
session.NewSession(
&aws.Config{
Credentials: credentials.NewStaticCredentials(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
Region: aws.String(os.Getenv("AWS_REGION")),
},
),
),
)
result, err := svc.GetQueueUrl(&sqs.GetQueueUrlInput{
QueueName: aws.String("queue-name"),
})
if err != nil {
return err
}
_, err = svc.SendMessage(&sqs.SendMessageInput{
MessageBody: aws.String(string(b)),
QueueUrl: result.QueueUrl,
MessageGroupId: aws.String("group-id"),
MessageDeduplicationId: aws.String("deduplication-id"),
})
return err
}
This previous code can be broken into a few simple steps:
- Define a message struct that contains the data you need in the background task
- Marshal your struct into JSON and send that JSON as message to the SQS queue We can also send this through the message attributes, but I found this method more convenient to use
- Give the user a
200response and return. Your lambda function will finish after this, but the message you sent on the SQS queue will trigger the other lambda function to perform the background task
The Lambda Function to Handle SQS Event
Let's create another program to handle sqs events.
package main
import (
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"encoding/json"
"log"
)
func main() {
lambda.Start(handleSqsRequest)
}
func handleSqsRequest(sqsEvent events.SQSEvent) error {
for _, message := range sqsEvent.Records {
var request SqsTriggerMessage
err := json.Unmarshal([]byte(message.Body),&request)
if err!=nil {
log.Println(err)
continue
}
// task you need to perform based on the data in the message
log.Println(request.Message)
...
}
return nil
}
type SqsTriggerMessage struct {
Message string
}
This previous code can be broken into a few simple steps:
- We wrote a
handleSqsRequestfunction that will receive the messages from queue and registered it in the main function using theAWS Lambda for Go library - In the event handler function, we are unmarshalling the message body into our message struct, and printing the message on the console. You can send the details of the task you need to perform and use those to call appropriate function to do that task
Now we have 2 Lambda functions the first one sends the message to SQS queue, that needs to trigger our second lambda function to perform the background task
Deployment
Our lambda functions are now ready, and we can proceed by deploying it with the Serverless Framework. Our application is deployed by the Serverless framework based on the serverless.yml configuration file.
If you are not familiar with the
.ymlsyntax, you can read this serverless.yml guide.
We must first create a serverless.yml file that defines what we are deploying.
service: your-service-name
provider:
name: aws
runtime: go1.x
package:
exclude:
- ./**
include:
- ./bin/**
functions:
main-program:
handler: bin/main-program
events:
- http:
path: /
method: get
...
sqs-handler:
handler: bin/sqs-handler
events:
- sqs: <replace_this_with_your_sqs_queue_arn>
In the above file, we defined both our lambda functions.
- The first one runs the main program on the HTTP requests through
Amazon API Gateway. In this program we need to perform the background task, so we will send a message on the SQS queue - The second one runs our sqs handler on an SQS event in the queue which
arnwe provide here. You can get thearnfrom your aws account where we created it.Amazon SQS > Queues > test.fifo
Next up, we will build our code, and deploy it using the serverless deploy command.
GOOS=linux GOARCH=amd64 go build -o bin/main-program .
GOOS=linux GOARCH=amd64 go build -o bin/sqs-handler ./sqs
serverless deploy
You can see the logs of all your lambda functions through the CloudWatch. To access the logs, go to CloudWatch > Log groups in your amazon account.
You can see the logs from here. Let's check the sqs-handler logs. It must have printed the received message.
References
Get the full source code from this github repository.
Top comments (0)