A Go project with AWS (SNS, SQS, Lambda)

In this post, we'll explore the simple-encryptor project, which aims to provide a practical way to practice Go concepts and work with AWS services such as SNS, SQS, and Lambda. The project focuses on encrypting and decrypting messages in a simple manner, serving as a learning exercise rather than a production-ready solution.

Project Overview

The simple-encryptor project consists of five independent components:

Consumer - Consumes messages from SQS.
Lambda (lambda_func) - Encrypts messages and sends them to SQS.
Producer - Sends messages to SQS.
Publisher - Publishes messages to an SNS topic.
Subscriber - Subscribes to an SNS topic (email type).

Basic Workflow

The following workflow describes how the components interact with each other:

Publisher application:

• Reads a message from the terminal.
• Publishes the message to an SNS topic.

AWS side:

• The SNS topic has an SQS subscription.
• The SQS is consumed by the lambda function.
• The lambda function encrypts the message and sends the encrypted message to another SQS (encrypted messages).

Consumer application:

• Consumes the SQS (encrypted messages).
• Prints the result to the terminal.

The Code

The code can be accessed here.

Consumer

This code defines a function subscribe that subscribes to an Amazon Simple Queue Service (SQS) queue and continuously receives and processes messages from it.

queueUrl: is the URL of the SQS queue to be subscribed to.
cancel: is a channel used to signal the cancellation of the subscription. The cancel channel is used to gracefully exit the subscription loop when a cancellation signal is received.

The function enters a loop where it repeatedly receives messages from the queue using the receiveMessages function.

It also asynchronously deletes the message from the queue by calling the deleteMessage function in a separate goroutine.

Finally, the subscribe function listens for cancellation signals on the cancel channel using a select statement. If a cancellation signal is received, it returns, effectively ending the subscription loop. Otherwise, it waits for 100 milliseconds before starting the next iteration of the loop.

func subscribe(queueUrl string, cancel <-chan os.Signal) {
  awsSession := common.BuildSession()
  svc := sqs.New(awsSession, nil)

  for {
    messages := receiveMessages(svc, queueUrl)
    for _, msg := range messages {
      if msg == nil {
    continue
      }

      fmt.Println("Original: ", *msg.Body)
      fmt.Println("Decripted: ", decryptor.DecryptMessage(*msg.Body))
      go deleteMessage(svc, queueUrl, msg.ReceiptHandle)
    }

    select {
      case <-cancel:
        return
      case <-time.After(100 * time.Millisecond):
    }
  }
}

The receiveMessages constructs a ReceiveMessageInput with various configurations. It then calls the ReceiveMessage method of the SQS service client to receive messages from the queue. If there is an error during the receive operation, it prints the error and returns nil. Otherwise, it checks if any messages were received and returns the list of received messages.

func receiveMessages(svc *sqs.SQS, queueUrl string) []*sqs.Message {
  receiveMessagesInput := &sqs.ReceiveMessageInput{
    AttributeNames: []*string{           
      aws.String(sqs.MessageSystemAttributeNameSentTimestamp),
    },
    MessageAttributeNames: []*string{
      aws.String(sqs.QueueAttributeNameAll),
    },
    QueueUrl:            aws.String(queueUrl),
    MaxNumberOfMessages: aws.Int64(10),
    WaitTimeSeconds:     aws.Int64(3),
    VisibilityTimeout:   aws.Int64(20),
  }

  receiveMessageOutput, err := svc.ReceiveMessage(receiveMessagesInput)
  if err != nil {
    fmt.Println("Receive Error: ", err)
    return nil
  }

  if receiveMessageOutput == nil || len(receiveMessageOutput.Messages) == 0 {
    return nil
  }

  return receiveMessageOutput.Messages
}

The deleteMessage constructs a DeleteMessageInput with the queue URL and the receipt handle, and then calls the DeleteMessage method of the SQS service client to delete the message from the queue. If there is an error during the delete operation, it prints the error.

func deleteMessage(svc *sqs.SQS, queueUrl string, handle *string) {
  deleteInput := &sqs.DeleteMessageInput{
    QueueUrl:      aws.String(queueUrl),
    ReceiptHandle: handle,
  }
  _, err := svc.DeleteMessage(deleteInput)

  if err != nil {
    fmt.Println("Delete Error", err)
    return
  }
}

Lambda (lambda_func)

The main function is the entry point of the program and starts the Lambda execution by calling lambda. Start with the Handler function as the handler.

func main() {
  lambda.Start(Handler)
}

In the Handler, our trigger source is an SQS, which means that the function receives an SQSEvent. It reads a message from the body of the SQS and parses it into JSON format. By parsing the body, we can access the "Message" property of the message.

Next, the Lambda function encodes the text and sends the "encrypted" message to another SQS. The address of this destination SQS (QueueUrl) is obtained from an environment variable called AWS_SQS_ENCRYPTED_QUEUE. This environment variable is set within the AWS Lambda configuration (see Step 6 - Env Vars for more details).

func Handler(ctx context.Context, sqsEvent events.SQSEvent) error {
  message, err := UnmarshalMessage(sqsEvent)
  if err != nil {
    return errors.New("the message is empty")
  } else {
    sess, _ := session.NewSession()
    svc := sqs.New(sess, nil)

    sendInput := &sqs.SendMessageInput{
      MessageBody: aws.String(encode(message)),
      QueueUrl:    aws.String(os.Getenv("AWS_SQS_ENCRYPTED_QUEUE")),
    }

    _, err := svc.SendMessage(sendInput)
    if err != nil {
      return err
    }

    return nil
  }
}

More information about Lambda in Step 6: Configure AWS Lambda.

Producer

This code demonstrates how to send a message directly to an SQS (Simple Queue Service) queue using the AWS SDK for Go.

The sendMessage function takes the queue URL as an input and prompts the user to enter a message from the command line. It uses bufio.NewReader to read input from os.Stdin (standard input) and waits for the user to enter a message.

func sendMessage(queueURL string) error {
  reader := bufio.NewReader(os.Stdin)
  for {
    text, _ := reader.ReadString('\n')
    if text == "\n" {
      continue
    }
    sendSQS(queueURL, text[:len(text)-1])
  }
}

The sendSQS function then sends a message to the SQS queue using the SendMessage method of the SQS client. The message has a delay of 10 seconds and its body is set to the user-entered message. The QueueUrl is set to the queue URL obtained from the function's parameter.

If there's an error sending the message, it is printed to the console using fmt.Println. Otherwise, the function returns without any further action.

func sendSQS(queueURL string, message string) {
  awsSession := common.BuildSession()
  svc := sqs.New(awsSession, nil)
  _, err := svc.SendMessage(&sqs.SendMessageInput{
    DelaySeconds: aws.Int64(10),
    MessageBody:  aws.String(message),
    QueueUrl:     aws.String(queueURL),
  })
  if err != nil {
    fmt.Println(err.Error())
    return
  }
}

Publisher

The publishMessages function takes a destination parameter, which is the ARN (Amazon Resource Name) of the SNS topic where the messages will be published, and prompts the user to enter a message from the command line. It uses bufio.NewReader to read input from os.Stdin (standard input) and waits for the user to enter a message. It calls the sendSNS function to send the text as a message to the specified SNS topic.

func publishMessages(destination string) {
  reader := bufio.NewReader(os.Stdin)
  for {
    text, _ := reader.ReadString('\n')
    if text == "\n" {
      continue
    }
    sendSNS(destination, text[:len(text)-1])
  }
}

The sendSNS function constructs a PublishInput with the message and destination and calls the Publish method of the SNS service client to publish the message to the specified SNS topic. If there is an error during the publishing operation, it prints the error message.

func sendSNS(destination string, message string) {
  awsSession := common.BuildSession()
  svc := sns.New(awsSession)
  pubInput := &sns.PublishInput{
    Message:  aws.String(message),
    TopicArn: aws.String(destination),
  }
  _, err := svc.Publish(pubInput)
  if err != nil {
    fmt.Println(err.Error())
    return
  }
}

Overall, this code provides a simple interactive console interface to publish messages to an SNS topic. It continuously reads input from the user, discards empty lines, and sends the messages to the specified SNS topic.

Subscriber

This code provides a simple way to subscribe an email address to an SNS topic. It uses the AWS SDK to establish a connection, initiate the subscription, and display the subscription ARN upon successful subscription.

The subscribeTopic function calls the Subscribe method of the SNS service client, passing a SubscribeInput with the emailAddress, protocol set to "email", and topicArn. This initiates the subscription process. If there is an error during the subscription, it prints the error message and returns. Otherwise, it prints the subscription ARN.

func subscribeTopic(emailAddress string, topicArn string) {
  awsSession := common.BuildSession()
  svc := sns.New(awsSession)
  result, err := svc.Subscribe(&sns.SubscribeInput{
    Endpoint: aws.String(emailAddress),
    Protocol: aws.String("email"),
    TopicArn: aws.String(topicArn),
  })
  if err != nil {
    fmt.Println("Got an error subscribing to the topic: ", err)
    fmt.Println()
    return
  }

  fmt.Println(*result.SubscriptionArn)
}

AWS

This section will guide you through the process of creating an account on AWS and configuring SNS (Simple Notification Service) and SQS (Simple Queue Service) for your application.

Step 1: Create an AWS Account

To get started, visit aws.amazon.com and create a new account if you don't have one. Once your account is set up, you'll have access to the AWS Management Console.

Step 2: Configure SNS (Simple Notification Service)

SNS is a publish/subscribe messaging service that enables applications, services, and devices to send and receive notifications. Follow these steps to configure SNS:

1. Access the SNS console at console.aws.amazon.com/sns.
2. Click on "Topics" in the left menu.
3. Click on "Create topic" and provide the following details:
   • Type: Standard
   • Name: EncryptMessage (you can choose any name you prefer)
   • The other fields are optional.
4. Click on "Create topic" and save the ARN (Amazon Resource Name): arn:aws:sns:us-east-1:000000000001:EncryptMessage

Step 3: Configure SQS (Simple Queue Service)

SQS is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Follow these steps to configure SQS:

1. Access the SQS console at console.aws.amazon.com/sqs.
2. Click on "Queues" in the left menu.
3. Click on "Create queue" and provide the following details:
   • Type: Standard
   • Name: NormalMessageQueue (you can choose any name you prefer)
   • Leave the other fields as default.
4. Click on "Create queue" and save the ARN: arn:aws:sqs:us-east-1:000000000001:NormalMessageQueue
5. Save the URL: https://sqs.us-east-1.amazonaws.com/000000000001/NormalMessageQueue

Repeat the above process to create another queue called EncryptedMessageQueue. Save the ARN and URL for this queue as well.

Step 4: Subscribe to the SNS Topic

To receive notifications from the SNS topic, you need to subscribe to it. Follow these steps:

1. From the SNS console, click on "Subscriptions" and fill in the required fields, or click on the "EncryptMessage" topic and click on "Create subscription".
2. Select the protocol as Amazon SQS.
3. Select the Endpoint ARN as arn:aws:sqs:us-east-1:000000000001:NormalMessageQueue. In our case, the simplest way is to subscribe from SQS:
   • Select the SQS NormalMessageQueue.
   • Click on "Subscribe to Amazon SNS Topic."
   • Select the SNS topic: EncryptMessage.
   • Save the subscription.

Step 5: Test the Subscription

Now, let's test the subscription to ensure it's working correctly. Follow these steps:

1. Go to the SNS topic "EncryptMessage" in the SNS console.
2. Click on "Publish message."
3. Enter a message in the provided field.
4. Click "Publish message."

Next, go to the SQS NormalMessageQueue in the SQS console:

1. Click on "Send and receive messages."
2. Click on "Poll for messages."
3. Click on the message ID to read the message.

Step 6: Configure AWS Lambda

AWS Lambda is a serverless computing service that allows you to run your code without provisioning or managing servers. Follow these steps to configure Lambda:

1. Access the Lambda console at console.aws.amazon.com/lambda.
2. Click on "Functions" in the left menu.
3. Click on "Create function" and choose "Author from scratch".
4. Provide the following details:
   • Function name: encryptorMessage (you can choose any name you prefer)
   • Runtime: Go 1.x
   • Leave the other fields as default.
5. Click "Create function" and save the ARN: arn:aws:lambda:us-east-1:000000000001:function:encryptorMessage.

Setup:
At the Runtime settings, edit the handler from "hello" to "main" (since we'll be using the main function).

Permissions:
To avoid permission errors when creating the trigger, we need to add the necessary permissions for Lambda to call ReceiveMessage on SQS. Follow these steps:

1. Click on "Configuration" in the Lambda console.
2. Click on "Permissions".
3. Click on the role name: encryptorMessage-role-xyz. This will open the IAM roles page.

IAM Roles:

1. On the IAM roles page, click on "Permissions".
2. Click on "Add permissions".
3. Click on "Attach policies".
4. Filter by "sqs" and select "AmazonSQSFullAccess" (note: avoid this in production environments).
5. Click "Attach policies".

Trigger:

A trigger is a resource you configure to allow another AWS service to invoke your function when certain events or conditions occur. Your function can have multiple triggers. Each trigger acts as a client invoking your function independently, and each event that Lambda passes to your function has data from only one trigger.

1. Click on "+ Add trigger" in the Lambda console.
2. Select the source as "SQS (Amazon Simple Queue Service)".
3. Enter the SQS queue ARN: arn:aws:sqs:us-east-1:000000000001:NormalMessageQueue.
4. Leave the other fields as default.
5. Click "Add" to add the trigger.

Env Vars:
To provide the Lambda function with the necessary environment variables, follow these steps:

1. Go to the Lambda function configuration.
2. Click on "Environment variables".
3. Click "Edit" and add the following environment variable:
   • Key: AWS_SQS_ENCRYPTED_QUEUE (same name used in the code)
   • Value: https://sqs.us-east-1.amazonaws.com/000000000001/EncryptedMessageQueue (URL of the SQS)
4. Save the changes. Note: You can add additional environment variables as needed.

Build/Deploy:
To build and deploy the Lambda function, follow these steps:

1. In the lambda folder (lambda_func), run the following commands in the command prompt:

$ go mod init github...
$ go mod tidy
$ GOOS=linux go build main.go
$ zip function.zip main

1. Use the "function.zip" file to upload your function to AWS Lambda.
   • Go to the Lambda console.
   • Click on "Code".
   • Choose "Upload from" and select ".zip file".
   • Upload the "function.zip" file.
   • Save the changes.

Test:
To test the Lambda function, follow these steps:

1. Go to the Lambda console.
2. Click on "Test".
3. Enter the following details for the test event:
   • Event name: TestLambda01
   • Event JSON:

{
  "Records": [
    {
      "messageId" : "MessageID_1",
      "receiptHandle" : "MessageReceiptHandle",
      "body": "{\"Type\": \"Notification\",\"MessageId\": \"af45217e-295b-5e93-9e1e-7d2bca04dc47\",\"TopicArn\": \"arn:aws:sns:us-east-1:000000000001:MyTopic\",\"Subject\": \"The subject\",\"Message\": \"Message from SNS\",\"Timestamp\": \"2023-02-02T16:13:19.477Z\",\"SignatureVersion\": \"1\",\"Signature\": \"V+/mFcQqIj1O8J0Af+fLx6CO\",\"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService.pem\",\"UnsubscribeURL\": \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe\"}",
      "md5OfBody" : "fce0ea8dd236ccb3ed9b37dae260836f",
      "md5OfMessageAttributes" : "582c92c5c5b6ac403040a4f3ab3115c9",
      "eventSourceARN": "arn:aws:sqs:us-west-2:123456789012:SQSQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-west-2",
      "attributes" : {
        "ApproximateReceiveCount" : "2",
        "SentTimestamp" : "1520621625029",
        "SenderId" : "AROAIWPX5BD2BHG722MW4:sender",
        "ApproximateFirstReceiveTimestamp" : "1520621634884"
      },
      "messageAttributes" : {}
    }
  ]
}

• Click "Test".

At SQS EncryptedMessageQueue you'll be able to see the received message from Lambda.

You can now send and receive encrypted messages between applications with the added security and reliability of AWS services.

Running the projects

Step 1: Configuring Environment Variables

To begin, navigate to the .env.sample file in your project's directory. Make a copy of this file and rename it to .env. Now, it's time to fill in the necessary environment variables. Here are the variables you'll need to set:

AWS_REGION=us-east-1
AWS_ACCESS_KEY=XXXXXXXXXXXXXXXX //don't share this information
AWS_SECRET_KEY=XXXXXXXXXXXXXXXX //don't share this information
AWS_SNS_TOPIC_ARN=arn:aws:sns:us-east-1:000000000001:EncryptMessage
AWS_SQS_ENCRYPTED_QUEUE=https://sqs.us-east-1.amazonaws.com/000000000001/EncryptedMessageQueue
AWS_SQS_NORMAL_QUEUE=https://sqs.us-east-1.amazonaws.com/000000000001/NormalMessageQueue
EMAIL_ADDRESS_SUB=[e-mail]

Ensure that you replace the XXXXXXXXXXXXXXXX placeholders for AWS_ACCESS_KEY and AWS_SECRET_KEY with your actual AWS access and secret keys. Remember, never share this sensitive information with anyone.

Step 2: In the terminal

Open your project in the terminal and locate the following components: producer, consumer, subscriber, and publisher. Let's run all of these projects together. In your terminal, execute the following command:

$ go run main.go

Step 3: Subscribing to the SNS Topic

Now, let's set up the subscriber to receive email notifications from the SNS Topic. When you run the subscriber project, you'll receive a confirmation email with a message stating "pending confirmation." Follow the instructions provided in the email to confirm your subscription. Once confirmed, you'll be able to receive messages from the SNS Topic named "EncryptMessage."

Step 4: Sending an Encrypted Message

To send an encrypted message, we'll use the producer project. In your terminal, execute the following steps:

• Type your message in the terminal:

$ H/3ll/0, /7h/1/5 /1/5 /0ur m/3/5/5/4/9/3

• Press Enter.

Step 5: Consuming the Encrypted Message

The consumer project is responsible for consuming messages from the EncryptedMessageQueue (SQS). Once the consumer is up and running, you'll be able to see the following messages:

Original:  H/3ll/0, /7h/1/5 /1/5 /0ur m/3/5/5/4/9/3
Decrypted: Hello, this is our message

Step 6: Publishing Messages to the SNS Topic

Lastly, we'll use the publisher project to publish messages to the SNS Topic "EncryptMessage." Follow these steps:

• Type your message in the terminal:

$ Hello World!

• Press Enter.

Once the message is published, you'll be able to see it in the consumer project and also receive it in your email.

That's it! Now you can proceed with your project development with confidence. If you need further assistance or want to dive deeper into the topic, be sure to check out the references below. Happy coding!

References:

• go.dev
• AWS Lambda
• AWS SNS
• AWS SQS
• GitHub - AWS SDK for Go Examples
• YouTube - Sending messages to SQS from AWS Lambda
• YouTube - AWS Lambda in Go - Publish to SNS and SQS
• YouTube - Lambda triggers - SQS
• YouTube - AWS Lambda SQS Integration

Comments

[Victor Dorneanu]

I think this is a very good example explaining the main concepts! While skimming through your code, I have following observations:

• you might want to use AWS SDK v2
• for better testability you might also want to abstract away 3rd-party dependencies (such as SNS, SQS etc.) from your real/business code
  • usually you do this by using interfaces (which implement the exact methods you need from services like SQS/SNS) and use these interfaces in your application layer
  • this way you can better test your code while at the same time not being dependant on "real" calls to the SQS/SNS API

[Linielson Rosa]

Thank you for your comment, good points noted.
I will apply these improvements.