DEV Community: Vladyslav Budichenko

Deploy Go AWS lambda function using Terraform

Vladyslav Budichenko — Tue, 14 Feb 2023 20:54:52 +0000

Overview

In this article, we'll deploy a simple Golang hello-world lambda function into AWS using Terraform and cover all the necessary steps to do that. All resources we deployed have a free tier and thus will cost us 0$.

If you want to not miss similar articles subscribe to receive the new articles updates in TheDevBook 📩.

AWS Lambda is a popular cloud computing service that also allows developers to run their code without the need to manage infrastructure. With AWS Lambda, you only pay for the computing resources you consume, making it a cost-effective solution for running your applications. For more details about AWS lambda check What is AWS lambda.

In this tutorial, we will be exploring how to deploy a Lambda function using Terraform, a powerful tool for provisioning infrastructure as code. Terraform is a popular open-source tool that enables you to define and manage your infrastructure as code. With Terraform, you can define and manage all the components of your infrastructure, including AWS Lambda functions, in a single file. This makes it easy to version control your infrastructure and manage changes over time.

Prerequisites

To deploy the lambda function we need to have an AWS account and set up Terraform first. A detailed guide on how to do that can be found in the guide in configure environment section.

Note: We are going to use a profile tutorial-terraform-profile with admin permissions, that is not the best way to use in production, because of security concerns, but convenient for test purpose.

We also need a Go installed for the tutorial to compile our go code. Go installation instructions can be found in official docs.

Structure of the project

The project structure contains multiple terraform files and a lambda directory with a code of hello-world Go lambda function.

tutorial/
|-- lambda/
|   `-- hello-world
|      `-- main.go
|-- iam.tf
|-- lambda.tf
|-- locals.tf
`-- main.tf

Basic setup

First, we need to provide a basic setup of terraform, specify the dependencies we are going to use, and set up the AWS provider.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
    archive = {
      source = "hashicorp/archive"
    }
    null = {
      source = "hashicorp/null"
    }
  }

  required_version = ">= 1.3.7"
}

provider "aws" {
  region  = "us-east-1"
  profile = "tutorial-terraform-profile"

  default_tags {
    tags = {
      app = "tutorial-terraform"
    }
  }
}

The terraform block defines the Terraform configuration itself and specifies the required Terraform providers and required terraform version, which is >= 1.3.7. The required_providers section lists the providers that will be used in the Terraform configuration. In the tutorial we need:

aws - to create AWS resources;
archive - manage archive files, such as ZIP files, within Terraform; in our case, we need to archive Go binary in zip format to upload to lambda;
null - run local commands; in our case, we are going to build a go binary using go build command.

The provider "aws" block specifies the AWS provider settings, which include the AWS region (us-east-1) and the AWS profile (tutorial-terraform-profile) to use for authentication.

Note: You need to specify the appropriate profile you have in .aws/config for the terraform aws provider

The default_tags section sets the default tags that will be applied to AWS resources created by Terraform, with the tag app = "tutorial-terraform" in this case.

Tags mostly used for filter resources and track usage and costs

Identity and access management setup

// allow lambda service to assume (use) the role with such policy
data "aws_iam_policy_document" "assume_lambda_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

// create lambda role, that lambda function can assume (use)
resource "aws_iam_role" "lambda" {
  name               = "AssumeLambdaRole"
  description        = "Role for lambda to assume lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_lambda_role.json
}

We use assumeRole for principal lambda to allow lambdas this particular role. It means this role can be assumed only by lambdas.

data "aws_iam_policy_document" "allow_lambda_logging" {
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = [
      "arn:aws:logs:*:*:*",
    ]
  }
}

// create a policy to allow writing into logs and create logs stream
resource "aws_iam_policy" "function_logging_policy" {
  name        = "AllowLambdaLoggingPolicy"
  description = "Policy for lambda cloudwatch logging"
  policy      = data.aws_iam_policy_document.allow_lambda_logging.json
}

// attach policy to out created lambda role
resource "aws_iam_role_policy_attachment" "lambda_logging_policy_attachment" {
  role       = aws_iam_role.lambda.id
  policy_arn = aws_iam_policy.function_logging_policy.arn
}

Go lambda function code

The hello world function for lambda in Go is a simple executable code in main package:

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/lambda"
)

func HandleRequest(ctx context.Context, event interface{}) (string, error) {
    fmt.Println("event", event)

    return "Hello world", nil
}

func main() {
    lambda.Start(HandleRequest)
}

We use github.com/aws/aws-lambda-go/lambda package to make out main function be executed as lambda. We need to run in main function lambda.Start(HandleRequest) to process events that trigger the lambda. In the handler function HandleRequest, we will receive the context of the event and the event itself.

Go lambda function resource

To deploy the AWS Lambda function we need to create a binary for our lambda and upload it in a zip format:

// build the binary for the lambda function in a specified path
resource "null_resource" "function_binary" {
  provisioner "local-exec" {
    command = "GOOS=linux GOARCH=amd64 CGO_ENABLED=0 GOFLAGS=-trimpath go build -mod=readonly -ldflags='-s -w' -o ${local.binary_path} ${local.src_path}"
  }
}

// zip the binary, as we can use only zip files to AWS lambda
data "archive_file" "function_archive" {
  depends_on = [null_resource.function_binary]

  type        = "zip"
  source_file = local.binary_path
  output_path = local.archive_path
}

// create the lambda function from zip file
resource "aws_lambda_function" "function" {
  function_name = "hello-world"
  description   = "My first hello world function"
  role          = aws_iam_role.lambda.arn
  handler       = local.binary_name
  memory_size   = 128

  filename         = local.archive_path
  source_code_hash = data.archive_file.function_archive.output_base64sha256

  runtime = "go1.x"
}

We first compiled the binary using null_resource to execute the local command and run Go build. After that, we will have a binary tf_generated/hello-world.

After this, we need to archive our binary in zip as lambda accepts only zip archive uploads. For that we use archive_file and specify file to our binary file.

To create the AWS Lambda function we need to use the aws_lambda_function resource. The function's name, description, IAM role, handler, memory size, zip archive, and runtime are all specified in this step.

In addition to the Lambda function, we need to create a
CloudWatch Log Group, using the aws_cloudwatch_log_group resource, to gather logs from the deployed function. This allows you to monitor and debug the function's performance in a centralized manner.

// create log group in cloudwatch to gather logs of our lambda function
resource "aws_cloudwatch_log_group" "log_group" {
  name              = "/aws/lambda/${aws_lambda_function.function.function_name}"
  retention_in_days = 7
}

Deploy Go lambda function

Before deploying the resources, we need to initialize terraform:

terraform init

After that, we are ready to plan our resources. That means we are not going to create them first, but only to validate out terraform code is correct and check the resources that will be created:

terraform plan

And we will see something like:

...

Terraform will perform the following actions:

  # data.archive_file.function_archive will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "archive_file" "function_archive" {
      + id                  = (known after apply)
      + output_base64sha256 = (known after apply)
      + output_md5          = (known after apply)
      + output_path         = "./tf_generated/hello-world.zip"
      + output_sha          = (known after apply)
      + output_size         = (known after apply)
      + source_file         = "./tf_generated/hello-world"
      + type                = "zip"
    }

  # aws_cloudwatch_log_group.log_group will be created
  + resource "aws_cloudwatch_log_group" "log_group" {
      + arn               = (known after apply)
      + id                = (known after apply)

...

Plan: 6 to add, 0 to change, 0 to destroy.

In total, we are going to create 6 resources:

the binary file of compiled Go code
zip archive of the binary
iam policy for logs
iam role for lambda
iam policy attachment (that's a separate resource)
lambda function

That looks correct and we can create our resources. We need to run:

terraform apply

and type yes to confirm applied resources.

Note: we can use -auto-approve for auto approve, but it is more appropriate for automated runs in CI/CD pipeline. For manual deployments better to avoid this parameter to double-check the resources that are going to be created.

Validation of lambda function deployment

After deployment is ok, we can check the deployed lambda. We need to Go to lambda section in the search of AWS console:

There we can find hello-world function.

Going inside the function we can see the details of the lambda function version. To test it there is a tab called Test:

We need to specify event-name, event JSON and click Test to run the function:

After that, we see the execution summary. The function returns a "Hello world" message 🎉

In the execution summary, we can see the duration of the function run and the billing duration (for what time we are going to pay). There is a direct link to the logs of the function.

In the logs, we can see the event, that we passed to the function:

Destroy resources

To destroy all resources we need to run:

and type yes to confirm.

Conclusion

We deployed an AWS Lambda function with Terraform. By following the steps in this tutorial, you'll be able to manage your Lambda functions with ease and ensure consistency across your infrastructure. And let's be real, who doesn't love a little automation in their life?

💡 If You Liked The Article

If you like the article and want to stay updated on the latest and greatest in software engineering, make sure to subscribe for new articles updates in TheDevBook.

Happy coding 💻

Learn how to scrape Trustpilot reviews using Go

Vladyslav Budichenko — Sat, 04 Feb 2023 22:25:37 +0000

Overview

Web scraping is a powerful tool for extracting information from websites. With web scraping, you can gather data from websites and use it for a variety of purposes, such as data analysis, machine learning, and research. The process of web scraping involves making HTTP requests to a website, parsing the HTML content, and extracting the desired information.

Go is a popular programming language that is well-suited for web scraping due to its efficiency, scalability, and built-in libraries. In this article, we will explore the process of web scraping using Go and learn the best programming practices for extracting information from websites.

Whether you are a seasoned developer or a beginner, this article will provide you with the knowledge and resources you need to get started with web scraping using Go. So let's dive in and learn how to extract information from websites with Go!

If you don't want to read more similar tutorials subscribe to the newsletter in TheDevBook.

Now let's take a look at scraping.

Trustpilot page overview for scraping

Before we start writing code to extract information from a website, it is important to understand the structure of the page we are going to parse.

The HTML source code of a website defines the structure, layout, and content of the page. To inspect the HTML source code of a website, right-click anywhere on the page and select "Inspect" (both Chrome and Firefox).

This will open development tools with all the HTML code that makes up the page.

Once we have the HTML source code, we need to identify the information we want to extract. The basics of the HTML is not a goal of the article, but for more information there are good explanation from Mozilla or W3Schools.

From the page structure we can see, that each review that card is a div element with classes styles_cardWrapper__LcCPA and styles_reviewCard__9HxJJ. That means we can extract the card by using a queries, like jQuery to extract the particular block. After that we can extract the text, that contains in the block and save it as a review data.

Generally, by understanding the HTML structure and using HTML selectors, we can extract the information we want from a website. In the next sections, we will learn how to write Go code to make HTTP requests, parse HTML, and extract data.

Scraping tools in Go

There are several different libraries, that are commonly used for web scraping in Go:

net/http - standard library in Go that provides the ability to make HTTP requests and receive HTTP responses. It is a low-level library, which means that it requires a bit more code to use than some other libraries, but it is also very flexible and can be used for any type of HTTP request.
github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.
github.com/PuerkitoBio/goquery - library that provides a convenient and concise way to query HTML and XML documents. It provides a jQuery-like API for selecting elements and extracting data, making it a popular choice for web scraping in Go.
github.com/yhat/scrape - library that provides a convenient and flexible way to extract information from websites. It provides a high-level API that makes it easy to extract specific elements from a web page, and it also supports custom extractors for more complex scraping tasks.

In the tutorial, we are going to use simple net/http for requesting pages and github.com/PuerkitoBio/goquery to make queries for the HTML document.

Scraping the page

Let's go top-down through the code idea.

First, we need to identify the review internal structure. We checked that it should contain fields like text, date, title, etc, so let's create a Go representation of the Review and all product Reviews structure:

type Review struct {
    Text   string `json:"text"`
    Date   string `json:"date"`
    Rating string `json:"rating"`
    Title  string `json:"title"`
    Link   string `json:"link"`
}

type ProductReviews struct {
    ProductName string    `json:"product_name"`
    Reviews     []*Review `json:"reviews"`
}

It is JSON serializable using go json tags.

For the purpose of the article, let's get the particular product invideo.io and extract it with URL constant as a separate section:

const (
    scrapingURL     = "https://www.trustpilot.com/review/%s"
    scrapingPageURL = "https://www.trustpilot.com/review/%s?page=%d"
    productName     = "invideo.io"
)

Now let's write a main function with a general idea of what we need:

func main() {
    log.Printf("Start scraping reviews for %s", productName)

    productReviews, err := getProductReviews(productName)
    if err != nil {
        log.Fatal(err)
    }

    jsonFile, err := os.Create(fmt.Sprintf("trustpilot_reviews_%s.json", productName))
    if err != nil {
        log.Fatal(err)
    }
    defer jsonFile.Close()

    jsonEncoder := json.NewEncoder(jsonFile)
    err = jsonEncoder.Encode(productReviews)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Successfully scraped %d reviews for %s", len(productReviews.Reviews), productName)
}

We get all the reviews using getProductReviews function and after that save them into a JSON file. Now we need to create a function to extract all reviews for a particular product. First, we need to get the HTLM page and transform it into a goquery document:

productURL := fmt.Sprintf(scrapingURL, name)
// make a request to the product page
res, err := http.Get(productURL)
if err != nil {
    return nil, err
}
defer res.Body.Close()

// transform the HTML document into a goquery document which will allow us to use a jquery-like syntax
doc, err := goquery.NewDocumentFromReader(res.Body)
if err != nil {
    return nil, err
}

Note: Trustpilot doesn't block simple requests at the moment of writing the article, but many big services will block such requests, as it's definitely done not via browser user. In such a case we need to use another approach to extracting page URLs, like using scraping tools or headless browsers and proxies.

After extracting the document we can need to prepare a reviews slice, where we want to store all the reviews. As we are going to make requests to Trustpilot in parallel for different pages and slice if not concurrency safe (that means we cannot simply write into it in parallel, as it causes data race conditions), we need a mechanism to synchronize the data. Here, we are going to use channels.

reviews := make([]*Review, 0)
// we synchronize reviews processing with a channel, as we scrape reviews from multiple pages in parallel
reviewsChan := make(chan *Review)
quitChan := make(chan struct{})

// we append reviews in a separate goroutine from reviewsChan
go func() {
    for review := range reviewsChan {
        reviews = append(reviews, review)
    }

    close(quitChan)
}()

Now we can use goquery the document we created and extract elements we are interested in, particularly all div elements that contain classes with prefixes styles_reviewCard__ and styles_cardWrapper__. Let's extract all the div and process them:

// to avoid one extra request, we process first page here separately
doc.Find("div").Each(extractReviewFunc(reviewsChan, productURL))

Let's dive deep into the extractReviewFunc function. In the implementation, we filter classes of the page element and identify if it contains necessary prefixes:

classes, exists := s.Attr("class")
if !exists {
    return
}

// validate if the div is a review card and a card wrapper (to avoid processing other divs, like advertisement)
isReviewCard := false
isCardWrapper := false

for _, class := range strings.Split(classes, " ") {
    if strings.HasPrefix(class, "styles_reviewCard__") {
        isReviewCard = true
    }

    if strings.HasPrefix(class, "styles_cardWrapper__") {
        isCardWrapper = true
    }
}

if !isReviewCard || !isCardWrapper {
    return
}

If it is a review card, we can extract all the necessary information using the element selection and send it to reviewsChan:

// extract review data
dateOfPost := s.Find("time").AttrOr("datetime", "")
textOfReview := s.Find("p[data-service-review-text-typography]").Text()

title := s.Find("h2").Text()
link, _ := s.Find("a[data-review-title-typography]").Attr("href")
if link != "" {
    link = productURL + link
}

// we don't transform the data in place, as we want to keep the original data for future analysis
rating := s.Find("img").AttrOr("alt", "")

reviews <- &Review{
    Text:   textOfReview,
    Date:   dateOfPost,
    Rating: rating,
    Title:  title,
    Link:   link,
}

As doc.Find("div").Each takes a func(i int, s *goquery.Selection) as input, and we use external parameters inside the function, particularly reviewsChan and productURL, we need to wrap such a function into another function that can return it. It's a common pattern in Go when we need to create a function with a predefined signature and pass specific parameters to use inside it.

func extractReviewFunc(reviews chan<- *Review, productURL string) func(i int, s *goquery.Selection) {
    return func(i int, s *goquery.Selection) {
        // our function code here
    }
}

The whole function code looks like

func extractReviewFunc(reviews chan<- *Review, productURL string) func(i int, s *goquery.Selection) {
    return func(i int, s *goquery.Selection) {
        classes, exists := s.Attr("class")
        if !exists {
            return
        }

        // validate if the div is a review card and a card wrapper (to avoid processing other divs, like advertisement)
        isReviewCard := false
        isCardWrapper := false

        for _, class := range strings.Split(classes, " ") {
            if strings.HasPrefix(class, "styles_reviewCard__") {
                isReviewCard = true
            }

            if strings.HasPrefix(class, "styles_cardWrapper__") {
                isCardWrapper = true
            }
        }

        if !isReviewCard || !isCardWrapper {
            return
        }

        // extract review data
        dateOfPost := s.Find("time").AttrOr("datetime", "")
        textOfReview := s.Find("p[data-service-review-text-typography]").Text()

        title := s.Find("h2").Text()
        link, _ := s.Find("a[data-review-title-typography]").Attr("href")
        if link != "" {
            link = productURL + link
        }

        // we don't transform the data in place, as we want to keep the original data for future analysis
        rating := s.Find("img").AttrOr("alt", "")

        reviews <- &Review{
            Text:   textOfReview,
            Date:   dateOfPost,
            Rating: rating,
            Title:  title,
            Link:   link,
        }
    }
}

Now we need only extract the number of pages and run the same logic for other pages with reviews, as they have the same HTML structure. For that we need to process a pagination link element:

// we need to find a link to last page and extract the number of pages for the product
doc.Find("a[name='pagination-button-last']").Each(extractReviewsOverPagesFunc(reviewsChan, name))

For extractReviewsOverPagesFunc we use a similar approach, as we did for extractReviewFunc. We extract a href attribute of our a element and using regular expressions to get to the last page of the product.

href, exists := s.Attr("href")
if !exists {
    return
}

// we need to find a link to pages and extract the number of pages for the product
match, err := regexp.MatchString("page=\\d+", href)
if err != nil || !match {
    return
}

re := regexp.MustCompile("\\d+")
lastPage := re.FindString(href)
lastPageInt, err := strconv.Atoi(lastPage)
if err != nil {
    log.Printf("Cannot parse last page %s: %s\n", lastPage, err)

    return
}

When we know the number of pages we need to process, we can do that in parallel using wait groups to wait while all goroutines finish.

// scrape all pages in parallel
wg := &sync.WaitGroup{}
for i := 2; i <= lastPageInt; i++ {
    wg.Add(1)
    go func(pageNumber int) {
        defer wg.Done()

        pageReviews, err := getPageProductReviews(name, pageNumber)
        if err != nil {
            log.Printf("Cannot get page %d product reviews: %s", pageNumber, err)

            return
        }

        for _, review := range pageReviews {
            reviews <- review
        }
    }(i)
}

wg.Wait()

We start from the second page, as the first one (the current page, from where we extracted the last page) was already processed.

The getPageProductReviews is a similar function, as getProductReviews but we don't extract a last link to get all pages:

func getPageProductReviews(name string, page int) ([]*Review, error) {
    log.Printf("Start scraping page %d for %s", page, name)

    // productURL is used to construct a link to the review. It's pure, without query params
    productURL := fmt.Sprintf(scrapingURL, name)
    // actual request URL for scraping a page
    productRequestURL := fmt.Sprintf(scrapingPageURL, name, page)
    res, err := http.Get(productRequestURL)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()

    doc, err := goquery.NewDocumentFromReader(res.Body)
    if err != nil {
        return nil, err
    }

    reviews := make([]*Review, 0)
    reviewsChan := make(chan *Review)
    quitChan := make(chan struct{})

    go func() {
        for review := range reviewsChan {
            reviews = append(reviews, review)
        }

        close(quitChan)
    }()

    // extract reviews from the page
    doc.Find("div").Each(extractReviewFunc(reviewsChan, productURL))

    close(reviewsChan)
    <-quitChan

    return reviews, nil
}

As we can notice, there is one thing we didn't mention in the function getPageProductReviews yet, particularly the channels synchronization part:

close(reviewsChan)
<-quitChan

After running all goroutines, we close the reviewsChan, as we won't write into it anymore and wait until we process all the reviews and save them into reviews slice.

Basically, that's it. We now can extract reviews and dump them into a JSON object for future analysis. The final code can be found on GitHub.

Let's run it and check the data. After execution, there is a new file trustpilot_reviews_invideo.io.json. There is a bunch of data, and it looks like

{
  "product_name": "invideo.io",
  "reviews": [
    {
      "text": "excellent application, a bit expensive for my economic situation but if you have the money it is an application that will help you in a simple way in editing videos",
      "date": "2023-01-26T01:10:22.000Z",
      "rating": "",
      "title": "excellent application",
      "link": "https://www.trustpilot.com/review/invideo.io/reviews/63d1b6de9b64b1bdaf432bef"
    },
    ...
  ]
}

Conclusion

In this article, we have covered the basics of web scraping and how to parse Trustpilot reviews using Go. We've discovered how to check the HTML structure of a particular website, identify the necessary elements we need to extract the data, and explored some of the most popular libraries for web scraping in Go.

As we move forward, I encourage you to continue learning and experimenting with web scraping. This is a powerful and valuable skill, and there are endless opportunities to apply it in your work and personal projects, like lead generation, trends analysis, generally gathering databases for data analysis, etc.

If you want to not miss other similar articles, sign up for the newsletter in TheDevBook.

Writing a simple AI-powered chrome extension for LinkedIn

Vladyslav Budichenko — Sun, 29 Jan 2023 14:40:00 +0000

Introduction

A wonderful method to improve the user experience and add unique features to the browser is to build a Chrome Extension. It might add a special feature. The increased development of AI technologies opens up a number of beneficial productivity extensions.

In this tutorial, we'll access the LinkedIn page and extract the post text using JavaScript and basic web scraping. We'll also use GPT-3 to generate comments on LinkedIn. We'll go over the key elements of Chrome extensions and build a simple one. The backend part and prompt engineering are not covered in this tutorial; instead, they will be covered separately. Here, we will only use a ready-to-use endpoint as a black box.

If you want to check the final extension work, you can check the video:

Things to know before start coding

Before we start coding, there are several important parts of chrome extensions concept that we need to discover and understand. If you know that you can feel free to skip that section.

Structure of chrome extension

Chrome extensions has a common structure that contains essential parts of the app:

manifest.json - The manifest of the extension is the only required file with a specific file name: manifest.json. It must also be in the extension's root directory. The manifest contains important metadata, defines resources, declares permissions, and specifies which files should run in the background and on the page.
content.js - Content scripts are files that run in the context of web pages. By using the standard Document Object Model (DOM), they are able to read details of the web pages the browser visits, make changes to them, and pass information to their parent extension. More in docs. In our case we will put the content-related logic, such as parsing the LinkedIn page, adding custom buttons to the comment section and extracting post text.
background.js - background service worker. It is used to run tasks continuously, even when the extension's popup or options pages are not open. In our case, this file will handle the background processes of the extension, such as making API calls to the backend endpoint and processing the response. As we want to make API request to out custom endpoint, we need to do that separately from service worker, to prevent cross origin requests.

There is also a popup part, that defines the logic of Chrome extension popup, that we will skip, as in our tutorial we don't need any popup UI and logic for the app.

We add file styles.css to the structure to make it not so boring and improve UI part. This file provides the CSS styles for the extension's additional elements on the LinkedIn page. It will be used to style the custom buttons added to the comment section.

More details about the structure of the of chrome extension could be found in the official docs.

Manifest.json overview

The manifest.json file is a configuration file for a chrome extension that contains important information about the extension such as its name, version, and permissions required. In version 3 of the manifest file, there are several new features and changes to the format, including support for extension modules and service worker scripts, as well as increased security measures.

One of the most important sections in the manifest.json file is the permissions section, where the extension specifies what permissions it needs in order to function properly. For example, an extension that needs to access a user's browsing history would need the "history" permission. In our case we only need tabs permission to send events between service worker and content page.
All permissions can be checked in the official docs.

Another important part to make API calls is
host_permissions. It contains one or more match patterns that give access to specified hosts. In our case we host our backend endpoint on AWS and need to specify that as https://*.amazonaws.com/* to make it possible to interact with the API.
Note: Better to provide a particular pattern with minimum usage of *, like https://12345.amazonaws.com/* as we minimise the potential surface of security issues. For the tutorial purpose we simplify this part.

We need also specify content script and styles file paths, that we are going to use to inject custom logic and styles into the page in content_scripts section.

Finally, we have next manifest.json file:

{
  "manifest_version": 3,
  "name": "LinkBuddy",
  "version": "0.0.1",
  "permissions": [
    "tabs"
  ],
  "host_permissions": [
    "https://*.amazonaws.com/*"
  ],
  "content_scripts": [
    {
      "matches": ["https://www.linkedin.com/*"],
      "css": ["styles.css"],
      "js": ["content.js"]
    }
  ],
  "action": {
    "default_title": "LinkBuddy"
  },
  "background": {
    "service_worker": "background.js"
  }
}

Event messaging in chrome extension

One more concept to understand before start is message passing or event messaging.

Event messaging is a key concept in building a Chrome extension. It allows different parts of the extension, such as the content script and the background script, to communicate and exchange data.

In a simple example, let's say we have a button on a website that a user can click. The button is in the content script and when it is clicked, it sends a message to the background script to perform a specific action.

When the button is clicked, the content script uses the chrome.runtime.sendMessage() function to send a message to the background script. The message can be in the form of an object that contains any data that needs to be passed.

document.getElementById("button").addEventListener("click", function() {
    // we send message of particular type "button-clicked"
    chrome.runtime.sendMessage({
        type: "button-clicked"
    });
});

In the background script, we listen for the message using the chrome.runtime.onMessage.addListener function. When the message is received, we can perform the desired action based on the data in the message.

chrome.runtime.onMessage.addListener(
    // we listen to all messages and process them
    function(request, sender, sendResponse) {
        if (request.type === "button-clicked") {
            // Perform desired action
        }
    }
);

This is just a simple example, but event messaging can be used in a variety of ways in a Chrome extension. It can be used to send data between different parts of the extension, trigger actions, or even send data to a server. The key is to use a consistent naming convention for the message and its data, so that the different parts of the extension can understand it.

Writing the content logic

Once we have the structure of the extension set up, understand messaging system in the extension and wrote manifest json we are good to start.

Data flow overview

The data flow of the extension is described below. Below is a description of the extension's data flow. Knowing what parts pass what data and how they interact with one another is crucial when designing an application.

Listen to focusin event on the page. If it is a comment section add 3 buttons with event onClick event handlers.
When button is clicked collect text data from post, generate request body and send it as generate-comment event to background worker using message passing.
On background worker listen to the generate-comment event and make an API call with the provided body. Send a response as generate-comment-response event to content.
On generate-comment-response event trigger type the text of comment in the comment section.

LinkedIn page analysis

To extract post text and identify comment section to insert custom button and comment we need to discover them on page and scrape them after that. To do the we need to use elements tab from Chrome dev tools.

In the inspect section of chrome we can discover that all the feed page split into same blocks and comment sections are implemented as custom div with class ql-editor.

To extract for the particular comment the relevant post section we can find the parent container of the whole post, that is a div with class feed-shared-update-v2 and extract text from div with class feed-shared-update-v2__description-wrapper.

Content.js script

Now we can write some code of the extension.

Listen to `focusin` event

In order to make our chrome application interactive, we need to implement logic for extracting elements from the page and responding to user actions. One way to do this is by using event listeners.

We need to use the focusin event listener to detect when a user has clicked on a specific element on the page. We can then extract that element and perform some action, such as displaying information about the element or modifying its properties.

Note: It's important to note that, in order to use event listeners in a chrome extension, we need tabs permission in manifest.json.

In content.js we need to add the code:

document.addEventListener("focusin", function(event) {
    // the focused element is `event.target`
});

Add buttons

As we discovered during LinkedIn page analysis, to identify that the focusin target is a comment section, it should be a div with class ql-editor. So we need to add to content.js:

if (event.target.classList.contains("ql-editor")) {
    // process the div as a comment field
}

After that we can add 3 buttons. In the code example below we add only one button, but other are added in the same way. The processing code looks like:

if (event.target.classList.contains("ql-editor")) {
    const parentForm = event.target.closest(".comments-comment-texteditor");

    if (parentForm && !parentForm.classList.contains("buttons-appended")) {
        // add appended class to add buttons only on the first event trigger
        parentForm.classList.add("buttons-appended");

        // create and append engage button
        let engageBtn = document.createElement("button");
        engageBtn.classList.add("rounded-button");
        engageBtn.innerText = "🤝 Engage";

        parentForm.appendChild(engageBtn);

        engageBtn.addEventListener("click", function(event) {
            processButtonClicked(event, "engage", parentForm);
        })
    } else {
        console.log("No parent with the class 'comments-comment-texteditor' found for the focused element.");
    }
}

We get out buttons:

Let's add some beauty to that with styles.

Add styles to the page

To add custom styling we need to append an additional stylesheet link to the head section of the page. We need to add to content.js:

let link = document.createElement("link");
link.setAttribute("rel", "stylesheet");
link.setAttribute("href", chrome.runtime.getURL("styles.css"));
document.head.appendChild(link);

As styles we have button style and simple pulse animation in styles.css:

.rounded-button {
    border-width: 1px;
    /*border-color: #000000;*/
    border-color: rgba(0,0,0,0.3);
    border-style: solid;
    margin: 10px 3px 10px 3px;
    padding: 5px 10px 5px 10px;
    border-radius: 20px;
}

.first-rounded-button {
    margin-left: 10px;
}

.loading-animation {
    box-shadow: 0 0 0 0 rgb(247, 77, 77);
    transform: scale(1);
    animation: pulse_animation 2s infinite;
}

@keyframes pulse_animation {
    0% {
        transform: scale(0.95);
        box-shadow: 0 0 0 0 rgb(247, 77, 77);
    }

    70% {
        transform: scale(1);
        box-shadow: 0 0 0 10px rgba(0, 0, 0, 0);
    }

    100% {
        transform: scale(0.95);
        box-shadow: 0 0 0 0 rgba(0, 0, 0, 0);
    }
}

.disabled {
    opacity: 0.8;
    pointer-events: none;
}

As the result we will have such buttons:

Button click processing

Button click processing handles by processButtonClicked function.

To avoid simultaneous request we need to check if there is already clicked button handler running. If there is, we simply return and do nothing to prevent multiple comments from being generated simultaneously. Also, we disable all other buttons on the page to ensure that only one action can be performed at a time.

// global in content.js
let loading = false

// function processButtonClicked
if (loading) {
    console.log('already loading');

    return;
}

document.querySelectorAll(".rounded-button").forEach(function(button) {
    button.setAttribute("disabled", true);
    button.classList.add("disabled");
});

Not to be boring, we add a pulse animation to the clicked button to provide visual feedback to the user that the extension is working.

event.currentTarget.classList.add("loading-animation");

We then extract the full text of the parent post and remove any "see more" text that may be present. This text will be used in the generation of the comment.

// extract full text of the parent post
const parent = event.currentTarget.closest(".feed-shared-update-v2");
const elements = parent.getElementsByClassName("feed-shared-update-v2__description-wrapper")
let text = elements[0].innerText;
const textWithoutSeeMore = text.replace(/…see more/g, "");

Finally, we save the current state of the app and send a message to the background script, passing the button type and the text of the parent post.

// save current state of the app
loading = true
processButton = event.currentTarget
processParent = parentForm

// send the event
chrome.runtime.sendMessage({
    type: "generate-comment",
    buttonType: buttonType,
    event: event,
    parentForm: parentForm,
    text: textWithoutSeeMore,
});

The final code of the function is:

function processButtonClicked(event, buttonType, parentForm) {
    // check if we already loading the response
    if (loading) {
        console.log('already loading');

        return;
    }

    // disable all other buttons to avoid multiple comments creation simultaneously
    document.querySelectorAll(".rounded-button").forEach(function(button) {
        if (button.id !== "expertBtn") {
            button.setAttribute("disabled", true);
            button.classList.add("disabled");
        }
    });

    // add pulse animation to the clicked button
    event.currentTarget.classList.add("loading-animation");

    // extract full text of the parent post
    const parent = event.currentTarget.closest(".feed-shared-update-v2");
    const elements = parent.getElementsByClassName("feed-shared-update-v2__description-wrapper")
    let text = elements[0].innerText;
    const textWithoutSeeMore = text.replace(/…see more/g, "");

    // save current state of the app
    loading = true
    processButton = event.currentTarget
    processParent = parentForm

    // send the event
    chrome.runtime.sendMessage({
        type: "generate-comment",
        buttonType: buttonType,
        event: event,
        parentForm: parentForm,
        text: textWithoutSeeMore,
    });
}

Background.js script

In the background.js we write a simple background worker logic.

When event listener receives "generate-comment" message, it calls the processGenerateCommentRequest function with the message's data.

chrome.runtime.onMessage.addListener(async (request, sender, sendResponse) => {
    switch (request.type) {
        // listen to the event
        case "generate-comment":
            await processGenerateCommentRequest(request);

            break;
        default:
            console.log('unknown request type', request.type);
    }
});

This function makes a POST request to an API endpoint with text of the post and button type, process the response and sends the result back to the extension's front-end.

async function processGenerateCommentRequest(request) {
    const config = {
        text: request.text,
        commentType: request.buttonType,
    }

    const requestOptions = {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify(config)
    };

    let response = {
        type: "generate-comment-response",
        error: "something went wrong",
    };
    try {
        // send API request
        let res = await fetch(`${apiBaseURL}/comment`, requestOptions);

        // process the results
        const results = await res.json()
        response = {
            type: "generate-comment-response",
            parentForm: request.parentForm,
            comment: results.results.comment,
        }
    } catch (error) {
        response = {
            type: "generate-comment-response",
            error: error,
        };
    }

    // send the event with response
    chrome.tabs.query({
        active: true,
        currentWindow: true
    }, function(tabs) {
        chrome.tabs.sendMessage(tabs[0].id, response, function(response) {
            console.log('send response', response)
        });
    });
}

Receive comment in `content.js` and display it

The final step of building our chrome extension is receiving the event in the content.js script and displaying the comment.

To do this, we use the chrome.runtime.onMessage.addListener function to listen for messages sent from other scripts. In this case, we are listening for messages with the type generate-comment-response, which we sent from the background.js script.

chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    switch (request.type) {
        case "generate-comment-response":
            // stop loading process and enable all buttons
            loading = false;
            processButton.classList.remove("loading-animation");

            document.querySelectorAll(".rounded-button").forEach(function(button) {
                button.removeAttribute("disabled");
                button.classList.remove("disabled");
            });

            if (request.error) {
                console.error(request.error);

                return
            }

            emulateWriting(processParent, request.comment);

            break;
        default:
            console.log('unknown request type', request.type);
    }
});

After that we process the message and if it is successful type it into the comment section using emulateWriting function.

function emulateWriting(parentElement, text) {
    let input = parentElement.querySelector(".ql-editor.ql-blank p");
    let i = 0;
    let interval = setInterval(() => {
        if (i < text.length) {
            input.innerText += text[i];
            i++;
            for (let j = 0; j < 10; j++) {
                if (i < text.length) {
                    input.innerText += text[i];
                    i++;
                }
            }
        } else {
            clearInterval(interval);
            // we need to remove `ql-blank` style from the section by LinkedIn div processing logic
            input.parentElement.classList.remove("ql-blank");
        }
    }, 10);
}

Conclusion

We have explained the basic concepts of the chrome extension and covered the steps to build a simple one that generates comments using the power of GPT-3.

Using a similar approach we can make personalized processing of different social media, summaries of articles, generation of other types of messages, and whatever you can think about. Of course, the quality of the response could be good enough, but it is mostly for drafts of some comments and still need to be checked by the human first. But the results for such a simple project look great.

Thank you for reading this tutorial, and if you have any questions or feedback, please don't hesitate to reach out. To read more similar tutorial articles or subscribe to TheDevBook blog 📩.

Happy coding! 💻

Deploy a static website on AWS with terraform

Vladyslav Budichenko — Thu, 25 Aug 2022 23:10:46 +0000

Introduction

In the previous article, we deployed an API as a docker container on AWS EC2. In the current article, we are going to deploy a simple website on AWS without the backend (so next we can add it 😉). We will distribute the content using AWS S3 and AWS CloudFront (content delivery network service). As a frontend will be used an open-source Start Bootstrap example with an MIT license. Everything we deploy in the article is supported by AWS free tier. All the code related to the article can be found in GitHub repo.

Important notes

We are going to simplify some parts of the system to improve it in other articles. Because of these simplification current code is not recommended to use for production system as is and need to be improved. Some of the important simplifications that we use:

Content type serving was done as a simple map with the default text/htmp; charset=UTF-8 value.
We serve much more files that need to be served so some extra files are exposed.
We don’t encrypt buckets and don’t have versioning to simplify removing the bucket.
Usage of the default CloudFront certificate is not recommended as it uses an old SSL version. In production system should be used an ACM certificate.

As we don’t have any backend we are not going to use AWS WAF as the most useful thing it can do here is bot protection. It will be overhead for the simple static website. CloudFront already provides a decent level of level 3 and 4 DoS attacks.

Prerequisite

In the article, we are going to use terraform and need to have an AWS account. If you don’t have them installed and want to try to deploy everything by yourself you can find the installation instruction in the first article.

Code

For the frontend we use an example of an admin dashboard made with bootstrap. It is an MIT-licensed open-source project that can be found in the GitHub repo.

Overview

The basic architecture of the project is described in the diagram.

We are going to upload all our website files to AWS S3. To serve the website we will use AWS CloudFront to reduce latency and provide a caching layer for the static content. To protect the access to S3 objects we are going to use IAM so the object can be accessed by customers only through CloudFront. The main components we will review are:

CloudFront - an AWS content delivery network (CDN). We are going to use it to serve our website and reduce latency to provide a good user experience.
AWS S3 - storage for all the static data. The bucket is blocked for all public access and the objects could be accessed only from CloudFront.

Deploy and distribute static content

S3

As we described before, there are 2 important parts we need. First is our S3 bucket, that we need to create:

resource aws_s3_bucket static {
  // important to provide a global unique bucket name
  bucket = "boodyvo-go-example-static"
}

To make the bucket private and remove all potential access (except AWS God account access), we need to update the bucket:

resource aws_s3_bucket_acl static {
  bucket = aws_s3_bucket.static.id
  acl    = "private"
}

resource aws_s3_bucket_public_access_block website_bucket_public_access_block {
  bucket                  = aws_s3_bucket.static.id
  ignore_public_acls      = true
  block_public_acls       = true
  restrict_public_buckets = true
  block_public_policy     = true
}

Another thing that is recommended to do is to enforce bucket ownership for all objects that are added to the bucket. It will disable ACL usage so the bucket always has full control over objects and manage access to them via privileges.

resource aws_s3_bucket_ownership_controls meta_static_resources {
  bucket = aws_s3_bucket.static.bucket

  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

To serve our website from S3 we need to upload all files that we have we need to upload all of them first. Based on our project structure, we are going to upload all the files in frontend/ directory using fileset terraform function. It’s important for serving the files via browser to provide a correct content_type. As mentioned in a simplification part at the beginning, we are going to provide a content_type based on simple mapping where "text/html; charset=UTF-8" is a default value. The map is represented with content_type_map local variable.

locals {
  content_type_map = {
    css:  "text/css; charset=UTF-8"
    js:   "text/js; charset=UTF-8"
    svg:  "image/svg+xml"
  }
}

resource aws_s3_bucket_website_configuration static {
  bucket = aws_s3_bucket.static.id

  index_document {
    suffix = "index.html"
  }

  routing_rule {
    redirect {
      replace_key_with = "index.html"
    }
  }
}

resource aws_s3_object assets {
  for_each = fileset("${path.module}/../frontend", "**")

  bucket = aws_s3_bucket.static.id
  key    = each.value
  source = "${path.module}/../frontend/${each.value}"
  etag   = filemd5("${path.module}/../frontend/${each.value}")

  // simplification of the content type serving
  content_type = lookup(
    local.content_type_map,
    split(".", basename(each.value))[length(split(".", basename(each.value))) - 1],
    "text/html; charset=UTF-8",
  )
}

Note: That is enough for the example project, but for production project map could be way more granular or the dist can be more specified based on paths. If we don’t control the object upload process via Lambda@Edge or Lambda functions during uploading the file on S3 depends on the content. Moreover, we need to filter out dev files, like .scss, license, readme, etc

Another thing we need to do to make it possible to serve our content using CloudFront is to create an Origin Access Identity (OAI). It is a policy that restricts access to the bucket so only CloudFront can access objects. In such a way the end user can access the objects only through CloudFront and cannot bypass it.

// --- cloudfront.tf ---

locals {
  s3_origin_id = "s3-origin-example"
}

resource aws_cloudfront_origin_access_identity frontend {
  comment = "OAI for S3 frontend"
}

// --- s3.tf ---

data aws_iam_policy_document oai_access_policy {
  statement {
    actions   = ["s3:GetObject"]
    // as we use the bucket only for static content we provide an access for all objects in the bucket
    resources = ["${aws_s3_bucket.static.arn}/*"]

    principals {
      type        = "AWS"
            // the identity specifies in cloudfront.tf
      identifiers = [aws_cloudfront_origin_access_identity.frontend.iam_arn]
    }
  }
}

resource aws_s3_bucket_policy oai_access {
  bucket = aws_s3_bucket.static.id
  policy = data.aws_iam_policy_document.oai_access_policy.json
}

One more thing we are going to simplify is related to CORS, as we will provide a possibility to access our website from everywhere by allowed_origins = ["*"].


resource aws_s3_bucket_cors_configuration website {
  bucket = aws_s3_bucket.static.id

  cors_rule {
    allowed_headers = ["*"]
    allowed_methods = ["GET"]
    allowed_origins = ["*"]
    expose_headers  = []
    max_age_seconds = 3600
  }
}

CloudFront

CloudFront provides a possibility to create a distribution, that configures how the content should be distributed and served. In such a way we control geo-restrictions, caching rules, request redirection rules, security, access, etc. We are going to create a distribution.

resource aws_cloudfront_distribution frontend {
  origin {
    domain_name = aws_s3_bucket.static.bucket_regional_domain_name
    origin_id   = local.s3_origin_id

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.frontend.cloudfront_access_identity_path
    }
  }

  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Distribution for example website"
  default_root_object = "index.html"

  aliases = []

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = local.s3_origin_id

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  }

  price_class = "PriceClass_100"

  viewer_certificate {
    cloudfront_default_certificate = true
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  wait_for_deployment = true
}

There are several places that we can pay attention to in this distribution resource:

price_class = "PriceClass_100" represents the Price Class of the distribution. In other words, it specifies where the content is cached and served. We chose the most simple option when we serve the data in North America, Europe, and Israel. To make it more distributed we can upgrade the class and cover many more regions.
aws_cloudfront_origin_access_identity specifies the OAI that we described before in the S3 section. The identity will provide an access to S3 bucket objects to CloudFront distribution.
viewer_protocol_policy = "redirect-to-https" configure redirection of HTTP requests to HTTPS to force HTTPS connection.
viewer_certificate {cloudfront_default_certificate = true} configures the default CloudFront certificate. It is not recommended to use it for production because of security reasons, as it is used in the old version of the TLS protocol.

Except for those important points we also specified cache behavior, geo-restriction, and default index page.

We want to have a URL of the distribution as an output to use it to test our website:

output service_ip {
  value = aws_cloudfront_distribution.frontend.domain_name
}

Deploying

To deploy everything we need to run from the directory 2_static_website:

terraform -chdir=iaac init  # initialization of terraform modules
terraform -chdir=iaac apply # applying the resources

The distribution takes can take about 20 minutes to be created. After that we will have an output with a host_name value, that represents the URL of the website.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

host_name = "dqiv81ppt2ffj.cloudfront.net"

Using the URL of our distribution dqiv81ppt2ffj.cloudfront.net from output we can access the website. It should look like this:

To destroy the deployment run:

terraform -chdir=iaac destroy

Afterward

In the article, we deployed the simple static web application and distributed it using AWS CDN service CloudFront. We made some simplifications and going to improve the deployment in the future.

If you have any suggestions for improvement or topics, or any questions don’t hesitate to reach me in the comments 👇😊.

Deploying Go application on AWS with terraform

Vladyslav Budichenko — Mon, 18 Jul 2022 23:43:23 +0000

Overview

In the article, we will deploy a simple Golang service on elastic cloud compute (EC2) in AWS. As we deploy the application as a docker image, we can run almost any application inside a docker container in the similar way.

We won’t deploy a highly available and reliable project. The goal of the article is to get familiar with terraform as infrastructure as a code (IaaC) tool and how to deploy a simple application to AWS.

All the code can be found in the GitHub repository. We are going to use it to go through the terraform code in other sections of the article.

Simple server

We are not going to dive deep into the application For the creation we will use the simplest gin API server.

router.GET("/", server.CounterHandler)
router.GET("/health", server.HealthHandler)

// HealthHandler returns a success message with code 200 when the server is running
func (s *Server) HealthHandler(ctx *gin.Context) {
    ctx.JSON(200, gin.H{"success": true})
}

// CounterHandler calculates number of requests and return a json with counter number
func (s *Server) CounterHandler(ctx *gin.Context) {
    counter := atomic.AddInt64(&s.counter, 1)

    ctx.JSON(200, gin.H{"counter": counter})
}The implementation of handlers

Configure environment

Before starting to provision the infrastructure we need to set up all tools we are going to use: AWS account, terraform, and docker.

Create an AWS account

If you don’t have an account on AWS you need to create one first. You can find the instruction in the official AWS guide.

Create a role for the terraform with permissions

It’s highly bad practice to use a root AWS user for anything except critical account management tasks, so we are going to create a new user inside the AWS account that terraform will use for the creation of the infrastructure. It can be done via identity and access management (IAM) service.

After that, we can create a new user.

Provide a User name and choose only programmatic access for the user, as it will be used only via terraform.

Add admin group to the user using search so terraform can provision all resources.

Note: One of the security best practices is the least privilege principle, when only necessary permissions should be provided to each service and user. In the example, for simplicity, we provide full admin control to the user to keep it simple.

For step 3, where we set tags, we can set project: go-example or something similar. Tags in AWS are mostly used for filtering and grouping resources and gathering analytics, like costs.

Step 4 is validation, where we can check all the data.

During step 5 we need to store the Access key ID and Secret access key. They will provide us with programmatic access to the AWS account.

Note: We will see the secret access key only once so it should be stored in a durable and secure place.

Create ~/.aws/config and ~/.aws/credentials to use profiles

To use the AWS profile we need to configure it. The AWS configuration is stored in the ~/.aws directory and consists of two important files:

~/.aws/config - stores the AWS configuration. We will store the profile with the name go-example in the config file, and it should be like this:

[profile go-example]
region=eu-central-1
output=json

~/.aws/credentials - stores credentials for profiles. We should create the file and copy access and secret keys.

[go-example]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>

Install terraform

We will use Terraform for deployment. To install terraform follow the official instruction.

Tip: For more real project usage it’s more convenient to use https://github.com/tfutils/tfenv, as it can manage multiple versions of Terraform for many projects.

Docker

To build docker images we need to install docker using official instruction.

Set up the infrastructure

There are several possibilities to deploy an application on AWS: using AWS Elastic Beanstalk, AWS Elastic container service (ECS), or directly on an EC2 instance. We are going to use the last approach and deploy the application on the system and for simplicity wrap it with a docker.

We are going to

Set up a virtual private cloud (VPC), Internet gateway (IG), network access control list (NACL), and security groups to protect network access to our instance.
Create a docker image for our application and upload the image to the elastic container registry (ECR), which is an AWS registry for images (similar to the Docker hub).
Set up an EC2 instance and use run our application as a docker inside it.

In this case, we can manage versions of the application. With ECR it’s easier to use ECS, as a container management system. But we are going to set up the EC2 instance, as a virtual machine, and run an app as a docker container inside it.

Note: ECR is not in the free tier, so you will pay several cents per month for the usage.

Set up network

We are going to create a new VPC with an Internet gateway and one subnet, a basic security group, and NACL. The network will be simple as we are trying to avoid complexity here. The security group will allow connection only to the application port (80), while NACL will allow all the traffic for the subnet.

Note: There is a better practice to keep the network traffic inside AWS for better security and access management. It can be done using the VPC endpoint - a private connection, that allows traffic between AWS services not to leave the AWS network. We are not going to create a VPC endpoint to allow internal connection to ECR, so we are going to download the docker image via the internet. For the production system, better to keep the connection private to improve security.

All the network setup can be found in [network.tf](http://network.tf) file in the Github repo. First, we need a new VPC resource with a defined CIDR block.

resource aws_vpc main {
  cidr_block = "10.0.0.0/20"
}

VPC is an logically isolated virtual network. Here we can set up different subnets, that can be public or private, and manage traffic as we need. VPC doesn't have an access to the Internet. As we expect incoming traffic from the Internet we need an Internet gateway (IG).

resource aws_internet_gateway gateway {
  vpc_id = aws_vpc.main.id
}

There is only a default route table initially in the VPC. To provide an access to the internet via IG we need to add a route table and associate it with the VPC. In this case, all the traffic that is not local (in our case, not 10.0.0.0/20) we need to route to IG.

resource aws_route_table public {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gateway.id
  }
}

resource aws_main_route_table_association main {
  route_table_id = aws_route_table.public.id
  vpc_id         = aws_vpc.main.id
}

We will have only one EC2 instance in the example, so we are going to create one subnet in one availability zone (AZ) - for AWS it is similar to one big data center in a particular region. To provide a network routing in the subnet we also need to make an association with the router main we created before.

resource aws_subnet main {
  vpc_id = aws_vpc.main.id
  cidr_block = "10.0.0.0/24"
  map_public_ip_on_launch = true
  availability_zone = "us-east-1a"
}

resource aws_route_table_association main {
  subnet_id      = aws_subnet.main.id
  route_table_id = aws_route_table.public.id
}

By default, all traffic inside the created subnet will be denied (terraform updates the default VPC during creation) and we need to update the default NACL to allow incoming and outgoing traffic and associate it with the subnet.

resource aws_default_network_acl main {
  default_network_acl_id = aws_vpc.main.default_network_acl_id

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }
}

resource aws_network_acl_association main {
  network_acl_id = aws_default_network_acl.main.id
  subnet_id = aws_subnet.main.id
}

Security group allows stateful connection to AWS services, in our case to the future instance. The ingress rule (for incoming connections) should allow HTTP (TCP) traffic to port 80 and egress traffic (as we are going to pull docker image).

resource aws_security_group public {
  name = "public-sg"
  vpc_id = aws_vpc.main.id

  ingress {
    from_port = local.application_port
    to_port   = local.application_port
    protocol  = "tcp" // allows only tcp trafic for the provided port

    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port   = 0
    protocol  = "-1" // allows all outgoing traffic from the instance

    cidr_blocks = ["0.0.0.0/0"]
  }
}

Note: Security groups can only allow some traffic and they are stateful (for example, if the request was made the response will pass without any additional rules). NACL can deny traffic and they are stateless (for example, if the request was made we need separate rules for both incoming and outgoing traffic).

Create a docker image

We are going to wrap our application with docker and create a docker image. To store this image we need an image repository. AWS has a service ECR for that purpose, that we are going to create and use for the application. We need to allow ECR actions that are described in aws_iam_policy_document to manage docker images.

resource aws_ecr_repository go_server {
  name = local.service_name
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

data aws_iam_policy_document ecr_policy {
  statement {
    effect = "Allow"
    principals {
      identifiers = ["*"]
      type        = "*"
    }

    actions = [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
      "ecr:PutImage",
      "ecr:InitiateLayerUpload",
      "ecr:UploadLayerPart",
      "ecr:CompleteLayerUpload",
      "ecr:DescribeRepositories",
      "ecr:GetRepositoryPolicy",
      "ecr:ListImages",
      "ecr:DeleteRepository",
      "ecr:BatchDeleteImage",
      "ecr:SetRepositoryPolicy",
      "ecr:DeleteRepositoryPolicy"
    ]
  }
}

resource aws_ecr_repository_policy go_server {
  repository = aws_ecr_repository.go_server.name
  policy = data.aws_iam_policy_document.ecr_policy.json
}

For the docker management, we are going to use terraform module kreuzwerker/docker. It provides us an opportunity to build and upload docker image to a docker repository.

We can create a terraform provider that will manage the authentication to ECR

data aws_ecr_authorization_token go_server {
  registry_id = aws_ecr_repository.go_server.registry_id
}

provider docker {
  registry_auth {
    address  = split("/", local.ecr_url)[0]
    username = data.aws_ecr_authorization_token.go_server.user_name
    password = data.aws_ecr_authorization_token.go_server.password
  }
}

To build the image from docker file we need a separate docker_registry_image resource, that will build the image and upload it to ECR.

resource docker_registry_image go_example {
  name = "${local.ecr_url}:v1"

  build {
    context    = "${path.module}/../app/."
    dockerfile = "app/Dockerfile"
    no_cache   = true
  }

  depends_on = [aws_ecr_repository.go_server]
}

Set up the app in EC2

Before creating an instance and running docker container inside it we need to allow the EC2 instance to pull the image from the repository. AWS has identity and access management service (IAM) to control all access between AWS services. To allow EC2 to pull images from ECR we need to create a separate IAM role - an identity that has particular permissions.

data aws_iam_policy_document assume_role_ec2_policy_document {
  statement {
    effect = "Allow"
    principals {
      identifiers = ["ec2.amazonaws.com"]
      type        = "Service"
    }

    actions = [
      "sts:AssumeRole"
    ]
  }
}

resource aws_iam_role role {
  name               = "application-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role_ec2_policy_document.json
}

To provide permission to pull docker image we need to create an IAM policy and attach it to the role we created.

// policy allows only ecr:ecr:BatchGetImage action for all resources that have such permission.
data aws_iam_policy_document ecr_access_policy_document {
  statement {
    effect = "Allow"
    actions = [
      "ecr:BatchGetImage",
    ]

    resources = ["*"]
  }
}

resource aws_iam_policy ecr_access_policy {
  name = "ecr-access-policy"
  policy = data.aws_iam_policy_document.ecr_access_policy_document.json
}

resource aws_iam_policy_attachment ecr_access {
  name       = "ecr-access"
  roles      = [aws_iam_role.role.name]
  policy_arn = aws_iam_policy.ecr_access_policy.arn
}

For the EC2 instance, we need to create a profile with this role.

resource aws_iam_instance_profile profile {
  name = "application-ec2-profile"
  role = aws_iam_role.role.name
}

To create an EC2 instance we need to know the amazon machine image (AMI) id, which is a virtual machine where the application will be run. For that purpose, we are going to use ubuntu-20.04 image.

data aws_ami ubuntu {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

After that, we are ready to create an EC2 instance. The instance defines resources for the virtual machine. t2.micro type provides 1 vCPU, and 1 GB of RAM, available for a free tier and is enough for the simple application.
Initial script is provided in user_data - script that is run during the initializations of the EC2 instance, where we install docker, pull the app image, and run it.

resource aws_instance application {
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t2.micro"
  vpc_security_group_ids      = [aws_security_group.public.id]
  subnet_id                   = aws_subnet.main.id
  associate_public_ip_address = true
  iam_instance_profile        = aws_iam_instance_profile.profile.name

  user_data = <<EOF
#!/bin/bash
sudo apt-get update
sudo apt-get install -y pt-transport-https ca-certificates curl gnupg lsb-release software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu `lsb_release -cs` test"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo gpasswd -a $USER docker
newgrp docker

echo ${data.aws_ecr_authorization_token.go_server.password} | docker login --username=${data.aws_ecr_authorization_token.go_server.user_name} --password-stdin ${aws_ecr_repository.go_server.repository_url}

docker run -p ${local.application_port}:${local.application_internal_port} -d --restart always ${docker_registry_image.go_example.name}
EOF

  depends_on = [aws_internet_gateway.gateway]
}

To get the IP address of the created server we can create a terraform output.

output application_ip {
  value       = aws_instance.application.public_ip
  description = "Application public IP"
}

Provision resources

Before creating the resources we described in terraform file we need to initialize terraform first to install the necessary modules. From the root directory of the repo run:

terraform -chdir=iaac init

To create the resources we need to run apply and terraform will manage all the resource.

terraform -chdir=iaac apply

Before applying terraform provides a plan of changes (diff or resources it is going to do). We can go through the plan and see all the changes, and after that type yes to apply them. In the output, we will see created resources information and output with IP of the application, like:

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

Outputs:

application_ip = "44.204.255.51"

After several minutes the application can be accessed via application_ip.

To destroy all resources we need to run destroy command:

terraform -chdir=iaac destroy

Note: Terraform will create all the resources and save state terraform.tfstate and terraform.tfstate.backup. For production system state files are usually stored remotely in a durable place, like S3. In such a case multiple developers can update the environment and keep the terraform state consistent.

Аfterword

I wrote the article to share the experience of managing infrastructure via IaaC and best practices for building and deploying applications. To help with choosing topics that would be more valuable to write about share what would be an interesting improvement of the current application deployment:

Add persistent layers to the application, like database and object storage.
Scaling the server for high availability.
Adding frontend deployment.
More basic details about technologies that were used.

All tips of how to improve the content are valuable and helpful 😇

Want to check more articles or don't miss other similar articles? 📩 Subscribe to my blog.

DEV Community: Vladyslav Budichenko

Deploy Go AWS lambda function using Terraform

Overview

Prerequisites

Structure of the project

Basic setup

Identity and access management setup

Go lambda function code

Go lambda function resource

Deploy Go lambda function

Validation of lambda function deployment

Destroy resources

Conclusion

💡 If You Liked The Article

Learn how to scrape Trustpilot reviews using Go

Overview

Trustpilot page overview for scraping

Scraping tools in Go

Scraping the page

Conclusion

Writing a simple AI-powered chrome extension for LinkedIn

Introduction

Things to know before start coding

Structure of chrome extension

Manifest.json overview

Event messaging in chrome extension

Writing the content logic

Data flow overview

LinkedIn page analysis

Content.js script

Listen to focusin event

Add buttons

Add styles to the page

Button click processing

Background.js script

Receive comment in content.js and display it

Conclusion

Deploy a static website on AWS with terraform

Introduction

Important notes

Prerequisite

Code

Overview

Deploy and distribute static content

S3

CloudFront

Deploying

Afterward

Deploying Go application on AWS with terraform

Overview

Simple server

Configure environment

Create an AWS account

Create a role for the terraform with permissions

Create ~/.aws/config and ~/.aws/credentials to use profiles

Install terraform

Docker

Set up the infrastructure

Set up network

Create a docker image

Set up the app in EC2

Provision resources

Аfterword

Listen to `focusin` event

Receive comment in `content.js` and display it