DEV Community: t.okazaki

Save the HTTP request body received by Go Gin to AWS Kinesis Firehose

t.okazaki — Sat, 15 May 2021 11:10:35 +0000

I wanted to run Kinesis Firehose in Go, but I couldn't find any sample code.
Here is a simple example.

package main

import (
    "fmt"
    "encoding/json"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/awserr"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/firehose"
    "github.com/gin-gonic/gin"
)

type EntryRecord struct {
    PostBodyField string `json:"entry"`
    Host string `json:"host"`
    RemoteAddr string `json:"remoteaddr"`
}

const (
    deliveryStreamName = "firehose-dest-bucket"
)

func handleEntry(c *gin.Context) {

    buf := make([]byte, 2048)
    n, _ := c.Request.Body.Read(buf)
    body_filed := string(buf[0:n])
    fmt.Println(body_filed)

    entry_record := EntryRecord{
        PostBodyField: body_filed,
        Host: c.Request.Host,
        RemoteAddr: c.Request.RemoteAddr,
    }

    firehose_svc := firehose.New(session.New(), aws.NewConfig().WithRegion("ap-northeast-1"))
    record := &firehose.Record{}
    json, _ := json.Marshal(entry_record)

    record_byte := append([]byte(json), []byte("\n")...)

    record.SetData(record_byte)

    _, err := firehose_svc.PutRecord(
        &firehose.PutRecordInput{
            DeliveryStreamName: aws.String(deliveryStreamName),
            Record:             record,
        },
    )
    if err != nil {
        if awsErr, ok := err.(awserr.Error); ok {
            print(awsErr.Message())
        }
    }
}

func main() {
    r := gin.Default()
    r.POST("/submit", handleEntry)
    r.Run() 
}

Prerequisites

You have already set up Kinesis Firehose
You have already set up S3 bucket to store the request body data.

The following is the part that is sending data to Firehose.

&firehose.PutRecordInput{
            DeliveryStreamName: aws.String(deliveryStreamName),
            Record:             record,
        },

The argument Record is a structure defined in the SDK.

    record := &firehose.Record{}
    json, _ := json.Marshal(entry_record)

    record_byte := append([]byte(json), []byte("\n")...)

    record.SetData(record_byte)

Reference

AWS Go SDK

Creating AWS EKS cluster with Fargate

t.okazaki — Wed, 19 Aug 2020 11:21:10 +0000

Recently I've been experimenting with a configuration using Fargate with AWS EKS (Kubernetes version 1.17). I think I can reduce the burden of infrastructure management, by adoption Fargate which can worker nodes be managed. I wondered which terraform and eksctl are nicer to create Fargate on AWS EKS.

By the way, Fargate only supports ALB as a load balancer, and EKS only supports the Classic Load Balancer as the default.
In order to use Fargate and ALB, you need to follow the following documentation to deploy the ALB Ingress Controller on an EKS cluster.
https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

The above AWS documentation uses eksctl. When I created an EKS cluster with eksctl, I could do the procedure smoothly.
However, when I created an EKS cluster with Terraform, I was not able to build the ALB Ingress Controller with above documantation because the EKS cluster was not created by eksctl which uses CloudFormation stack.

So my conclusion, the following step would be easier to create and manage the structure of Fargate with EKS.
First, build VPC, subnetworks, databases and any AWS resources using Terraform.
Second, build an EKS cluster or ALB Ingress Controller using eksctl.

My Website about Terraform examples

t.okazaki — Sun, 16 Aug 2020 06:39:41 +0000

I recently created the following web page, instead of posting on DEV.

https://www.learningcloudinfra.tokyo/en/aws/

I built various AWS resources using Terraform for my own learning and summarized what I built in each posts.
Each code is available under the MIT license on my Github repository.
Some of the code examples are simply to launch EC2 and RDS or to launch the Go application with Fargate.
I will continue to update this website with articles like EKS and other AWS resources.

By the way, this site was created using Hugo and Netlify.
It's easy for me to build a website because I can write articles in Markdown and deploy them just pushing Github ;)

Vuls: Open-source vulnerability scanner

t.okazaki — Tue, 14 Jul 2020 22:54:18 +0000

Vuls is an open-source vulnerability scanner. It automates security vulnerability checks on the software installed on a system.
Vuls comes with an agent-less architecture, meaning that it uses SSH to scan remote hosts.

Vuls checks the following vulnerability information sources
see: https://github.com/future-architect/vuls#high-quality-scan

How to install

We will install Vuls in the AWS EC2 AmazonLinux2.

Logging on an EC2 instance by ec2-user

$ sudo yum -y install docker git
$ sudo usermod -aG docker ec2-user
$ cd /home/ec2-user/
$ git clone https://github.com/vulsio/vulsctl.git

# logging out and logging in the instance again.

$ sudo systemctl start docker

$ cd vulsctl && ./update-all.sh

# it takes about 20-30 minitues.

Preparation

After Vuls is installed, we prepare the configuration file.

$ cp -p config.toml.template config.toml
$ vim config.toml

specify default section.
Note that if you place the SSH key in your /home/ec2-user/.ssh/id_rsa, you have to write this way.
Because Vuls runs on Docker container and it mounts SSH key on "/root/.ssh/id_rsa" inside the container.

[default]
port               = "22"
user               = "ec2-user"
keyPath            = "/root/.ssh/id_rsa"
scanMode           = ["fast"]

And you write the hostname or IP address of servers which you want to scan.

[servers.name]
host                = "10.10.1.251"

If you want to scan local host, you need to specify the IP address which allocated to the interface instead of "127.0.0.1" .

Setting SSH keys

You have to register your public key to known_hosts of the scanned servers. To do this, you logging on the server onece or use following command.

Generate a key pair locally.

$ ssh-keygen -t rsa -b 4096

ssh-copy-id ${USER}@${target_host}

Add the target host in local known_hosts file.

$ ssh-keyscan ${target_host} >> ~/.ssh/known_hosts

Scanning

Just execute the following shell script.

$ ./scan.sh -vvv

Using default tag: latest
latest: Pulling from vuls/vuls
Digest: sha256:e39edb92833e7d6f6490620e11221f1a456ca2dec4f5f3ab1c15e12c75ecdcbb
Status: Image is up to date for vuls/vuls:latest
docker.io/vuls/vuls:latest
[Jul 11 10:39:44]  INFO [localhost] Validating config...
[Jul 11 10:39:44]  INFO [localhost] Detecting Server/Container OS...
[Jul 11 10:39:44]  INFO [localhost] Detecting OS of servers...
[Jul 11 10:39:47]  INFO [localhost] (1/1) Detected: name: amazon 2 (Karoo)
[Jul 11 10:39:47]  INFO [localhost] Detecting OS of containers...
[Jul 11 10:39:47]  INFO [localhost] Checking Scan Modes...
[Jul 11 10:39:47]  INFO [localhost] Checking dependencies...
...(snip)...
[Jul 11 10:39:52]  INFO [localhost] Scanning vulnerabilities...
[Jul 11 10:39:52]  INFO [localhost] Scanning vulnerable OS packages...
[Jul 11 10:39:52]  INFO [name] Scanning in fast mode

One Line Summary
================
name    amazon2 (Karoo) 451 installed, 16 updatable

To view the detail, vuls tui is useful.
To send a report, run vuls report -h.

You can see the scan result on the command line.

$ ./report.sh

Using default tag: latest
latest: Pulling from vuls/vuls
...(snip)...
name (amazon2 (Karoo))
======================
Total: 10 (High:3 Medium:4 Low:3 ?:0), 10/10 Fixed, 451 installed, 16 updatable, 0 exploits, 0 modules, en: 0, ja: 2 alerts

+----------------+------+--------+-----+--------+---------+-------------------------------------------------+
|     CVE-ID     | CVSS | ATTACK | POC |  CERT  |  FIXED  |                       NVD                       |
+----------------+------+--------+-----+--------+---------+-------------------------------------------------+
| CVE-2018-20060 |  9.8 |  AV:N  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2018-20060 |
| CVE-2019-17041 |  9.8 |  AV:N  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2019-17041 |
| CVE-2019-17042 |  9.8 |  AV:N  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2019-17042 |
| CVE-2019-6477  |  7.8 |  AV:N  |     | JPCERT |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2019-6477  |
| CVE-2020-12762 |  7.8 |  AV:L  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2020-12762 |
| CVE-2018-5745  |  7.5 |  AV:N  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2018-5745  |
| CVE-2019-6465  |  7.5 |  AV:N  |     | JPCERT |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2019-6465  |
| CVE-2020-0543  |  6.5 |  AV:L  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2020-0543  |
| CVE-2020-0549  |  6.5 |  AV:L  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2020-0549  |
| CVE-2020-0548  |  5.5 |  AV:L  |     |        |   fixed | https://nvd.nist.gov/vuln/detail/CVE-2020-0548  |
+----------------+------+--------+-----+--------+---------+-------------------------------------------------+

The report files are generated in the directory named "results". Note that the report files and directory will be made as a root owner. If you want to access the files, you may change the permission the directory.

$ sudo chown -R ec2-user:ec2-user results/

VulsRepo: watch the result on your browser

You can also view the scan result on your browser using VulsRepo.
Github: https://vuls.io/docs/en/vulsrepo.html

$ cd /home/ec2-user/
$ git clone https://github.com/ishiDACo/vulsrepo

$ cd vulsrepo/server
$ cp vulsrepo-config.toml.sample vulsrepo-config.toml

$ vi vulsrepo-config.toml

[Server]
rootPath = "/home/ec2-user/vulsrepo"
resultsPath  = "/home/ec2-user/vulsctl/results
serverPort  = "5111"

$ ./vulsrepo-server
 [ec2-user@ip-10-10-1-82 server]$ ./vulsrepo-server
2020/07/11 10:53:11 main.go:153: INFO: RootPath Load:  /home/ec2-user/vulsrepo
2020/07/11 10:53:11 main.go:160: INFO: ResultsPath Load:  /home/ec2-user/vulsctl/results
2020/07/11 10:53:11 main.go:128: Start: Listening port: :5111

Access the server with port 5111 on your browser.
Actual screen images are shown in the official documentation.

Performance Test: Basics

t.okazaki — Fri, 10 Jul 2020 21:56:35 +0000

I have been working as an engineer for a cloud-based SaaS system. As the amount of data increases, I have encountered problems in the system. For example, the web servers are not down, but it's response time becomes extremely long. I come up with learning about performance testing because I wanted to anticipate these future problems in advance and prepare trump cards. In this article, I would like to summarize performance testing.

Purpose of Performance Testing

The goal of performance testing is to measure the performance of the system and plan to increase system's availability. We look at this following point of views.

To estimate the response performance of the system in each of the various use cases.
Improve system performance under high load
Ensure that the system is scalable.
Check the scale characteristics of the system.

In a cloud environment, web severs can be provisioned quickly and can be scaled out after actual user access occurs. However, depending on the application and the configuration of the middleware, scaling out in a hurry after the load has increased will not well handle traffics because of lacking of scalability.

System Performance Metrics

As performance metrics, throughput and latency should be considered.

Throughput

For a web application, this often indicates the number of HTTP requests to be processed per second. The throughput of the overall system is constrained by the throughput of the bottleneck portion of the system. We investigate where the bottleneck is located and improve it's throughput. It is important to correctly identify where the bottleneck is. If you get it wrong, the solution will not only be ineffective, but it can cause severe congestion at worse.

Latency

It is a processing time. There are two aspects:

Processing time from the user's point of view. The time between when the user sends a request and when the response is received.
Processing time in the system. This is the time between when the system sends the request and when it returns the response.

In a performance test, we will focus on the processing time in the system (2.) because it is difficult to simulate actual network conditions from the user to the system.

Latency is different from throughput improvement; reducing the latency at one location reduces the latency of the entire system.

Characteristics of scalability

As Characteristics of scalability, we consider What parts of the system need to be augmented to increase performance. For example, we think, can we increase the throughput by increasing the number of web servers? Do we need to increase the memory size of the database at the same time?

We want to mainly know following perspectives:

Optimal infrastructure configuration to handle several throughput levels (100rps, 500rps,,,)
The actual performance limit of the web system

It is difficult to know exactly what the marginal performance is. Because we don't know what our future application code or system configuration or users will look like.　However, it is meaningful to investigate the system performance. Because in the process of investigating the critical performance, it is possible to discover some improvements or unexpected bottlenecks.

Conclusion

At this post, I wrote about the purpose of performance testing, system metrics, and the system scalability these for the basis of performance testing. I will write about how to plan a performance test in the future post.

AWS Codebuild: Prepare your own custom build environment

t.okazaki — Sat, 04 Jul 2020 14:46:49 +0000

AWS Codebuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy.You can create custom build environments that use your own build tools.

This feature caught my attention because I am a little bothered that existing fully managed CI service has changed their build environment and our development team took a time to repair the build script.
( Of ourse, you can also get started quickly by using prepackaged build environments on Codebuild )

How to prepare your own custom build environment

We have to prepare a Docker image for custom build environment. Hopefully, AWS give us sample docker images. You can get them by following.

$ git clone https://github.com/aws/aws-codebuild-docker-images.git
$ cd aws-codebuild-docker-images
$ cd ubuntu/standard/4.0
$ docker build -t aws/codebuild/standard:4.0 .
$ ls -1
Dockerfile
amazon-ssm-agent.json
dockerd-entrypoint.sh
legal
runtimes.yml
ssh_config
tools

$ docker run -it --entrypoint sh aws/codebuild/standard:4.0 -c bash

In the Dockerfile, various programming language packages, browsers and another build tools are installed
For example:

ChromeDriver, Chrome, Firefox
Java, Ruby, Python, PHP, Golang, Nodejs

You can add or choose packages which you need by customising above Dockerfile.

Local debugging your custom build environment

You can run Codebuild on your laptop using Docker. You can test and build your custom Codebuild docker image and your application locally before committing.

Pull the docker image of the local CodeBuild agent.

docker pull amazon/aws-codebuild-local:latest --disable-content-trust=false

Download the shell script named codebuild_build.sh from AWS GIthub
https://github.com/aws/aws-codebuild-docker-images/tree/master/local_builds

./codebuild_build.sh -i codebuild4-test:1.0 -a /tmp -s /work/repo -m

options

-i Specify docker image tag of your custom build environment image.
-a Used to specify an artifact output directory.
-s Used to specify a source directory. Defaults to the current working directory.
-m Used to mount the source directory to the customer build container directly.
-c Use the AWS configuration and credentials from your local host. This includes ~/.aws and any AWS_* environment variables.

References

AWS Codebuild documentation

Launch an AWS Spot Fleet instance using Terraform

t.okazaki — Sat, 27 Jun 2020 15:25:00 +0000

Today, I share a way to launch an AWS instance using Terraform, which has following features:

Use EC2 Spot fleet instance which is cheaper than the on-demand instance.
Use AWS Session Manager to connect to the instance, so that you don't need to prepare the SSH setting. Also you don't need any bastion host to connect the instance.

Example

You can download the Terraform code on my Github repository.
Terraform example

I made these code with Terraform version v0.12.25.

Download the repository and type following command.

$ cd terraform-examples/spot-ssm-example
$ terraform init
$ terraform plan -var-file=aws.tfvars
$ terraform apply -var-file=aws.tfvars

After that you can connect the instance via Session Manager console in AWS console.You find the way on following short video and AWS document.

References:

AWS EC2 Spot instance document

AWS System Manager Session Manager document

Traffic Mirroring with GoReplay

t.okazaki — Sun, 21 Jun 2020 09:35:10 +0000

What is GoReplay?

Github: https://github.com/buger/goreplay

It captures HTTP requests received by the server and then duplicate the traffic to another server.
For example, when you test the new version of your application on the stating environment, you can check the behavior of the application by replicating the trrafic using GoReplay.
This can be done without affecting existing application and traffics. (Because GoReplay uses libpcap, which retrieves packets at the L2 level, as well as tcpdump)
Solves the problem of shadow proxies which is used for similar purposes. ( it can be a critical path of the traffic.)

You can do for example,

Capture HTTP request packets and duplicate them
Save the packet on the file, restoring it from the file
Filtering request packets
Rewriting request headers

How to install

You can download rpm, deb and tar.gz from
https://github.com/buger/gor/releases

For AmazonLinux, you can get a tar.gz of the executable binary. If you get the tar.gz and decompress it, you get "gor".

Simple Usage.

$ sudo ./gor --input-raw :8000 --output-stdout

By default, no response contents are received.
you can receive them by adding --output-http-track-response option.

Replaying Packets.

$ sudo ./gor --input-raw :8000 --output-http="http://localhost:8001"

Save the requests
to a file

$ sudo ./gor --input-raw :8000 --output-file=requests.gor

Restore and send the requests stored in the file

$ sudo ./gor --input-file requests.gor --output-http="http://localhost:8001"

You can also specify multiple IP addresses to send requests
by round-robin feature.

The response body is designed to receive up to 200KB by default.

Supports BASIC authentication. You can specify user:pass@ before the URL.
You can specify user:pass@ before the URL.

$ sudo ./gor --input-raw :80 --output-http "http://user:pass@staging.com"

Filtering Requests.
Replay only requests to /api

$ sudo ./gor --input-raw :8080 --output-http staging.com --http-allow-url /api

Excluding only requests to /api

$ sudo ./gor --input-raw :8080 --output-http staging.com --http-disallow-url /api

Example Use Case 1

Capture HTTP packets to /production/submission on port 80
The path changes from /production/submission to /staging/submission/
Saving the source IP addresses of the packets
Replay the packet to https://staging.abc.com

$ sudo ./gor --input-raw :80 --output-http 'https://staging.abc.com '
--http-allow-url /production/submission
--http-rewrite-url /production/submission:/staging/submission --input-raw-realip-header "X-Real-IP"

Example Use Case 2

Capture the requests and save it to a file.
AUTH-TOKEN is rewritten to an arbitrary value for the test user on the staging environment.
Do not save requests for sign_in, sign_out operation.

$ sudo ./gor --input-raw :80 --output-file=requests.gor 
--input-raw :80 --output-file=requests.gor
--http-set-header "X-HTTP-AUTH-TOKEN: abcdefghijk"
--http-disallow-url /sign_in
--http-disallow-url /sign_out