DEV Community: Shi Han

Managing S3 bucket for Terraform backend in the same configuration

Shi Han — Mon, 31 Jan 2022 11:12:16 +0000

What is Terraform backend?

A Terraform backend is a place where Terraform uses to store its state. Terraform supports various types of backend. The default local backend stores the state files in the local filesystem. Local backend is useful for creating resources for testing purposes in a "quick and dirty" manner. However, it is common practice to use backends that store states in remote services such as Amazon S3 (S3) when working on production resources as it supports state locking and versioning. Managing S3 bucket for Terraform backend in the same configuration

The chicken-and-egg problem

The following is an example of using an S3 bucket as Terraform backend. Notice that we need to specify the name of the S3 bucket.

terraform {
  required_version = "~> 1.1.4"
  backend "s3" {
    bucket = "terraform-s3-backend-pmh86b2v"
    region = "ap-northeast-1"
    key    = "terraform.tfstate"
  }
}

The above means that we need to have the S3 bucket for the backend before writing Terraform configurations to manage other resources. What if we also want to create the S3 bucket for the backend using Terraform? A straightforward approach is to manage the S3 bucket for the backend in a separate Terraform configuration that uses a local backend. We want to show how to manage that in the same configuration as other resources in this article.

The solution

Let’s take a look at the following Terraform configuration. We are going to create two S3 buckets: one for the backend (terraform-s3-backend-pmh86b2v) and another one is the actual bucket that we need for our project (my-project-...) (If you plan to follow this “tutorial,” please change the bucket name as they need to be globally unique). Notice that at this point, we are using a local backend.

terraform {
  required_version = "~> 1.1.4"
}

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_s3_bucket" "backend" {
  bucket = "terraform-s3-backend-pmh86b2v"
  acl    = "private"
}

resource "aws_s3_bucket" "main" {
  bucket_prefix = "my-project-"
  acl           = "private"
}

We start by only creating the S3 bucket (terraform-s3-backend-pmh86b2v) for the backend using the target flag -target. We can see that the command above also creates a state file (terraform.tfstate) in our local directory.

$ terraform plan -target=aws_s3_bucket.backend -out=/tmp/tfplan
$ terraform apply /tmp/tfplan

Then, we configure Terraform to use our new S3 bucket as its backend by changing the backend block.

diff --git a/main.tf b/main.tf
index c6d846b..19547ae 100644
--- a/main.tf
+++ b/main.tf
@@ -1,5 +1,10 @@
 terraform {
   required_version = "~> 1.1.4"
+  backend "s3" {
+    bucket = "terraform-s3-backend-pmh86b2v"
+    region = "ap-northeast-1"
+    key    = "remote_terraform.tfstate"
+  }
 }

 provider "aws" {

When we run terraform plan, Terraform will detect the changes in the backend and prompt us to reinitialize. Run recommended the command terraform init to make the switch.

$ terraform plan -out=/tmp/tfplan
╷
│ Error: Backend initialization required, please run "terraform init"
│
│ Reason: Initial configuration of the requested backend "s3"
...
$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes
...

The migration copies the state file to S3. We can confirm this via the AWS CLI or from AWS Management Console.

$ aws s3 ls terraform-s3-backend-pmh86b2v
2022-01-31 19:46:06       1734 remote_terraform.tfstate

After the migration is complete, we can use Terraform (plan, apply, etc.) as usual.

How to migrate from the S3 backend to the local backend?

As Terraform will detect the changes in the backend, change the backend block back to local to migrate the state file to a local directory.

diff --git a/main.tf b/main.tf
index 19547ae..e5cac0b 100644
--- a/main.tf
+++ b/main.tf
@@ -1,10 +1,6 @@
 terraform {
   required_version = "~> 1.1.4"
-  backend "s3" {
-    bucket = "terraform-s3-backend-pmh86b2v"
-    region = "ap-northeast-1"
-    key    = "remote_terraform.tfstate"
-  }
+  backend "local" {}
 }

 provider "aws" {

When running terraform plan, Terraform will notice the change in backend and prompt us to run

$ terraform init -migrate-state

Since Terraform will not remove the state files from the previous backend, we have to remove the contents in the S3 bucket before removing the bucket from the Terraform configuration. Or, we can configure the backend bucket with force_destroy = true before removing it from the Terraform.

Final remarks

As shown above, Terraform can automatically detect changes in the backend configuration. We can use this feature to help solve the chicken-and-egg problem when using the S3 bucket as Terraform backend. Please find the sample code in this article at https://github.com/shihanng/terraform-s3-backend.

Cover image by Taylor Vick, licensed under Unsplash License

Hosting personal website on DigitalOcean's App Platform

Shi Han — Sun, 10 Jan 2021 06:28:52 +0000

What I built

A simple portfolio website.

Category Submission:

Personal Site/Portfolio

App Link

https://ng.shihan.dev/

Screenshots

Description

It is a simple and clean portfolio with Gatsby, TypeScript, React, and tailwindcss. I come from a backend/infrastructure background, so frontend development is a challenge for me. I created this site without using any existing theme also made it responsive and looks good on mobile. It's not a super fanny site, but it's mine.

Link to Source Code

https://github.com/shihanng/ng.shihan.dev/

Permissive License

MIT

Background

I had a bit of free time at the end of 2020, so I've decided to rebuild my portfolio website from Hugo to Gatsby. My motivation for rebuilding is to get more practice in TypeScript / React, the skills I picked up in 2020. I also want to try out tailwindcss too. Coincidently, I found out about DigitalOcean App Platform Hackathon on DEV. So, I decided to host the new site on the platform instead of GitHub Pages.

How I built it (A detailed guide of deploying a Gatsby site to DigitalOcean’s App Platform)

I bootstrapped the site with gatsby new command. We don't need to tell Gatsby that we are using DigitalOcean's App Platform.

To create a new App, select App from the Create menu in the DigitalOcean console.

To deploy to DigitalOcean's App Platform from GitHub, we need to connect to our repository from the App console.

This means that we need to install DigitalOcean on GitHub. It can also be installed in an organization account.

We can also limit the repository that DigitalOcean has access to. E.g. for my case, I am only allowing it to access the repository that contains the source code of my site.

Once we have the permission setup on GitHub, we can then select a repository from DigitalOcean's App Platform.

Then, name our app, select a region, and pick the deployment branch. DigitalOcean will pick up any changes that I push to the trunk branch and deploy it.

We can see that DigitalOcean will be able to detect that the repository is a Gatsby project automatically. I did not need to change any settings here for my usecase.

Finally, we need to select a pricing plan. We can use the Starter plan for static sites like this one. We can build and deploy up to three Starter apps for free.

After this, we can relax and grab a cup of coffee while waiting for DigitalOcean to build and deploy the site.

Publishing DEV articles from GitHub

Shi Han — Sun, 13 Sep 2020 06:02:57 +0000

How do you write your article for DEV? Do you use the markdown editor provided by DEV on the browser? Or do you use your favorite editor on your local machine? If you do use a regular editor on your PC, how do you submit the article to DEV?

I like to use my favorite editor to write articles. When I am ready to publish an article, I have to copy and paste the texts into the online editor, then click the "Publish" button. Sometimes I want to see how DEV is showing the article while I am in the middle of the writing. Therefore, my writing experience usually involves more than one copy-pasting. As you can imagine, this is troublesome, tedious, and not a good experience at all.

I also like to use Git to manage the versions of my articles. Then I can also push the articles into GitHub. Since GitHub renders Markdown files, readers also can read the articles on GitHub if they choose to. Of course, images should work as expected on GitHub if we use relative links and commit the image files too. Now I also have a backup system for my articles.

Workflow with GitHub Actions

I want everything described above to be fully automated. The diagram above shows the workflow I want to realize with GitHub Actions. When I want to start writing a new article, I will branch out from the main branch and start writing. When I want to see how this article looks on DEV, I can push the new branch to GitHub and make a (draft) pull request. At this instance, my GitHub Action should submit the article to DEV as a draft. I can merge the branch back into the main branch when I am ready to publish the article.

`devto`: the CLI Tool

The first thing that I need to make all these happen is a CLI tool that helps me submit articles to DEV from the terminal. For this, I've created a CLI tool called devto with Go.

shihanng / devto

CLI tool to publish article to https://dev.to/

devto -- publish to dev.to from your terminal

What is this?

devto is a CLI tool that helps submit articles to DEV from the terminal. It makes use of the APIs that DEV kindly provides in OpenAPI specification. devto mainly does two things:

It collects all image links from the Markdown file into a devto.yml file with the generate subcommand. For example, if we have ./image-1.png and ./image-2.png in the Markdown file, we will get the following:
```
images
  ./image-1.png: ""
  ./image-2.png: ""
```
It submits the article to DEV with the submit subcommand. The submit subcommand creates a new article in DEV and updates the devto.yml with the resulting article_id. devto will use this article_id in the following execution to perform an update operation instead of creating a new entry for the same article.

The DEV API does not have a way of uploading…

View on GitHub

The tool makes use of the APIs that DEV kindly provides in OpenAPI specification. devto mainly does two things:

It collects all image links from the Markdown file into a devto.yml file with the generate subcommand. For example, if we have ./image-1.png and ./image-2.png in the Markdown file, we will get the following:

  images:
    ./image-1.png: ""
    ./image-2.png: ""

It submits the article to DEV with the submit subcommand. The submit subcommand creates a new article in DEV and updates the devto.yml with the resulting article_id. devto will use this article_id in the following execution to perform an update operation instead of creating a new entry for the same article.

The DEV API does not have a way of uploading images yet. If we submit a Markdown content with relative paths of image links, DEV will not be able to show those images. As a workaround of this problem, we need to provide a full path for the images either manually via the devto.yml file or using the --prefix flag. We will elaborate more on this in the GitHub Actions section.

`devto-act`: the GitHub Action

shihanng / devto-act

devto's GitHub Action

devto-act is a GitHub Action that helps you to submit your article to DEV from GitHub. It uses devto internally.

Inputs

`devto_api_key`

We need this to submit the article. You can get it from https://dev.to/settings/account.

`skip_generate`

Skip generation of links.

`published`

Make the article public if non empty.

`dry_run`

Dry run if not empty. Will not send to DEV.

`auto_prefix`

Generate prefix (cover) image links based on the branch and path to markdown files. Will be applied to the devto submit as the value of --prefix flag.

`markdown_files`

Required Path to markdown files (space separated) to be submitted. Can be relative to the root of the repository.

Example Usage

- name: Publish to DEV
  uses: shihanng/devto-act@v0.0.6
  with:
    auto_prefix: "yes"
    markdown_files: "filename.md"
  env:
    DEVTO_API_KEY: ${{ secrets.DEVTO_API_KEY }}

View on GitHub

This GitHub Action mainly uses the devto CLI tool with one "special" feature: the auto prefix functionality. If you have the following GitHub repository user/repo and you are pushing a new branch new-article for a new article, devto-act will use https://raw.githubusercontent.com/user/repo/new-article/ as the value of the --prefix flag for the submit subcommand. As a result, image links in the Markdown content, e.g.:

![Image One](./images/one.jpg)
![Image Two](./images/two.jpg)

will become

![Image One](https://raw.githubusercontent.com/user/repo/new-article/./images/one.jpg)
![Image Two](https://raw.githubusercontent.com/user/repo/new-article/./images/two.jpg)

on DEV. That means we are hosting the images on GitHub instead of DEV.

My Workflow

Using the CLI tool and the GitHub Action above, I put together a workflow that suits my needs in shihanng/articles:

name: DEV
on:
  push:
    branches:
      - "trunk"

  pull_request:
    branches:
      - "*"

jobs:
  submit:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2

      - name: Find Markdown files
        uses: technote-space/get-diff-action@v3
        id: diff
        with:
          SUFFIX_FILTER: .md

      - name: Trim output from diff
        id: trim
        run: |
          files="$(echo -n "${{ steps.diff.outputs.diff }}" | tr -d "'")"
          echo "::set-output name=trimmed::$files"

      - name: Submit to DEV as draft
        uses: shihanng/devto-act@v0.0.6
        if: ${{ github.event_name == 'pull_request' }}
        with:
          auto_prefix: "yes"
          markdown_files: ${{ steps.trim.outputs.trimmed }}
        env:
          DEVTO_API_KEY: ${{ secrets.DEVTO_API_KEY }}

      - name: Publish to DEV
        uses: shihanng/devto-act@v0.0.6
        if: ${{ github.event_name == 'push' }}
        with:
          auto_prefix: "yes"
          markdown_files: ${{ steps.trim.outputs.trimmed }}
          published: "yes"
        env:
          DEVTO_API_KEY: ${{ secrets.DEVTO_API_KEY }}

      - name: Update devto.yml
        uses: EndBug/add-and-commit@v4
        if: ${{ github.event_name == 'pull_request' }}
        with:
          author_name: ${{ secrets.AUTHOR_NAME }}
          author_email: ${{ secrets.AUTHOR_EMAIL }}
          message: "Update devto.yml"
          add: "**/devto.yml"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The workflow contains the following steps:

First, we need to know which Markdown files that we need to publish because we don't want to republish everything on every pull request. We do this with the help of technote-space/get-diff-action.
Next, we have two independent steps that help us submit articles to DEV with devto-act. One submits the article as a draft if the GitHub event is a pull request. The other one publishes the article if the event is a push (which we limit to the main branch).
Finally, we commit the changes in the devto.yml file to the same branch because we want to keep the article_id for future updates. This step is made possible with EndBug/add-and-commit.

Submission Category:

Wacky Wildcards

Ending

I've been using devto and devto-act to publish my last few articles on DEV. Although it is still not perfect, I think it is time to share it with you. If you are like me, want to write DEV articles in your favorite and manage it on GitHub, please give these two tools a try:

devto: If you want to submit the article from your terminal
devto-act: If you want to automate the publishing workflow

Feedbacks in any form (issues, PRs, comments below) are welcome. Please give these repositories a ⭐ if you find them useful. Those are always my main source of motivation!

Thank you for reading ❤️.

How to read CSV file from Amazon S3 in Python

Shi Han — Sat, 22 Aug 2020 13:38:53 +0000

Here is a scenario. There is a huge CSV file on Amazon S3. We need to write a Python function that downloads, reads, and prints the value in a specific column on the standard output (stdout).

Simple Googling will lead us to the answer to this assignment in Stack Overflow. The code should look like something like the following:



import codecs
import csv

import boto3


client = boto3.client("s3")

def read_csv_from_s3(bucket_name, key, column):
    data = client.get_object(Bucket=bucket_name, Key=key)

    for row in csv.DictReader(codecs.getreader("utf-8")(data["Body"])):
        print(row[column])

We will explore the solution above in detail in this article. Imagine this like a rubber duck programming and you are the rubber duck in this case.

Downloading File from S3

Let's get started. First, we need to figure out how to download a file from S3 in Python. The official AWS SDK for Python is known as Boto3. According to the documentation, we can create the client instance for S3 by calling boto3.client("s3"). Then we call the get_object() method on the client with bucket name and key as input arguments to download a specific file.

Now the thing that we are interested in is the return value of the get_object() method call. The return value is a Python dictionary. In the Body key of the dictionary, we can find the content of the file downloaded from S3. The body data["Body"] is a botocore.response.StreamingBody. Hold that thought.

Reading CSV File

Let's switch our focus to handling CSV files. We want to access the value of a specific column one by one. csv.DictReader from the standard library seems to be an excellent candidate for this job. It returns an iterator (the class implements the iterator methods __iter__() and __next__()) that we can use to access each row in a for-loop: row[column]. But what should we pass into X as an argument? According to the documentation, we should refer to the reader instance.

All other optional or keyword arguments are passed to the underlying reader instance.

There we can see that the first argument csvfile

can be any object which supports the iterator protocol and returns a string each time its next() method is called

botocore.response.StreamingBody supports the iterator protocol 🎉.

Unfortunately, it's __next__() method does not return a string but bytes instead.



_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

Reading CSV file from S3

So how do we bridge the gap between botocore.response.StreamingBody type and the type required by the cvs module? We want to "convert" the bytes to string in this case. Therefore, the codecs module of Python's standard library seems to be a place to start.

Most standard codecs are text encodings, which encode text to bytes

Since we are doing the opposite, we are looking for a "decoder," specifically a decoder that can handle stream data: codecs.StreamReader

Decodes data from the stream and returns the resulting object.

The codecs.StreamReader takes a file-like object as an input argument. In Python, this means the object should have a read() method. The botocore.response.StreamingBody does have the read() method: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html#botocore.response.StreamingBody.read

Since the codecs.StreamReader also supports the iterator protocol, we can pass the object of this instance into the csv.DictReader: https://github.com/python/cpython/blob/1370d9dd9fbd71e9d3c250c8e6644e0ee6534fca/Lib/codecs.py#L642-L651

The final piece of the puzzle is: How do we create the codecs.StreamReader? That's where the codecs.getreader() function comes in play. We pass the codec of our choice (in this case, utf-8) into the codecs.getreader(), which creates thecodecs.StreamReader. This allows us to read the CSV file row-by-row into dictionary by passing the codec.StreamReader into csv.DictReader:

Thank you for following this long and detailed (maybe too exhausting) explanation of such a short program. I hope you find it useful. Thank your listening ❤️.

Memory Reservation in Amazon Elastic Container Service

Shi Han — Wed, 12 Aug 2020 01:27:17 +0000

Amazon Elastic Container Service (ECS) provides the means to developers to orchestrate Docker containers running in Amazon Web Services (AWS). Using ECS develops will be able to deploy and command a fleet of Docker containers, scale the services, etc. in a very flexible manner within the ECS cluster.

We use task definition to describe how we want a Docker container to be deployed in an ECS cluster. memoryReservation is one of the container definitions that need to be specified when writing the task definition, see Task definition parameters by AWS:

If a task-level memory value is not specified, you must specify a non-zero integer for one or both of memory or memoryReservation in a container definition.

Specifying X amount of memoryReservation tells ECS that this particular Docker container might need X amount of memory in MiB. It is a soft limit, which means that the container is allowed to use more than we reserved. Also from the same reference:

When system memory is under contention, Docker attempts to keep the container memory to this soft limit; however, your container can consume more memory when needed, up to either the hard limit specified with the memory parameter (if applicable), or all of the available memory on the container instance, whichever comes first.

The Docker daemon reserves a minimum of 4 MiB of memory for a container, so you should not specify fewer than 4 MiB of memory for your containers.

Reading the above, some might get the idea that why not be flexible and just set memoryReservation for all containers to 4 MiB then let ECS figures out how many memory that the container needs. In this article, we want to help the readers to understand why this is not a good idea. Then also try to help the readers to choose a reasonable value for memoryReservation.

Experimental Setup

For demonstration purposes, we are going to setup a very simple playground in ECS. It is an EC2 launch type ECS cluster with a single ECS instance of type t2.micro that has 1 GiB of memory. In the cluster, we will create an ECS service that consists of following task definition:



[
  {
    "name": "worker",
    "image": "alexeiled/stress-ng:latest",
    "command": [
      "--vm-bytes",
      "300m",
      "--vm-keep",
      "--vm",
      "1",
      "-t",
      "1d",
      "-l",
      "0"
    ],
    "memoryReservation": 400
  }
]

Essentially, this task definition will launch a container that is constantly consuming 300 MB / 1.049 = 286.102 MiB of memory (see stress-ng). We can think of this as a memory hungry worker. The Terraform configuration of the set up is available in GitHub:

shihanng / ecs-resource-exp

ecs-resource-exp

Terraform configuration to setup simple Amazon Elastic Container Service (ECS) cluster for demonstrating memory usage.

cd tf
terraform plan -out tfplan
terraform apply tfplan

View on GitHub

The Experiments

The following shows the metrics that we collected during our tests. We tested two different ECS configurations. The first half shows what happened when the memory is properly reserved. The second half was where a very small value was used as memoryReservation. We will walkthrough the whole process from (A) to (F) in the following text.

First part: The right way

Since we know that our container will consume around 286 MiB of memory, the memoryReservation is set to 400 MiB to begin the test.
Let's start the test by placing one task into the ECS cluster. We accomplish that by setting the desired count to 1.
From our CloudWatch metrics (see (A)), we can see that after placing one task into the cluster, we essentially reserving 41% and using 30% of the memory from the cluster in average. Service memory utilization is at 75.25% means that we are using 75.25% or the memory that we reserved for the service. The formulas for how these values are derived is available in the Amazon ECS CloudWatch metrics documentation.

The metrics also show that increasing the number of desired count from 1 to 2 (see (B)) worked as expected: Both memory reservation and utilization average percentage doubled. Service memory utilization stays at 75.25% because we are still using 75.25% or the memory that we reserved for the service even though the usage of the memory in MiB is doubled.

Finally we want to try to "break" the system by increasing the desired task to 3 in (C). Adding the third task means that we need to reserve 3 x 400 MiB = 1200 MiB for our workers.

Since this is more than what the cluster could support, ECS will tell us the following:

service ecs-memory-exp-default-worker was unable to place a task because no container instance met all of its requirements. The closest matching container-instance xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx has insufficient memory available. For more information, see the Troubleshooting section.

instead of forcefully placing the third task. Note that we observed no fluctuation in our CloudWatch metrics in (C).
This is a good fail-safe. Failing to place the third task does not cause any disruption to the running tasks. What we need to do now is to add more resources into the cluster for the additional tasks that we want to run.

Second part: What could go wrong?

Now, let's repeat the same experiment with one different setting: only 100 MiB of memoryReservation. Here, we set the desired count to 1 at (D) and then to 2 at (E).
We can observe in the metrics that cluster memory utilization is always higher than what we've reserved. Service memory utilization is around 301% that means we are using three times more than what we promised to use.

Let's try to put the third task into the cluster by setting the desired task to 3. Since ECS thought that the task only need 100 MiB, it will try to run the third task. However, this will cause one of the three tasks to stop due to insufficient memory, then ECS will again rerun the task to fulfill the desired count condition. As we can see from the events log and task statuses, and the fluctuation in the metrics (F), there is an endless loop of running and stopping of ECS task. The fluctuation is a sign of an unstable system.

Note that although it may seem that we still have enough memory (there was still 39% leftover) for another task, we also need to consider the memory that is required by the host system (see Container Instance Memory Management).

We also need to consider the problem where we have no direct control of which task/container to kill when there is memory insufficient issue (see How Amazon ECS manages CPU and memory resources by AWS):

If containers try to consume memory between these two values (or between the soft limit and the host capacity if a hard limit is not set), they may compete with each other. In this case, what happens depends on the heuristics used by the Linux kernel’s OOM (Out of Memory) killer.

What if the Linux kernel's OOM killer decides to terminate a worker that is currently processing huge data rather than the newly spun-up worker? In a radical case, this means that it is possible that we could not process anything even though it may seem that we still have two workers running, they might be killed before completed their works.

Wrap-up

Yes, ECS allows us to set a very small value for memoryReservation, but that does not mean it is a good idea to do it. We might find a “clever” workaround in such a small and simple system as one demonstrated above. However, production systems are usually more complex with multiple task definitions and container instances. Therefore, it is more difficult to spot the memory issue and to find the root cause of the endless restart in the real world. We can avoid all of these issues by using proper values set as memoryReservation. Docker also recommends that we correctly limit the memory usage of a container to mitigate the risk of system instability caused by Out Of Memory Exception (OOME).

So if not a minimal value, how should we set the memoryReservation? AWS recommends that:

... if your container normally uses 128 MiB of memory, but occasionally bursts to 256 MiB of memory for short periods of time, you can set a memoryReservation of 128 MiB, and a memory hard limit of 300 MiB. This configuration would allow the container to only reserve 128 MiB of memory from the remaining resources on the container instance, but also allow the container to consume more memory resources when needed.

In the real world, it is not always that straight forward to know how much memory our application needs in contrast to the example we have above. Service memory utilization can be a metric to help us figure out how much memory our container normally uses. When its average is close to or exceeds 100%, it shows that our application is using more than what we expected, and we should increase the reservation or the resource accordingly.

Thanks for reading. I hope that you find this article useful ❤️.

tfvar - A tool to help you write Terraform's variable definitions

Shi Han — Tue, 18 Feb 2020 12:16:18 +0000

Whenever I want to start working on an existing Terraform project or using a Terraform module, the first thing that I find myself doing often is looking for the input variables. If the project is small enough, the input variables are usually being declared in the main.tf file. For a well organized Terraform configurations we can find a variables.tf file. However, the locations or where to put the input variable declarations is just a convention and is not enforced syntactically. It is our job as the user of the configurations/modules to track down all variables and make sure that they have the values assigned properly before running terraform plan or terraform apply. It is possible that some variables are hiding somewhere unexpectedly, or we just missed one during the assignment and only realize it when we encounter the following message during the terraform plan phase:

var.bucket_name
  Enter a value:

tfvar is created to solve the problems mentioned above. It is a CLI tool that extracts all variables declared in your Terraform configuration and output them in the format of .tfvars file so that you can use it as a template to write the variable definitions.

In this article, I would like to show you how this tool works.

Simple Demo

Let's say we have the following variable declared in our Terraform configuration

variable bucket_name {
  type    = string
  default = "mybucket"
}

variable instance_name {
  type = string
}

running tfvar . in the root directory of the Terraform configuration will give us the following in the standard output (stdout):

$ tfvar .
bucket_name   = "mybucket"
instance_name = null

Notice that the default value are automatically assigned in the generated definitions. If we want to ignore the default values, there is a --ignore-default flag that we can use. As shown below the with the --ignore-default flag, we get null instead:

$ tfvar . --ignore-default
bucket_name   = null
instance_name = null

A more common usage of tfvar might be to redirect the stdout to a file e.g. inputs.tfvars then change the value null in the file to a desired value.

$ tfvar . --ignore-default > inputs.tfvars

Terraform also allows input variables to be assigned via environment variables. For this, tfvar also provides the -e flag to generate a template in the environment variables format.

$ tfvar . --ignore-default -e
export TF_VAR_bucket_name=''
export TF_VAR_instance_name=''

If we already have some variables defined in terraform.tfvars[.json], *.auto.tfvars[.json], or environment variables (TF_VAR_ followed by the name of a declared variable), (see section "Assigning Values to Root Module Variables" in Terraform's documentation) and want to include there defined values in the generated one, we can use the --auto-assign flag:

$ export TF_VAR_bucket_name="mybucket"
$ export TF_VAR_instance_name="myinstance"
$ tfvar . --auto-assign
bucket_name   = "mybucket"
instance_name = "myinstance"

Finally for completeness sake, we also provide --var and --var-file flags to allow individual variables to be set, just like the -var and -var-file that we have in the terraform (plan|apply) commands. However, I honestly could not think of any use case that I would need this in my workflow 😆.

All --auto-assign, --var, and --var-file flags can be used together, and when they are, they follow the variable definition precedence set by Terraform.

How to get it?

The source codes of tfvar is publicly available on GitHub. In the repository below you will find the installation instructions.

shihanng / tfvar

Terraform's variable definitions template generator.

`tfvar`

tfvar is a Terraform's variable definitions template generator.

For Terraform configuration that has input variables declared, e.g.,

variable "image_id" {
  type = string
}
variable "availability_zone_names" {
  type    = list(string)
  default = ["us-west-1a"]
}
variable "docker_ports" {
  type = list(object({
    internal = number
    external = number
    protocol = string
  }))
  default = [
    {
      internal = 8300
      external = 8300
      protocol = "tcp"
    }
  ]
}

tfvar will search for all input variables and generate template that helps user populates those variables easily

$ tfvar
availability_zone_names = ["us-west-1a"]
docker_ports            = [{ external = 8300, internal = 8300, protocol = "tcp" }]
image_id                = null

Note that default values are assigned to the definitions by default as shown above. Use the --ignore-default options to ignore the default values
```
$ tfvar
```
…

View on GitHub

tfvar is a tool to solve a trivial problem that would only cost me a few minutes. However, I decided to spend hours to build it hoping that it will save a few extra minutes off from my fellow Terraformers. Some of the codes/ideas are borrowed from Terraform itself which I enjoy exploring a lot. Please give it a try and I would love hear your feedback (via issues/PRs in project repository or comments in this post). If you like it, give the repository a ⭐ which will help other people discover it.

Thanks for reading ❤️.

Improve your PostgreSQL Workflows with vim-tmux-runner

Shi Han — Fri, 17 Jan 2020 12:53:35 +0000

There are many client applications for working with databases (PostgreSQL, etc.) available in the market. Some of my colleagues use JetBrains's DataGrip, Postico, DBeaver, etc. While I do not work much directly with DBs on daily basis, I do sometimes find myself experimenting on local PostgreSQL instance via pgcli before "transferring" the queries into application code. During this process, one feature that I would really like to have from those GUI clients is the ability
execute selected/highlighted commands directly from the editor and get the result directly on screen. I use Vim as my main editor. So it would be nice if I can perform the same thing describe above
within Vim.

It turns out this can be achieve with combination of Vim and tmux and the vim-tmux-runner plugin. I've included a small demo in this article to show how that can be done with the tools mentioned before.

christoomey / vim-tmux-runner

Vim and tmux, sittin' in a tree...

VTR [Vim Tmux Runner]

A simple, vimscript only, command runner for sending commands from vim to tmux.

Usage

VTR provides a handful of commands for managing and interacting with tmux the terminal multiplexer. The main command is:

VtrSendCommandToRunner

This command will prompt for a command to run, then send it to the runner pane for execution. Subsequent calls to VtrSendCommandToRunner will reuse the provided command.

If you would like VTR to create a runner pane if one doesn't exist while issuing a command, a bang version can be used: VtrSendCommandToRunner!.

VTR provides configuration options that allow for control over the size and location of the VTR runner pane. In addition, VTR provides commands to resize, reorient, and even detach the runner pane making the interaction as painless as possible.

For a complete summary of the available commands and configuration options in VTR, check the included doc file.

…

View on GitHub

DB Preparation

For demo purposes we are going to setup a sample PostgreSQL database in a Docker container. If you already have a DB to play with, feel free to skip this part.

docker run -d --name postgres_demo -p 5432:5432 --rm postgres:11.6
echo "CREATE DATABASE pagila;" | psql -h localhost -p 5432 -U postgres
curl -s https://raw.githubusercontent.com/devrimgunduz/pagila/68b7bab066a18988694a8b698533c3c507f7b133/pagila-schema.sql | psql -h localhost -p 5432 -U postgres pagila -f -
curl -s https://raw.githubusercontent.com/devrimgunduz/pagila/68b7bab066a18988694a8b698533c3c507f7b133/pagila-data.sql | psql -h localhost -p 5432 -U postgres pagila -f -

Vim configurations

First install the christoomey/vim-tmux-runner plugin with your favorite Vim plugin manager. Then, for my setup, I've included the following in my vimrc:

let g:VtrUseVtrMaps = 1
let g:VtrStripLeadingWhitespace = 1
let g:VtrClearEmptyLines = 1

DEMO

This workflow requires a tmux session, so go ahead and start up one.
Then we launch Vim within that tmux session.
In Vim, create a new tmux pane (referred as "runner" in the plugin) using the :VtrOpenRunner command.
Run pgcli in the newly created pane and connect to the DB (pagila) created above.
```
pgcli -h localhost -p 5432 -u postgres pagila
```
Turn on Multi-line Mode in pgcli by pressing the <F3> key. This means that our PostgreSQL commands in Vim need to end with semi-colon ;.

Once we have pgcli running on the runner pane, we can write down our SQL query in Vim then select the query in Visual mode.

SELECT film_id, title FROM film LIMIT 5;

Finally use the binding <Leader>sl to send the selected SQL commands to the runner. The binding (enable via g:VtrUseVtrMaps mentioned above) executes command :VtrSendLinesToRunner.

Often, we might be in a state where we already have pgcli opened but not attached to Vim as the runner. In this situation, if we do :VtrSendLinesToRunner, we will get VTR: No runner pane attached. error. For this case, we can use :VtrAttachToPane command to attach the pane as runner then continue with our workflow.

This concludes the article about me indulging myself with another "How to do X in Vim?" Of course, usage of vim-tmux-runner is not limited to PostgreSQL but you can do the same on other REPL shell e.g. IRB, IPython, etc. Hope this article gives you some ideas of how to integrate this plugin into your workflow. Thanks for reading ❤️.

Cleanup 🗑️

If you want to clean up the demo DB created above, simply do:

docker stop postgres_demo

gig: a gitignore generator

Shi Han — Sun, 15 Dec 2019 10:24:48 +0000

If you are using git in your project, you probably know what is .gitignore. Since I never seem to be able to memorized gitignore's pattern format correctly, when I needed to write a gitignore file for a specific project, I'll visit github/gitignore in my web browser, search for gitignore file, open the file, copy its content, then finally paste it into my editor...

As we can imagine, if you need to create gitignore file for multiple programming languages the process becomes very tedious.

A while back I was introduced to gitignore.io which is a web service that allows us to create gitignore file easily for multiple languages. gitignore.io also provides API access and very nice documentation on how to run gitignore.io from your command line. It was a huge upgrade and I've been using this tool since then. I even submitted a PR to gitignore direnv:

Add direnv.gitignore #241

shihanng posted on Oct 03, 2019

Based on https://github.com/direnv/direnv/wiki/Git

Pull Request

Thank you for contributing to @dvcs/gitignore and https://www.gitignore.io.

New or update

Select the appropriate check box for this pull request. This helps when merging to ensure there are no conflicts with other templates or misunderstandings of how thee template list works.

New

[x] Template - New .gitignore template
[ ] Composition - Template made from smaller templates
[ ] Inheritance - Template similar to an existing template
[ ] Patch - Template extending functionality of existing template

Update

[ ] Template - Update existing .gitignore template

Details

direnv is a tool that can load and unload environment variables based on the current working directory. It is commonly advised to not to include its configuration file .envrc into a public repository therefore having this in the gitignore will be very handy. The values are based on this documentation: https://github.com/direnv/direnv/wiki/Git

Thank you.

View on GitHub

However, after the PR got merged, I found out that the change is not reflected in gitignore.io (yet). My guess is that the maintainer needs to "redeploy" the site in order to include the latest collections.



$ curl https://gitignore.io/api/direnv

# Created by https://www.gitignore.io/api/direnv
# Edit at https://www.gitignore.io/?templates=direnv

#!! ERROR: direnv is undefined. Use list command to see defined gitignore types !!#

# End of https://www.gitignore.io/api/direnv

gitignore.io's collections of gitignore templates are maintained in a separated repository: github.com/toptal/gitignore. They are extensions of github/gitignore that include additional types known as .patch, .stack. When I bumped into the error above, I wondered if I can somehow generate the gitignore file using the repository directly without the web service. Which then lead me deep into the rabbit hole and created gig...

shihanng / gig

Generate .gitignore files from your terminal (mostly) offline!

`gig` -- .gitignore generator

gig is a command line tool to help you create useful .gitignore files for your project It is inspired by gitignore.io and make use of the large collection of useful .gitignore templates of the web service This also means that gig supports the are four file types that gitignore.io recognizes. Content generated by gig should match the one generated by gitignore.io except the order of stacks in which gitignore.io does not seem to guarantee any.

Motivation

Prior to this project, I used to have one of these command lines in my .zshrc. However, problems I have with this command line are that

I won't be able to use it without internet access
It seems gitignore.io that is not using the latest templates in https://github.com/toptal/gitignore. The project needs to update to the latest git submodule then deploy it.

Therefore this tool is created to solve…

View on GitHub

Introducing gig

gig is a CLI port of gitignore.io and make use of the large collection of .gitignore templates of gitignore.io mentioned above. This also means that gig supports all four file types that gitignore.io recognizes and generates contents of gitignore like gitignore.io does.

Installing

If you are on macOS and are using Homebrew,



brew install shihanng/gig/gig

For other platforms, download the pre-built binaries from the release page then place the binary in the $PATH e.g. /usr/local/bin.

Using

The most direct way of using gig is to use the gen subcommand with a list of template names as arguments, e.g.



gig gen go terraform > .gitignore

Example above generates gitignore templates for Go and Terraform then redirects those into .gitignore file. With additional -f flag, we can omit the redirect part of the command. gig command will cache the content of github.com/toptal/gitignore repository into $XDG_CACHE_HOME/gig during the first run. This means that the next execute should work offline.

When we want to look for available templates, we can use the list command. It's a pretty long list, so it works better with a pager:



gig list | less

There is also a search command which calls fzf -m internally on the list above and allows user to select (by pressing tab by default) the templates in fzf.

Finally there is the least useful autogen subcommand which was added just for fun.



gig autogen

It will look into the current directory and try to figure out what to gitignore. Internally it uses source{d}'s enry package for programming language detection (which itself is a port of GitHub's linguist). Unfortunately it does not always work as expected:

It is not able to detect framework, e.g. .js files will be detected as javascript but there is not JavaScript gitignore template available.
go.mod and go.sum files will be detected as text for the reason mentioned in this PR.

gig started as a fun project to see how far I can go to build a CLI tool with the remix of

github.com/spf13/cobra: Go's framework for CLI,
github.com/src-d/go-git: Git implementation in Go,
github.com/src-d/enry: for programming language detection,
and many other libraries...

Now it is has become a tool that I will use it when starting a new project.

Please give it a try and I would love hear your feedback (via issues/PRs in project repository or comments in this post). If you like it, give it a ⭐ so that other people will hear about it.

Thanks for reading ❤️

Managing Go Versions with direnv

Shi Han — Wed, 14 Aug 2019 13:52:46 +0000

Recently I've joined a project that requires previous version of Go. Since I have the latest version already installed on my machine I was looking for a way to manage different versions of Go in a single machine. In Python world, there is a very well-known version management called pyenv which is highly inspired by the Ruby's one: rbenv.

As for Go, it seems that there is no authoritative way to manage the versions of Go. Probably it is not a big issue as many would just upgrade to the latest version of the language since it essentially promises that new minor version of 1.X would not break the older ones. Yet, through a very brief Googling, I found that there are three main approaches for this problem:

Installing each version of Go from source. This article by Kale Blankenship describe the steps in details.
There is also a tool that is similar to pyenv and rbenv available in the Go ecosystem known as gvm It also comes with additional features such as managing different locations of GOPATH.
Lastly, we can also use go get. Yes! GO GET!

Installing a Different Version of Go

In this post, we will explore how to use go get to manage different versions Go. One small caveat for this approach is that not all Go versions are support. Supported versions are listed on the download page. Let's say that we need go1.8.7, all we have to do is:

$ go get golang.org/dl/go1.8.7
$ go1.8.7 download

Once the download is complete, we can compile our source code with v1.8.7 using

$ go1.8.7 version
go version go1.8.7 darwin/amd64
$ go1.8.7 build

Uninstalling can be done by removing the GOROOT directory:

$ rm -rf "$(go1.8.7 env GOROOT)"

Using `goX.Y.Z` as `go`

Often in a project we have some kind of build scripts of Makefiles which will call go instead of go1.8.7 to build our Go binary. Therefore it would be very convenient if we can use the executable go1.8.7 as go for a specific directory/project. This is where direnv comes into play.

direnv is an environment switcher for the shell. It knows how to hook into
bash, zsh, tcsh, fish shell and elvish to load or unload environment
variables depending on the current directory

For demostration purposes, we will create a new project in the GOPATH as following:

$ export GOPATH="$(go1.8.7 env GOPATH)"
$ mkdir -p "${GOPATH}/src/github.com/user/hello"
$ cd "${GOPATH}/src/github.com/user/hello"
$ cat >main.go <<EOL
package main

import "fmt"

func main() {
    fmt.Println("Hello, world.")
}
EOL

In order to enable direnv in our project directory, we need to setup the .envrc file. This can be done by doing

$ direnv edit .

inside the root directory of our project. The content of .envrc file should be:

export GOROOT="$(go1.8.7 env GOROOT)"
PATH_add "$(go1.8.7 env GOROOT)/bin"

In the first line, we point GOROOT to the version that we want to work with. Then we add the Go 1.8.7 compiler into PATH so that it will be found before the default one. Once the setup is done, running go version will give us the older version.

$ go version
go version go1.8.7 linux/amd64

We can then build the main.go using

$ go build
$ gdb ./hello
...
(gdb) p 'runtime.buildVersion'
$1 = 0x4a5208 "go1.8.7"

The above also shows how to check the which version of Go is used to build the hello binary as described in this post.

That's all for the demo. Hope you find this post useful. Personally I think that this approach has a good balance between simplicity and usability. One other thing that I often find useful in tools like pyenv is the pyenv install --list subcommand which shows the available versions that can be installed with pyenv. Of course the approach above does not have this feature. One hack we can use to list all available versions on terminal is by using the pup command with curl:

$ curl -s https://godoc.org/golang.org/dl | pup 'body > div.container > table > tbody > tr > td:nth-child(1) > a text{}'

go1.10
go1.10.1
go1.10.2
go1.10.3
go1.10.4
...

It's not pretty but it does the trick 💪.

Found a typo?

Thank you for reading! Found a typo? See something that could be improved or anything else that should be updated on this blog post? Thanks to this project, you can easily create a pull request on https://github.com/shihanng/dev.to to propose a fix!

Looking into PostgreSQL DB's tables of your Development Environment

Shi Han — Mon, 15 Jul 2019 07:46:56 +0000

You just joined an established project and have been given access to the source code. Sometimes we find that it is easier to understand the structure of the project by looking into the database tables and the relations between them. Recently I found myself in this situation with a project that can be built locally with Docker Compose. This is a note about what I did to glimpse in the project's tables.

Setup

In order to allow readers to follow along for a more hands-on experience, we are going to use the Dockerize Rails app created by Georg Ledermann as the "established project." The project already has a docker-compose.yml setup that connects a PostgreSQL database to a Ruby on Rails application.

ledermann / docker-rails

Dockerize Rails 7 with ActionCable, Webpacker, Stimulus, Elasticsearch, Sidekiq

Docker-Rails

Simple Rails 7.0 application to demonstrate using Docker for production deployment. The application is a very simple kind of CMS (content management system) allowing to manage posts. Beside the boring CRUD functionality it has some non-default features.

This project aims to build a lean Docker image for use in production. Therefore it's based on the official Alpine Ruby image, uses multi-stage building and some optimizations that I described in my blog. This results in an image size of ~80MB.

Features

Auto refresh via ActionCable: If a displayed post gets changed by another user/instance, it refreshes automatically using the publish/subscribe pattern
Full text search via OpenSearch and the Searchkick gem to find post content (with suggestions)
Autocompletion with autocompleter
Editing HTML content with the WYSIWYG JavaScript editor Trix
Uploading images directly to S3 with the Shrine gem and jQuery-File-Upload
Background jobs with ActiveJob and the Sidekiq gem (to…

View on GitHub

Let's setup the project locally by referring to the Check it out! section of the repository:

$ git clone https://github.com/ledermann/docker-rails.git
$ cd docker-rails
$ docker-compose build
$ docker-compose run app yarn install
$ docker-compose up -d

At this point, you should have a few services running which you can check with the docker-compose ps command:

$ docker-compose ps
            Name                          Command               State            Ports
-----------------------------------------------------------------------------------------------
docker-rails_app_1             docker/startup.sh                Up      0.0.0.0:32774->3000/tcp
docker-rails_db_1              docker-entrypoint.sh postgres    Up      5432/tcp
docker-rails_elasticsearch_1   /usr/local/bin/docker-entr ...   Up      9200/tcp, 9300/tcp
docker-rails_redis_1           docker-entrypoint.sh redis ...   Up      6379/tcp
docker-rails_worker_1          bundle exec sidekiq              Up      3000/tcp

If you want to checkout the web application that is running on your host, visit http://localhost:32774. Note that the value 32774 comes from the Ports column of docker-rails_app_1 in the output of the docker-compose ps: container's port 3000 is exposed on the host's port 32774.

Accessing the Database

Here we will show how to access to the development database using two different tools:

pgcli,
SchemaSpy for a more visualized approach.

Before doing so, we need to know the credentials for the database which can be found on the docker-compose.yml. The database that we are interested in is docker-rails_development. Let's export these value as environment variables:

$ export DB_USER=postgres
$ export DB_PASSWORD=foobar123
$ export DB_NAME=docker-rails_development

pgcli

Install pgcli based on the instructions in official documentation. Based on the information below we want to connect to the database (docker-rails_db_1) on port 5432.

$ docker-compose ps db
      Name                     Command              State    Ports
--------------------------------------------------------------------
docker-rails_db_1   docker-entrypoint.sh postgres   Up      5432/tcp

However the port is not exposed to the host. Therefore, doing the following will fail

$ pgcli -h localhost -p 5432 -U "${DB_USER}" -d "${DB_NAME}"
could not connect to server: Connection refused
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 5432?
could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?

In this situation, we can use the IP address of the container instance to access the container without exposing the port to the host machine. The IP address can be obtained via

$ docker inspect docker-rails_db_1 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.24.0.4",

Using the IP address above, we can access the database via

$ pgcli -h 172.24.0.4 -p 5432 -U "${DB_USER}" -d "${DB_NAME}"
Server: PostgreSQL 11.4
Version: 2.1.1
Chat: https://gitter.im/dbcli/pgcli
Mail: https://groups.google.com/forum/#!forum/pgcli
Home: http://pgcli.com
docker-rails_development>

SchemaSpy

For SchemaSpy, we will try will a different approach: Using the SchemaSpy's Docker container to connect to the existing database and generate the database diagrams for us. For this we need to know the name of the network created by docker-compose for this project:

$ docker network list | grep docker-rails
27a3a7b42168        docker-rails_default   bridge              local

Now that we know the network name, we can use the --net option in the docker run command to connect the SchemaSpy container to the existing project network. The following will generate the database diagrams in the db_out/ directory.

$ export DB_HOST=docker-rails_db_1
$ export DB_NET=docker-rails_default
$ mkdir -p db_out
$ docker run --rm --net "${DB_NET}" \
  -v `pwd`/db_out:/output schemaspy/schemaspy:6.0.0 \
  -t pgsql \
  -db "${DB_NAME}" \
  -host "${DB_HOST}" \
  -port 5432 \
  -u "${DB_USER}" \
  -p "${DB_PASSWORD}"

Open db_out/index.html with a browser to checkout the generated diagrams.

Cleanup

That's all for this post. The tricks mentioned above might seem limited to PostgreSQL but with little modification we believe that it can be applied on other types of databases too. As for cleanup, since it is a Docker Compose project, cleanup is as simple as:

$ docker-compose stop
$ docker-compose rm

DEV Community: Shi Han

Managing S3 bucket for Terraform backend in the same configuration

What is Terraform backend?

The chicken-and-egg problem

The solution

How to migrate from the S3 backend to the local backend?

Final remarks

Hosting personal website on DigitalOcean's App Platform

What I built

Category Submission:

App Link

Screenshots

Description

Link to Source Code

Permissive License

Background

How I built it (A detailed guide of deploying a Gatsby site to DigitalOcean’s App Platform)

Publishing DEV articles from GitHub

Workflow with GitHub Actions

devto: the CLI Tool

shihanng / devto

CLI tool to publish article to https://dev.to/

devto -- publish to dev.to from your terminal

What is this?

devto-act: the GitHub Action

shihanng / devto-act

devto's GitHub Action

Inputs

devto_api_key

skip_generate

published

dry_run

auto_prefix

markdown_files

Example Usage

My Workflow

Submission Category:

Ending

How to read CSV file from Amazon S3 in Python

Downloading File from S3

Reading CSV File

Reading CSV file from S3

Memory Reservation in Amazon Elastic Container Service

Experimental Setup

shihanng / ecs-resource-exp

ecs-resource-exp

The Experiments

First part: The right way

Second part: What could go wrong?

Wrap-up

tfvar - A tool to help you write Terraform's variable definitions

Simple Demo

How to get it?

shihanng / tfvar

Terraform's variable definitions template generator.

tfvar

Improve your PostgreSQL Workflows with vim-tmux-runner

christoomey / vim-tmux-runner

Vim and tmux, sittin' in a tree...

VTR [Vim Tmux Runner]

Usage

DB Preparation

Vim configurations

DEMO

Cleanup 🗑️

gig: a gitignore generator

Add direnv.gitignore #241

Pull Request

New or update

New

Update

Details

shihanng / gig

Generate .gitignore files from your terminal (mostly) offline!

gig -- .gitignore generator

Motivation

Introducing gig

Installing

Using

Managing Go Versions with direnv

`devto`: the CLI Tool

`devto-act`: the GitHub Action

`devto_api_key`

`skip_generate`

`published`

`dry_run`

`auto_prefix`

`markdown_files`

`tfvar`

`gig` -- .gitignore generator

Using `goX.Y.Z` as `go`