DEV Community: Rene Hernandez

Introducing appfile: a declarative way of managing apps in DigitalOcean App Platform

Rene Hernandez — Wed, 25 Nov 2020 00:00:00 +0000

I have been experimenting with DigitalOcean App Platform for a while and I like how it helps me focus on defining only what I need to run my apps. Using the app.yaml spec, I can declare the app components and store it within the project codebase. Soon though, I started to run into the problem of how to manage different environments for the same application (e.g. review, staging and production).

After unsuccessfully searching online for anything that would fit my use case, I figured I would solve the problem myself. I wanted a tool that would allow me:

Declare the different environments for a given App specification
Have diff capabilities
Deploy multiple apps at once

After a couple of days of tinkering, I had an up and running the first version of appfile. If you want to go straight to the code, check the repo at renehernandez/appfile.

Ready? Ok, let's discuss what appfile is all about.

Features #

The main capabilities that I set out to have and are implemented as of the current version (v0.0.2) are outlined below:

Declare the different environments for a given App specification
Support templates to customize the final app specification based on the selected environment
Have diff capabilities
Deploy multiple apps at once

CLI #

The full CLI help can be seen by either typing on the terminal:

$ appfile

Or:

$ appfile --help

It will output help information like:

$ appfile

Deploy app platform specifications to DigitalOcean

Usage: 
  appfile [command]

Available Commands: 
  destroy Destroy apps running in DigitalOcean 
  diff Diff local app spec against app spec running in DigitalOcean 
  help Help about any command 
  sync Sync all resources from app platform specs to DigitalOcean

Flags: 
  -t, --access-token string API V2 access token
  -e, --environment string root all resources from spec file (default "default")
  -f, --file string load appfile spec from file (default "appfile.yaml")
  -h, --help help for appfile
  --log-level string Set log level (default "info")
  -v, --version version for appfile

Use "appfile [command] --help" for more information about a command.

The available sub-commands are:

appfile sync: Sync all resources from app platform specs to DigitalOcean
appfile diff: Diff local app spec against app spec running in DigitalOcean
appfile destroy: Destroy apps running in DigitalOcean

Github Action #

There is also a Github Action that you can use to automate the deployment of Apps to DigitalOcean with appfile. Check the action-appfile Action at:

Marketplace: https://github.com/marketplace/actions/github-action-for-appfile-cli
Repository URL: https://github.com/renehernandez/action-appfile

Installation #

Currently, you would need to install appfile by downloading a corresponding release from the latest Github release for your platform of choice.

For Mac:

$ wget https://github.com/renehernandez/appfile/releases/latest/download/appfile_darwin_amd64
$ chmod +x appfile_darwin_amd64
$ mv ./appfile_darwin_amd64 /usr/local/bin/appfile

For Linux:

$ wget https://github.com/renehernandez/appfile/releases/latest/download/appfile_linux_amd64
$ chmod +x appfile_darwin_amd64
$ mv ./appfile_darwin_amd64 /usr/local/bin/appfile

For Windows:

> Invoke-WebRequest -Uri "https://github.com/renehernandez/appfile/releases/latest/download/appfile_windows_amd64.exe" -OutFile appfile.exe
> $env:Path += "./appfile.exe"

Usage #

Let's look at the following example to start seeing the power of appfile. We want to deploy a Rails application to the DigitalOcean App Platform. This Rails app would have different components depending if we are deploying to production or a review environment.

To start, we need to define our appfile.yaml spec:

# appfile.yaml
environments: 
  review:
  - ./envs/review.yaml
  production:
  - ./envs/production.yaml

specs:
- ./app.yaml

The above spec lays out that our App has 2 environments to get state values from: review and production from the ./envs/review.yaml and ./envs/production.yaml files respectively. It also defines that the App spec is located at ./app.yaml

Let's take a look a the app.yaml definition:

# app.yaml
name: {{ .Values.name }}

services:
- name: rails-app 
  image: 
    registry_type: DOCR 
    repository: <repo_name> 
    tag: {{ requiredEnv "IMAGE_TAG" }}
  instance_size_slug: {{ .Values.rails.instance_slug }}
  instance_count: {{ .Values.rails.instance_count }}
  envs: 
{{- range $key, $value := .Values.rails.envs }}
  - key: {{ $key }}
    value: {{ $value }}
{{- end }} 

{{- if eq .Environment.Name "review" }}
- name: postgres
  image:
    registry_type: DOCR
    repository: postgres
    tag: '12.4'
  internal_ports:
    - 5432
  envs:
{{- range $key, $value := .Values.postgres.envs }}
  - key: {{ $key }}
    value: {{ $value }}
{{- end }}
{{- end }}

jobs:
- name: migrations
  image:
    registry_type: DOCR
    repository: <repo_name>
    tag: {{ requiredEnv "IMAGE_TAG" }}
  envs:
{{- range $key, $value := .Values.migrations.envs }}
  - key: {{ $key }}
    value: {{ $value }}
{{- end }}

{{- if eq .Environment.Name "production" }}
databases:
- name: db
  production: true
  cluster_name: mydatabase
  engine: PG
  version: "12"
{{- end }}

As you can see, the app.yaml is leveraging templates to abstract the values that can change (e.g. tag: {{ requiredEnv "IMAGE_TAG" }}), as well as, determining which components need to be deployed based on the environment (e.g. the usage of a postgres container in review environments vs the usage of a managed database in production).

Next, we define the values for each of the environments that are going to be merged with the app.yaml to produce the final app specification. First, the values definition for the review environment:

# review.yaml
name: sample-{{ requiredEnv "REVIEW_HOSTNAME" }}

.common_envs: &common_envs
  DB_USERNAME: postgres
  DB_PASSWORD: password
  RAILS_ENV: production

rails:
  instance_slug: basic-xxs
  instance_count: 1
  envs:
  <<: *common_envs

postgres:
  envs:
    POSTGRES_USER: postgres
    POSTGRES_DB: mydatabase
    POSTGRES_PASSWORD: password

migrations:
  envs:
  <<: *common_envs`

And second, the values definition for the production environment:

# production.yaml
name: sample-production

.common_envs: &common_envs
  DB_USERNAME: postgres
  DB_PASSWORD: strong_password
  RAILS_ENV: production

rails:
  instance_slug: professional-xs
  instance_count: 3
  envs:
  <<: *common_envs

migrations:
  envs:
  <<: *common_envs

With all the required files in place, we can now proceed to deploy our app to DigitalOcean

As a review environment:

$ IMAGE_TAG='fad7869fdaldabh23' REVIEW_HOSTNAME='fix-bug' appfile sync --file /path/to/appfile.yaml --environment review

This would deploy a public Rails service, and internal Postgres service (the database running on a container) and would run the migration job. The final App spec to be synced to DigitalOcean would look like:

# final app specification with review environment values
name: sample-fix-bug

services:
- name: rails-app
  image:
    registry_type: DOCR
    repository: <app-repo>
    tag: fad7869fdaldabh23
  instance_size_slug: basic-xxs
  instance_count: 1
  routes:
  - path: /
  envs:
  - key: DB_PASSWORD
    value: password
  - key: DB_USERNAME
    value: postgres
  - key: RAILS_ENV
    value: production

- name: postgres
  image:
    registry_type: DOCR
    repository: postgres
    tag: '12.4'
  internal_ports:
    - 5432
  envs:
  - key: POSTGRES_DB
    value: mydatabase
  - key: POSTGRES_PASSWORD
    value: password
  - key: POSTGRES_USER
    value: postgres

jobs:
- name: migrations
  image:
    registry_type: DOCR
    repository: <migration-repo>
    tag: fad7869fdaldabh23
  envs:
  - key: DB_PASSWORD
    value: password
  - key: DB_USERNAME
    value: postgres
  - key: RAILS_ENV
    value: production

As a production deployment:

$ IMAGE_TAG='fad7869fdaldabh23' appfile sync --file /path/to/appfile.yaml --environment production

This would deploy a public Rails service and a migration job. Both components would connect to an existing database. The final App spec to be synced to DigitalOcean would look like:

# final app specification with production environment values
name: sample-production

services:
- name: rails-app
  image:
    registry_type: DOCR
    repository: <app-repo>
    tag: fad7869fdaldabh23
  instance_size_slug: professional-xs
  instance_count: 3
  routes:
  - path: /
  envs:
  - key: DB_PASSWORD
    value: strong_password
  - key: DB_USERNAME
    value: postgres
  - key: RAILS_ENV
    value: production

jobs:
- name: migrations
  image:
    registry_type: DOCR
    repository: <migration-repo>
    tag: fad7869fdaldabh23
  envs:
  - key: DB_PASSWORD
    value: strong_password
  - key: DB_USERNAME
    value: postgres
  - key: RAILS_ENV
    value: production

databases:
- name: db
  production: true
  cluster_name: mydb
  engine: PG
  version: "12"

Future steps #

There are several areas where the tool could move forward in the future:

Provide packages for homebrew and chocolatey to ease the installation process in MacOS and Windows respectively.
Providing a lint command, that would allow to validate the final spec without connecting to the DigitalOcean API. Usage would be: appfile lint -f <appfile.yaml> -e <env_name>
Load App specs from a remote URL. That would be a first step towards a reusability of Apps in DigitalOcean and having access to pre-defined, customizable Apps
Support secrets encryption through an integration with sops

Conclusion #

Let's recap quickly the post. First, I talked about the DigitalOcean App platform and the obstacle of customizing the App specification to suit different environments requirements, resulting on the creation of appfile. Next, I provided an overview of the tool, how to install it, main features and how to use. Finally, I mentioned some future ideas where I could see appfile evolving to.

appfile has been a very interesting project to work on for the past few days. I have learned a lot about the DigitalOcean API and the App Platform in particular. I see the value that it brings to developers and some of the directions where it could go in the future are pretty interesting.

To conclude, thank you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

Terraforming AWS VPC

Rene Hernandez — Mon, 19 Oct 2020 00:00:00 +0000

AWS VPC stands for Virtual Private Cloud and it represents the networking layer for the AWS EC2 services (computing). In this post, we are going to cover how to automate the configuration of AWS VPC using Terraform. If you are not familiar with Terraform, you can check my introductory post here.

If you want to go straight to the code, you can check it out at aws-terraform-examples/03-terraforming-aws-vpc.

Already back? Great! Let's dive in.

Key concepts #

Before getting into the implementation, let's briefly mention some of the essential concepts around networking and VPC.

Availability Zones (AZ) and Regions #

AWS has two main concepts to refer to the physical locations of their data centers:

Availability Zone (AZ): It represents one or more discrete data centers with redundant power, networking and connectivity in a particular AWS Region. All AZs are connected with high-bandwidth, low-latency networking with fully redundant and dedicated fiber communications, and traffic is encrypted.
Region: Consists of multiple, isolated and physically separate AZ's within a geographic area.

Virtual Private Cloud (VPC) #

There are several key components in a VPC:

Subnet: A range of IP addresses. Can be either public or private.
Route table: A set of rules that are used to determine where network traffic is directed. Each subnet is associated with a route table.
Internet Gateway: Allows enabling communication between resources in your VPC and the internet and it serves two main purposes:
- Provides a target in your VPC route tables for internet-routable traffic.
- Performs network address translation (NAT) for instances that have been assigned public IPv4 addresses.
DNS in VPC: AWS provides an AWS Route53 Resolver to act as a DNS server for a given VPC. The key considerations are:
- Public and private DNS hostnames are provided for corresponding IPv4 addresses for each instance.
- Managing DNS in the VPC is done through the attributes enableDNSHostnames (indicates if instances with public IP addresses get corresponding DNS hostnames) and enableDNSSupport (indicates whether DNS resolution is supported).
NAT: A Network Translation Gateway (NAT) allows instances in a private subnet to connect to the internet while preventing external agents from initiating a request to the instance.

Terraform module #

In this post, we are going to use a Terraform module to facilitate the VPC declaration. According to the docs, a module is a container for multiple resources that work together. Modules can be used to create lightweight abstractions to describe infrastructure in terms of its architecture, rather than directly in terms of physical objects

Uff!! After a heavy dose of concepts, let's get right into the implementation.

Implementation #

The VPC declaration in Terraform is relatively short since we are leveraging the VPC Module) maintained by the Terraform community, which comes packed with sane abstractions and useful defaults.

To create the VPC, we need to obtain the existing Availability Zones in the current region. We can either manually specify the AZ names, or leverage instead the aws_availability_zones data source. This will allow Terraform to dynamically obtain the list of AZ from the region configured in the provider.

data "aws_availability_zones" "available" {}

Afterwards, we proceed to configure our new VPC as shown below:

module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "~> 2.0" name = "example" cidr = "10.0.0.0/16" azs = data.aws_availability_zones.available.names private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"] enable_nat_gateway = true single_nat_gateway = true enable_dns_hostnames = true tags = { Terraform = "true" Environment = "dev" }}

Let's discuss some of the fields in the module declaration above:

cidr field #

Represents the range of IPv4 addresses that are going to be available for this VPC. Generally should refer to IPs considered to be a private range (e.g. 10.0.0.0, 172.16.0.0 - 172.31.255.25, 192.168.0.0 - 192.168.255.255).
By assigning 10.0.0.0/16 to it, it means that the VPC will have access to 65536 (2^16) different IP addresses

Subnets fields #

Both private_subnets and public_subnets are specifying 3 different subnets to be created, each one with 256 (2^8) IP addresses allocated to it.
For both subnets, the module will make sure that the appropriate route tables and NAT gateways are created and attached.
For the public subnets, the module will wire them up with the Internet Gateway associated with the VPC.

NAT gateways fields #

enable_nat_gateway informs the module that NAT Gateways should be provisioned for the private networks
single_nat_gateway informs the module that all private subnets will share a single NAT Gateway

enable_dns_hostnames field #

Combined with enable_dns_support (which is enabled by default), this informs the module to configure DNS resolution for the public IP addresses associated with instances in the VPC.

Resources #

Below, it is a condensed list of all the resources mentioned throughout the post as well as a few others I consider may be of interest to deepen your knowledge.

AWS VPC:

AWS AZ:

AZ and region

Terraform:

Conclusion #

Let's sum up what we discussed in this post. First, we looked at some of the basic concepts around AWS VPC and AZ. Next, we dove into how to declare the VPC by leveraging the VPC module from Terraform and analyzed some of the fields that were specified as part of the module declaration. Finally, we listed the online resources used to create the post in case you want to go deeper in the topic.

Thank you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

Terraforming DNS with AWS Route53

Rene Hernandez — Fri, 09 Oct 2020 00:00:00 +0000

This post is part of a series about Terraform and AWS. You can check the other ones at:

First steps with Terraform in AWS
Terraforming DNS with AWS Route53 (this post)

AWS Route53 is a DNS service used to perform three main functions: domain registration, DNS routing, and health checking. In this post, we are going to cover how to automate the configuration of AWS Route53 as your DNS service using Terraform. If you are not familiar with Terraform, you can check my introductory post here.

If you want to go straight to the code, you can check it out at aws-terraform-examples/02-terraforming-dns-with-aws-route53.

Already back? Great! Let's dive in.

Basic concepts #

Before getting into the implementation, let's briefly mention some of the essential concepts around DNS and AWS Route53

Name servers: Servers in the DNS that help translate domain names (www.example.com) into IP addresses. They can be either DNS resolver or authoritative name servers.
Hosted zones: A container for DNS records, including information about how to route traffic for a domain (example.com) as well as its subdomains (sub.example.com).
DNS record: A particular entry in the hosted zone that specifies the traffic routing for the domain or subdomain.
Time To Live (TTL): The amount of time, in seconds, that the DNS resolver should cache the values for a record before submitting another request to the authoritative name servers.

This was just a general overview of the concepts that we are going to be leveraging for our infrastructure configuration. Let's move now to configure our DNS in AWS using Terraform.

Implementation #

To fully work with the code examples in this section, it is recommended that you use a domain that you own since you would need to configure the AWS Route53 nameservers on your domain registrar settings.

Configuring zone and nameservers #

The first step to configure the DNS service for your domain is to create the public hosted zone, which you can declare in Terraform as shown below.

resource "aws_route53_zone" "example" {
  name = "example.com"
}

As part of the creation of Route53 zones, the name server (NS) record, and the start of a zone of authority (SOA) record are automatically created by AWS. By using the allow_overwrite option below, Terraform is then capable of managing them in a single execution without the need for a subsequent terraform import.

resource "aws_route53_record" "nameservers" {
  allow_overwrite = true
  name = "example.com"
  ttl = 3600
  type = "NS"
  zone_id = aws_route53_zone.example.zone_id
  records = aws_route53_zone.example.name_servers
}

After this, if you are using a domain registrar other than Route53, you will need to add the name servers associated with your zone in Route53 to the configuration settings on your domain registrar website.

Configuring email #

Let's go through a possible email configuration as an example. We are going to be using ProtonMail as our email server of choice.

Verification #

First, we need to set up email verification. This is required by the email service to confirm that you own the domain. In the following example, we create a TXT record to hold the verification value.

resource "aws_route53_record" "protonmail_txt" {
  zone_id = aws_route53_zone.example.zone_id
  name = ""
  type = "TXT"
  ttl = 300
  records = ["protonmail-verification=<random_number>"]
}

The TXT record gets associated with the top-level domain in the zone by pointing to our previously created AWS Route53 zone using the zone_id attribute and setting the name attribute to empty. This effectively makes the TXT record refer to example.com in this scenario.

MX records #

The MX record specifies the mail server responsible to receive your domain's email.

resource "aws_route53_record" "protonmail_mx" {
  zone_id = aws_route53_zone.example.zone_id
  name = ""
  type = "MX"
  ttl = 1800
  records = [
    "10 mail.protonmail.ch.",
    "20 mailsec.protonmail.ch."
  ]
}

In TXT record configuration above, we are associating two ProtonMail servers for email deliverability. A lower number means a higher priority, therefore emails will be sent first to mail.protonmail.ch server and mailsec.protonmail.ch as a fallback in case of failure. Similar to the TXT record discussed before, the MX record is associated with the top-level example.com domain.

Sender Policy Framework (SPF) #

It is an authentication mechanism to validate that an email coming from a particular domain is being sent by an authorized IP address. For SPF, we need to use a TXT record entry specifying that the ProtonMail servers are authorized to send the emails. Since we already have a record resource representing a TXT record, we will reuse it and add a new entry in its records attributes as shown below.

resource "aws_route53_record" "protonmail_txt" {
  zone_id = aws_route53_zone.example.zone_id name = ""
  type = "TXT"
  ttl = 1800
  records = [
    "protonmail-verification=<random_number>",
    "v=spf1 include:_spf.protonmail.ch mx ~all"
  ]
}

DomainKeys Identified Mail (DKIM) #

It is another authentication technique that leverages cryptography to verify that email is sent by trusted servers. To manage the ProtonMail keys, we need to configure CNAME records to hold the public encryption keys. These keys will be used by the receiving servers to validate the emails, making sure that they haven't been tampered. Below, we define three new CNAME records in Terraform, one for each public encryption key provided by the email service.

resource "aws_route53_record" "protonmail_dkim_1" {
  zone_id = aws_route53_zone.example.zone_id
  name = "protonmail._domainkey"
  type = "CNAME"
  ttl = 1800
  records = ["domain_key1"]
}

resource "aws_route53_record" "protonmail_dkim_2" {
  zone_id = aws_route53_zone.example.zone_id
  name = "protonmail2._domainkey"
  type = "CNAME"
  ttl = 1800
  records = ["<domain_key2>"]
}

resource "aws_route53_record" "protonmail_dkim_3" {
  zone_id = aws_route53_zone.example.zone_id
  name = "protonmail3._domainkey"
  type = "CNAME"
  ttl = 1800
  records = ["domain_key3"]
}

Gotcha #

While reading several online guides, it is common to see the usage of @ as a hostname to refer to the top-level domain (in this post example.com). In AWS Route53, that doesn't work and instead, it is necessary to set the name attribute as empty to point to the root domain in the zone where the record is being added.

Resources #

Below, it is a condensed list of all the resources mentioned throughout the post as well as a few others I consider may be of interest to deepen your knowledge.

DNS:

AWS Route53:

Terraform:

Email:

Conclusions #

Let's briefly recap what we discussed in this post. First, we looked at some of the basic concepts around AWS Route53 and DNS. Next, we looked at how to attach a domain to AWS Route53 by using public hosted zone records and the nameservers. Afterward, we went through the steps of configuring an email service on our domain using different DNS records such as MX, TXT and CNAME. Finally, I analyzed a common gotcha regarding the usage of @ as the hostname and listed common resources mentioned throughout the post.

Thank you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

First steps with Terraform in AWS

Rene Hernandez — Wed, 30 Sep 2020 00:00:00 +0000

Terraform is a cloud-agnostic provisioning tool created by Hashicorp. It allows you manage your infrastructure in sane, safe and efficient manner by automating the proviisioning of your cloud resources (server, databases, DNS) in a declarative way, as well as leverage version control systems to keep track of the history of changes.

In this post, we are going to go over how to setup Terraform to work with AWS. If you want to go straight to the code, you can check it out at the setup example

Installing terraform #

There are different ways to install terraform depending on you operating system

Chocolatey on Windows #

choco install terraform

Homebrew on OS X #

Using the new hashicorp tap (recommended)

brew install hashicorp/tap/terraform

Using the community tap

brew install terraform

Linux #

Installing on Linux dependent on the distribution you are running. For Ubuntu/Debian:

Add the HashiCorp gpg key

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

Add the HashiCorp repository

sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

Update and install

sudo apt-get update && sudo apt-get install terraform

For more information and different ways to install, check the installation pages

Connecting AWS with terraform #

Now that we have the cli installed, let's get started connecting AWS with Terraform to manage the infrastructure.

Prerequisites #

To execute the rest of the project, you'll need to follow the next steps or use your existing configured AWS account.

An AWS account (a dedicated new one to execute terraform preferably, although it is OK to use your own AWS credentials for testing purposes)
The AWS CLI installed.
Configure your local aws cli with a dedicated terraform [profile

aws configure --profile terraform

Using the above command, input at the prompt your AWS Access Key ID and Secret Key

Defining the AWS provider #

According to the Terraform documentation, a provider is essentially a plugin that offers a set of abstraction for certain API resources and their interactions. Usually, each provider focuses on a specific infrastructure platform.

Terraform needs to know which provider to download from the Terraform Registry. For that, we can use the terraform block to list all the providers that the code will use. For our scenario, we can list the aws provider as shown below:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
  required_version = ">= 0.13"
}

Now that we listed the aws provider, let's add the configuration data for the AWS provider. Below, we are using the us-east-2 region and loading the credentials to connect to AWS from the terraform profile we created on the Prerequisites section.

provider "aws" { 
  region = "us-east-2" 
  profile = "terraform"
}

For alternative means of authentication, dive in the docs and always make sure that you don't store your AWS credentials in plaintext on terraform.

At this point, you are ready to use Terraform to automate your infrastructure. In the next section, we'll dive into storing the backend information remotely to facilitate team workflows.

S3 for remote backend #

First, I'll briefly mention some of the advantages of using a remote backend:

Improves working in a team since state can be protected with locks to avoid multiple interactions at the same time and possible corruptions.
It helps keep sensitive data off disk. Terraform stores the state in plaintext and if using a backend it will only be stored on the backend location (e.g. S3 bucket)
If the backend supports remote operations, it means that terraform apply can be executed on the backend instead of your local machine, for an overall improved experience.

Before going through the code, let's briefly look at what Terraform state entails.

Terraform state #

Terraform stores the state of your infrastructure for several reasons:

Mapping to the Real World: Terraform maps each configuration to a corresponding resource in the real world and it leverages the state to verify that no two configurations elements represent the same endpoint
Metadata: It includes tracking knowledge such as resource dependencies or workflows that are necessary for resources to work as expected.
Performance: Optionally, the state can be considered as a source of truth and therefore API request to the providers are done only when specified. This helps improve performance for large teams.

This state is stored in a file, usually called terraform.tfstate using a JSON format. Let's move on to the implementation

Implementation #

To set up the remote backend, we need the following resources:

A S3 bucket where the terraform state will stored
A DynamoDB table to lock the access to the state file
IAM policies to give permission to the user to access the S3 bucket and the DynamoDB table (required if you are using an IAM user with more restrictive credentials than just AdministratorAccess access)

S3 bucket #

resource "aws_s3_bucket" "terraform_state_storage" {
  bucket = "terraform-remote-state-storage-s3"
  acl = "private"
  tags = {
    name = "Terraform Storage"
    dedicated = "infra"
  }
  versioning {
    enabled = true
  }
  lifecycle {
    prevent_destroy = true
  }
}

The S3 bucket as shown above, gets created with the following settings:

Private Access List Control (ACL) grant to limit access to the bucket.
Versioning enabled to allow for state recovery in case of accidental deletion and file corruption
Prevention of accidental destruction of the S3 bucket while running terraform operations

DynamoDB table #

# create a dynamodb table for locking the state file.
# this is important when sharing the same state file across users
resource "aws_dynamodb_table" "terraform_state_lock" {
  name = "terraform-state-lock"
  hash_key = "LockID"
  read_capacity = 20
  write_capacity = 20
  attribute {
    name = "LockID"
    type = "S"
  }
  tags = {
    name = "DynamoDB Terraform State Lock Table"
    dedicated = "infra"
  }
  lifecycle {
    prevent_destroy = true
  }
}

The DynamoDB table gets configured with the following properties:

A LockID hash key of type string, so that all items created by terraform operations are stored together in the same bucket
The read and write capacity per seconds for the table. This specifies how read/write operations are we allowed to execute against the table
Similar to the S3 bucket above, the resource is created with prevention of accidental destruction while running terraform operations.

IAM policies #

data "aws_iam_policy_document" "terraform_storage_state_access" {
  statement {
    effect = "Allow"
    actions = ["s3:ListBucket"]
    resources = [aws_s3_bucket.terraform_state_storage.arn]
  } 
  statement {
    effect = "Allow"
    actions = ["s3:GetObject", "s3:PutObject"]
    resources = ["${aws_s3_bucket.terraform_state_storage.arn}/terraform.tfstate"]
  }
}

# Creates the IAM policy to allow access to the bucket
resource "aws_iam_policy" "terraform_storage_state_access" {
  name = "terraform_storage_state_access"
  policy = data.aws_iam_policy_document.terraform_storage_state_access.json
}

# Assigns the policy to the terraform user
resource "aws_iam_user_policy_attachment" "terraform_storage_state_attachment" {
  user = "terraform" policy_arn = aws_iam_policy.terraform_storage_state_access.arn
}

The above IAM policy describes the permission for a user to access the S3 bucket:

Ability to list the bucket
Get objects from the bucket
Add new objects to it

As part of the setup, the policy is attached to the terraform user, so it can have access to S3 bucket.

data "aws_iam_policy_document" "dynamodb_access" {
  statement {
    effect = "Allow"
    actions = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem"] 
    resources = ["arn:aws:dynamodb:*:*:table/terraform-state-lock"]
  }
}

# Creates the IAM policy to allow access to the dynamoDB
resource "aws_iam_policy" "dynamodb_access" {
  name = "dynamodb_access"
  policy = data.aws_iam_policy_document.dynamodb_access.json
}

# Assigns the policy to the terraform user
resource "aws_iam_user_policy_attachment" "dynamodb_attachment" {
  user = local.terraform_user policy_arn = aws_iam_policy.dynamodb_access.arn
}

Similar to the previous IAM policy, this IAM policy describes the permission given to the user when accessing the DynamoDB table:

Get items from the table
Add items to the table
Remove items from the table

The policy is then attached to the terraform user, so it can have access to the DynamoDB table.

Remote backend #

Finally, to inform terraform to use the s3 bucket as remote backend, just add the following block:

terraform {
  backend "s3" {
    bucket = "terraform-state-storage"
    key = "terraform.tfstate"
    region = "us-east-2"
    dynamodb_table = "terraform-state-lock"
    profile = "terraform"
  }
}

Resources #

Below, it is a condensed list of all the resources mentioned throughout the posts as well as a few others I consider may be of interest to deepen your knowledge.

Providers:

Terraform remote storage:

S3:

DynamoDB:

IAM:

Conclusions #

To sum up, this post provided an intro to Terraform in AWS. It covered how to install Terraform and configure it to manage AWS resources. As further steps, we went through the steps of configuring S3 and DynamoDB for managing the Terraform state remotely with locking included. This is a desired setup if working on a team. If you want to go straight to the code, go check it out at renehernandez/aws-terraform-examples

Thanks you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

Automate changelog and releases creation in GitHub

Rene Hernandez — Wed, 23 Sep 2020 00:00:00 +0000

Keeping up to date the Changelog and generating GitHub releases is one of those tasks I always think it is important to do, but I feel it becomes a chore the more you have to do it manually on a given project. In one of my recent OSS projects, camper, I decided from the beginning that I didn't want to be manually generating the Changelog and the GitHub releases information.

After researching different options, I landed on github-changelog-generator. A neat project, it is a Ruby gem that allows you to automatically generate a changelog based on tags , issues and merged pull requests.

Implementation #

Since the project is hosted on GitHub, I went with GitHub Actions for the implementation as part of the CI process. It was an opportunity to put Actions in practice and get familiar with it. This post is not about an introduction to GitHub Actions, check instead the Actions Docs for getting started and diving deep into the subject.

Already back!! Great, let's go into the details.

First let's discuss the main requirements that I had in mind:

When committing to main branch:

A new updated Changelog should be generated and committed to main. This would account for merged PRs, as well as any direct commit to main.
All latest changes that are not already part of the tagged released, should be grouped under an Unreleased section at the top of the Changelog

When pushing a new tag:

Update the Changelog moving all the unreleased changes under the new tag.
Create a new GitHub release containing all the information associated with the latest Changelog tagged entry.

CI - Changelog workflow #

As I explained in the previous section, the Changelog update process consists of two parts. One when merging to main and the other when pushing a new tag. The CI - Changelog workflow, as shown below, fulfills the requirement of updating the Changelog on every push to the main branch. You can find the most up to date version at here

name: CI - Changelog

on:
  push:
    branches: [main]

jobs:
  changelog_prerelease:
    name: Update Changelog For Prerelease
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: main
      - name: Update Changelog
        uses: heinrichreimer/github-changelog-generator-action@v2.1.1
        with:
          token: $
          issues: true
          issuesWoLabels: true
          pullRequests: true
          prWoLabels: true
          unreleased: true
          addSections: '{"documentation":{"prefix":" **Documentation:**","labels":["documentation"]}}'
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Update Changelog for PR
          file_pattern: CHANGELOG.md

It works as follows:

It checks out the code using the action/checkout v2
It proceeds to generate an update for the Changelog by using the heinrichreimer/github-changelog-generator-action action with the following customizations:
- All closed issues should be part of the Changelog, including those without labels (issues: true and issuesWoLabels: true)
- All pull requests should be part of the Changelog, including those without labels (pullRequests: true and prWoLabels: true)
- It should group all latest changes under an Unreleased section (unreleased: true)
- It adds a new Documentation section to group issues and pull requests with the documentation label
Then it commits the modified Changelog file back to main using the stefanzweifel/git-auto-commit-action action

Release workflow #

The Release workflow is a more complex pipeline since it not only updates the Changelog, but also handles the publishing of a new gem version as well as a new GitHub release associated with the tag being pushed. For this post, we are only focusing on the Changelog and GitHub release related jobs. If you are interested, check the full workflow here

name: Release

on:
  push:
    tags:
      - v*

jobs:
# Other jobs
# ...
  changelog:
    name: Update Changelog
    runs-on: ubuntu-latest
    steps:
      - name: Get version from tag
        env:
          GITHUB_REF: $
        run: |
          export CURRENT_VERSION=${GITHUB_TAG/refs\/tags\/v/}
          echo "::set-env name=CURRENT_VERSION::$CURRENT_VERSION"
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: main
      - name: Update Changelog
        uses: heinrichreimer/github-changelog-generator-action@v2.1.1
        with:
          token: $
          issues: true
          issuesWoLabels: true
          pullRequests: true
          prWoLabels: true
          addSections: '{"documentation":{"prefix":" **Documentation:**","labels":["documentation"]}}'
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Update Changelog for tag $
          file_pattern: CHANGELOG.md

  release_notes:
    name: Create Release Notes
    runs-on: ubuntu-latest
    needs: changelog
    steps:
      - name: Get version from tag
        env:
          GITHUB_REF: $
        run: |
          export CURRENT_VERSION=${GITHUB_TAG/refs\/tags\/v/}
          echo "::set-env name=CURRENT_VERSION::$CURRENT_VERSION"

      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: main

      - name: Get Changelog Entry
        id: changelog_reader
        uses: mindsers/changelog-reader-action@v1
        with:
          version: $
          path: ./CHANGELOG.md

      - name: Create Release
        id: create_release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: $ # This token is provided by Actions, you do not need to create your own token
        with:
          tag_name: $
          release_name: Release $
          body: $
          draft: false
          prerelease: false

The first job, Update Changelog , is almost the same as the one described on the previous section.The difference is that we are generating released versions only and thus, there is no unreleased: true entry.

The second , Create Release Notes , works as follows:

It relies on the updated Changelog, thus the presence of needs: changelog to force it to wait for the previous changelog job completion.
Using the mindsers/changelog-reader-action, it proceeds to select the changelog entry associated with the tag being pushed.
Using the actions/create-release, it generates the GitHub Release using the content of the changelog entry extracted on the previous step

GitHub action gotchas #

The changelog generator action has some gotchas that are not easy to spot:

The majority of options specified in the Update Changelog step, such as issues: true and pullRequests: true default to true on the underlying github-changelog-generator gem, but are required as part of the action, otherwise they get set to false. That tripped me over for a while, until I read the action's implementation, specifically the entrypoint.sh
Adding a new section using the addSections field fails if you specify a prefix with multiple words (e.g, Documentations updates as the changelog generator wiki suggests). The issue is with word splitting on the entrypoint.sh as discussed in issue#3.

Changelog generator limitation #

While iterating on the output produced by the changelog-generator gem, I realized that I was getting double entries between PRs that are linked to issues (i.e. PRs that close issues when merged). I dug in the documentation trying to find a way of just showing either the issue or PR to no avail. Then I posted an issue on the github repo and confirmed my suspictions that it is not currently a way to this, due to limitations on the GitHub REST API.

Conclusions #

In this post, we discussed how to automate Changelog and GitHub Releases creation. We went over the details of each of the workflows and describe the steps for each job involved in the workflows. We also mentioned some limitations and gotchas for the github actions and the github-changelog-generator gem.

To conclude, thanks you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

Faster deployments of mysql databases in k8s

Rene Hernandez — Tue, 11 Aug 2020 00:00:00 +0000

Since I started at my new job, I have been inmersed in learning about a whole lot of new things, including Kubernetes and cloud. My first task, very challenging at the time, was to optimize certain part of our CI pipeline.

What exactly? Read below to find out.

The problem #

As part of our CI pipeline, we provide our team with the ability to generate reviews environments with a one-click deployment step. These review environments provide a production-like environment for developers to test their changes in the main application. The main difference between the review environments and production environment is that all the dependent systems are configured using containers running in the cluster and hold comparatively smaller datasets versus production.

The problem we were facing at the time, was that review environments were taking a long time to start, almost reaching 10 minutes from when the developer started the deployment up until the application was running and ready to use. The culprit? The mysql container, which was taking the bulk of time loading a database dump on startup and thus the application was left hanging until the database was ready to use.

With this problem at hand, we looked at several options to better the performance and finally settled on trying to eliminate the dump loading step as a runtime stage during the deployment.

The Solution #

As mentioned above, we decided that the best approach would be to move the dump expansion out of the database initialization stage and instead, process the dump during the image build creation. That way, we would pay the price (in time spent expanding the dump and loading into the database container) just once.

With this idea in mind, there was one requirement that we needed to comply with:

Updates of review environments reuse the same database pod to preserve custom data that developers may have created during their testing efforts. That meant, we should only copy the expanded database on the first startup.

Custom mysql container with loaded data #

Our first attempt was to produce a custom mysql image with the dump already loaded in. After several iterations on how to achieve this, we finally settled on the following multi-stage Dockerfile shown below:

FROM mysql:5.7 as builder

RUN apt-get update && apt-get upgrade -y

# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$@\"/echo \"not running $@\"/", "/usr/local/bin/docker-entrypoint.sh"]

ARG dump
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD

ENV MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD MYSQL_DATABASE=$MYSQL_DATABASE MYSQL_USER=$MYSQL_USER MYSQL_PASSWORD=$MYSQL_PASSWORD

COPY ${dump} /docker-entrypoint-initdb.d

# Need to change the datadir to something else that /var/lib/mysql because the parent docker file defines it as a volume.
# https://docs.docker.com/engine/reference/builder/#volume :
# Changing the volume from within the Dockerfile: If any build steps change the data within the volume after
# it has been declared, those changes will be discarded.
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db"]

FROM mysql:5.7

COPY --from=builder ./initialized-db /var/lib/mysql/

As we can see, the first stage loads the database dump into mysql. Since we initially modify the entrypoint script for the image, it won't start the mysql process, just the initialization process of expanding the dump. Then, the second stage generates the final image, by copying the generated database files from the first stage to the /var/lib/mysql folder in the second stage.

For my initial testing with the resulting image, I was using a docker-compose deployment to simulate the kubernetes environment and results were promising. The startup time for the mysql container had been reduced significantly and the application was able to connect to the backend successfully. Then, I went to try deploying this custom mysql image with the mysql chart in kubernetes and it failed to start the corresponding mysql pod.

The reason for this failure is due to the difference on how docker treats volumes vs kubernetes, which I wasn't aware beforehand. In Docker, if the volume is empty, then the data on the container (/var/lib/mysql in this scenario), will be copied to the corresponding volume. On the other hand, Kubernetes will override whatever is in th container at the path where the Persistent Volume is being mounted, so I couldn't get the Mysql container up and running populated with the expanded data.

After several iterations and analysis, I landed in a feasible solution using init containers, which I dive into below.

Init container to load data #

Init containers run before the main application containers start in a pod and they are usually used to perform one-off tasks required before the main application boots. Below, is the Dockerfile definition for the image that will run as init container.

FROM mysql:5.7 as builder

RUN apt-get update && apt-get upgrade -y

# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$@\"/echo \"not running $@\"/", "/usr/local/bin/docker-entrypoint.sh"]

ARG dump
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD

ENV MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD MYSQL_DATABASE=$MYSQL_DATABASE MYSQL_USER=$MYSQL_USER MYSQL_PASSWORD=$MYSQL_PASSWORD

COPY ${dump} /docker-entrypoint-initdb.d

# Need to change the datadir to something else that /var/lib/mysql because the parent docker file defines it as a volume.
# https://docs.docker.com/engine/reference/builder/#volume :
# Changing the volume from within the Dockerfile: If any build steps change the data within the volume after
# it has been declared, those changes will be discarded.
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db"]

FROM alpine:3.7

RUN apk add --no-cache bash

COPY --from=builder ./initialized-db /mysql_data

WORKDIR /script

COPY scripts/copy_data_to_volume.sh ./

CMD ["bash", "copy_data_to_volume.sh"]

The main differences between this Dockerfile and the previous one are:

Instead of creating a mysql image, it creates an alpine image running a bash script on startup
It copies the mysql files after expanding the dump into a /mysql_data folder instead of the mysql location in the mysql container.

The copy_data_to_volume.sh script, as shown below, takes care of copying the data from the /mysql_data folder to a destination folder called /initialized-db, as long as the .data-initialized flag file is not present. This /initialized-db folder should be mounted as the data volume in the pod where the init container is running.

if [[-f "/initialized-db/.data-initialized"]]; then
  echo "DATABASE already initialized. Nothing else left to do"
  exit 0
fi

echo "Copying seeded database (one-time operation)"
cp -a /mysql_data/. /initialized-db

echo "Create file to mark first-time initialization completed"
touch /initialized-db/.data-initialized

The mysql chart supports specifying init containers through the extraInitContainers field, as shown in the following code section. As I mentioned above, we are mounting the data volume in the /initialized-db folder within the init container. And this is what makes the process work. That same data volume will be mounted by the main mysql container and it will have the already present on startup

mysql:
  image: mysql
  imageTag: 5.7
  extraInitContainers: >-
    - name: seed-database
      image: 
      volumeMounts:
        - name: data
          mountPath: "/initialized-db"

Results #

After deploying the new version of the helm chart supporting this feature, we saw a reduction of the startup time from around 10 minutes to less than 3 minutes, which means that we gave developers back 7 minutes of productivity :).

Conclusions #

To sum up, we discussed how to improve the startup performance of the mysql chart deployment by off-loading the process of expanding the sql dumps into the database to a dedicated data image. We discussed 2 solution attempts, the first one being the deployment of a mysql container with the dump already expanded. That solution wasn't feasible in a kubernetes environment, and we moved to the second solution attempt, which reused the same idea of expanding the dump, but instead deployed it as part of an init container running on the mysql helm chart.

To conclude, thanks you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!

Docs Experience - deploying Gitlab Wikis as mkdocs sites

Rene Hernandez — Mon, 18 May 2020 00:00:00 +0000

The past week, I have improved our documentation experience at work. The issues we have tackled recently ranged from implementing a search endpoint to scrape documentation from multiple different endpoints such as GitLab Wikis, mkdocs websites, among others. Being capable of processing documentation from different applications fits our goal of making easy for developers to write their docs, and we do this by meeting them where they write documention, instead of forcing them to follow specific patterns and guidelines on how to create documentation.

The first documentation endpoint that I processed was our Gitlab Wikis. When analyzing the layout of the wikis, I realize that it was going to take us more time to properly limit the scraping content to the GitLab Wiki and avoid processing other types of content in the GitLab website (e.g, code, issues, MR, etc.).

After exploring several alternative, I settled on generating mkdocs websites from Wiki content, mainly for the following reasons:

The Wiki's content is written in Markdown, which makes mkdocs a very good match to generate websites with.
We were already deploying documentation written in Markdown as mkdocs sites for several internal projects.

Wiki Releaser #

To automate transforming a GitLab Wiki into a mkdocs website, I created a new GitLab project which hosts the logic to go from cloning the GitLab repo to trigger a deployment in our kubernetes infrastructure of the the generated mkdocs docker image.

The repository contains only 3 files:

mkdocs.yaml: Configuration file used by mkdocs to produce the website.
Dockerfile: Specification to generate the resulting docker image with the built mkdocs website
.gitlab-ci.yaml: Pipeline configuration used by GitLab to build and deploy changes to the Wikis as websites

mkdocs.yml config #

Our mkdocs websites are built using the readthedocs theme, with several css customizations that are included through the override.css file. As part of the image build process (explained later), the <SITE_NAME> holder is replaced by the actual name of the site to be deployed.

site_name: <SITE_NAME>
extra_css:
  - "override.css"
theme:
  name: readthedocs
  collapse_navigation: false
  hljs_languages:
    - yaml

markdown_extensions:
  - admonition
  - fenced_code
  - tables
  - toc:
      permalink: true

Dockerfile #

The Dockerfile below is built as part of the CI process (explained in the next section), using the invocation:

docker build --build-arg WIKI_FOLDER="$PROJECT_NAME.wiki" --build-arg SITE_NAME="$SITE_NAME" -t <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:$TAG_REF .

which passes down the cloned wiki folder as the WIKI_FOLDER argument and the SITE_NAME variable as argument for the build image process.

Important points in the Dockerfile definition:

As mentioned in the mkdocs section above, one step is to replace the <SITE_NAME> holder with the SITE_NAME argument in the mkdocs.yml file.
mdkocs expects an index.md file, which will be the root page of the website. GitLab Wikis have a Home.md (or home.md) page as the root instead, so the Dockerfile renames it during the build process.

FROM python:3.7.2 as build

RUN pip install mkdocs==1.1.1 && mkdir /site

WORKDIR /site

ARG WIKI_FOLDER

ARG SITE_NAME

COPY $WIKI_FOLDER /site/docs

COPY override.css /site/docs

COPY mkdocs.yml mkdocs.yml

# Replace <SITE_NAME> holder with SITE_NAME argument value
RUN sed -i "s/<SITE_NAME>/${SITE_NAME}/g" mkdocs.yml

RUN if [-f ./docs/Home.md]; then mv ./docs/Home.md ./docs/index.md; else mv ./docs/home.md ./docs/index.md; fi

RUN mkdocs build

FROM nginx:1.17.2-alpine

COPY --from=build /site/site /usr/share/nginx/html

GitLab CI #

As shown in the .gitlab-ci.yml pipeline configuration below, our CI/CD process for the Wiki Releaser project has the following requirements:

It is executed only when a trigger is received
The trigger needs to provide PROJECT_PATH, PROJECT_NAME, SITE_NAME variables, otherwise the build job fails:
- PROJECT_PATH: Refers to the GitLab path of the project associated with the wiki in the format of <group>/<project_name> (e.g devs/hello_world).
- PROJECT_NAME: Refers to the project name of the GitLab project.
- SITE_NAME: Refers to the name that will be shown on the mkdocs site.
It will perform a shallow clone of the wiki repo (latest commit only and no tags) for performance reasons
It will the use the CI_JOB_TOKEN variable to authenticate against the wiki repo for the clone. This token provides the user read access to all projects that would be normally accessible to the user creating that job.
It will generate docker images containing the resulting mkdocs website with the following naming pattern:
- <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:$TAG_REF (TAG_REF holds the commit SHA value)
- <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:latest
We use helmfile to handle charts deployment to kubernetes

include:
  - project: 'cicd'
    file: 'ci/helm_pipeline.yaml'

variables: &variables
  HELMFILE: docs/helmfile.yaml
  DISABLE_STAGING: 'true'
  HELMFILE_ENV: wiki
  K8S_ENV: <kubernetes_env>
  OTHER_K8S_ENV: -e TAG_REF=latest -e PROJECT_NAME=$PROJECT_NAME -e PROJECT_PATH=$PROJECT_PATH # Variables we pass down to our helmfile deployment logic

.trigger: &trigger
  only:
    refs:
     - triggers
  after_script:
    - echo "Triggered from $PROJECT_PATH wiki using $PROJECT_NAME project and $SITE_NAME for site"

build:
  extends: .build
  <<: *trigger
  image: <internal_docker_repo>/devops/ci-images/docker-with-git:latest
  script:
    - if [$PROJECT_PATH == ""]; then exit 1; fi
    - if [$PROJECT_NAME == ""]; then exit 1; fi
    - if ["$SITE_NAME" == ""]; then exit 1; fi
    - git clone --depth=1 --no-tags https://gitlab-ci-token:${CI_JOB_TOKEN}@<gitlab_url>/$PROJECT_PATH.wiki.git
    - TAG_REF=$(git -C ./$PROJECT_NAME.wiki rev-parse HEAD)
    - docker build --build-arg WIKI_FOLDER="$PROJECT_NAME.wiki" --build-arg SITE_NAME="$SITE_NAME" -t <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:$TAG_REF .
    - docker push <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:$TAG_REF
    - docker tag <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:$TAG_REF <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:latest
    - docker push <internal_docker_repo>/devops/wiki-releaser/$PROJECT_PATH:latest

# Deployment to kubernetes
deploy:
  <<: *trigger
  environment:
    name: $PROJECT_PATH
    url: https://$PROJECT_NAME.docs.domain
  variables:
    <<: *variables

Triggers #

As I mentioned in the previous section, the GitLab pipeline is only executed with it is invoked by an incoming trigger from another repo that wants to build its wiki. For it to happen, we need to set up some configuration options in each of the repos.

Wiki Releaser #

We configure a Pipeline Trigger and we use the generated token as the to allow access to the dependent projects that are invoking this pipeline via the trigger.

Dependent projects #

On projects that want to invoke Wiki Releaser to generate mkdocs websites for their wikis, we configure a webhook. This webhook will only be triggered for Wiki Page events and the hook will be a URL to invoke the Wiki Releaser pipeline, passing along with it, the token specified on the trigger definition in the Wiki Releaser project, plus all the required variables (i.e. PROJECT_NAME, PROJECT_PATH and SITE_NAME).

Conclusion #

Let's recap the content of the post. First, I talked a bit about the importance of documentation and how in my company, we try to follow developers practices instead of enforcing new ones. Then, I dove into the main problem which was how to analyze and extract data from GitLab Wikis and why I set on using mkdocs to generate websites as a solution. Finally, I introduced the Wiki Releaser project, what its different components are and their purposes, and how the triggers tie everything together.

To conclude, thanks you so much for reading this post. Hope you enjoyed reading it as much as I did writing it. See you soon and stay tuned for more!!