DEV Community: David J Eddy

My thoughts on the HashiCorp Infrastructure Automation Certification

David J Eddy — Thu, 28 Oct 2021 16:17:30 +0000

As the landscape technologies that keep the internet running has changes over the past 60 years so have the tools that manage the technology. Now in 2021 Terraform is one of the leaders for managing cloud resources as code, commonly called Infrastructure as Code.

I started using Terraform (TF) around version 0.11 back in 2017. The first project published with TF code was in 2018 (https://github.com/davidjeddy/wordpress-terraform if you are curious). As my skills with Terraform have matured, so has the tool. But the core life cycle remains the same: write, plan, apply. Even as the tool passed the 1.x production ready release milestone; the core workflow remained unchanged.

Recently I found that Hashicorp has started providing certifications related to there tools; of course I jumped on the Terraform study track. The study plan was to go through the ACG resources, complete the Hashicorp resources, review my knowledge, and execute the test. Having taking a number of certifications the last step was going to be the easy one.

Here is the list of resources I used to to study.

The exam topics covers everything in the Terraform lifecycle from the subcommands, to state manipulation and a working knowledge of the Cloud / Enterprise offerings from Hashicorp.

Due to world events the exam was online proctored. Finding a place to take the exam was a non-issue for me due to current living arrangements. The monitoring personnel are _very_ strict about adhering to the guidelines. You have been warned.

With all that said, I passed with a score in the mid 80s. Not amazing, but not bad either. Two weeks after the exam I was watching the HashiConf Global 2021 and saw that only about 12,000 certs had been issued globally so far. That means I am one of the first 15,000 to have a certification from Hashicorp. Woot woot!

Would I recommend this certification? If you like to validate your knowledge and increase your salary; yes. 100% yes. Especially if you work in the cloud, IT infrastructure, or application development.

How to: AWS Service Endpoints via Terraform for fun and profit

David J Eddy — Wed, 01 Apr 2020 15:35:18 +0000

Originally posted on my blog.

Recently I found myself designing a system that had AWS Lambda functions inside a private VPC. But I needed to pass a payload from the output of the Lambda function to an AWS service that had to be publicly routable (specifically to SQS). I found there are really only three options to solve this situation:

The Options:

1) NAT Instance (Good)

This solution involves operating a compute instance to act as a network address translator (NAT) resource. When resources inside the private subnet needs to access a public DNS the traffic is routed through the NAT instance. This has the obvious disadvantage of needing to run a compute instance and being limited to the hardware related to it. This adds management and cost overhead that I really did not want to deal with. While the AMI Marketplace has pre-configured images available I still did not want to manage additional hardware for one Lambda that is invoked sporadically.

Here is what a NAT Instance network configuration looks like:

Credit: AWS

2) NAT Gateway

The better option is to leverage the NAT Gateway service. Imagine if you let AWS operate a NAT instance super-cluster with the additional benefit of lower cost to operate, easier setup, and higher network through put. The down side is the inability to use Security Groups with it This is the recommended solution for current NAT requirements going forward per AWS.

Here is what a NAT Gateway network configuration looks like:

Credit: AWS

3) Service Endpoint (Best)

The new kid on the block, Service Endpoints enable the ability to access supported services from within a private subnet with major benefits over NAT implementations. Imagine if you connected a network cable from your private subnet directly to the publicly routed resource. AWS does this via the an Elastic Network Interface (ENI) resource to the private subnet. The ENI even takes up an IP address in the CIDR range of the private subnet.

This solution have three big benefits:

Traffic stays inside your VPC, never traversing the public internet. Thus faster, cheaper, and more secure.
Similar to a NAT instance, Service Endpoints can have Security Groups applied to them.
The infrastructure to operate and manage a Service Endpoint is incredibly minimal. Saving time, money, and operational effort.

This is the solution I wanted! Service Endpoints checks all the requirement boxes I had.

*Side Note: Service Endpoint Interfaces are an AWS service implementations of the Private Link feature. Service Endpoint Gateways are only available for S3 and DynamoDB. The Terraform configuration is minimally different between the two.

Here is what a Service Endpoint network configuration looks like:

Credit: AWS

Lets Terraform This Bad Boy!

VPC

Leveraging Terraform (0.12.24 at time of writing) I configured a basic VPC, a single AZ with a private subnet, and a wide open Security Group. Very basic networking here; nothing special, the core building blocks of any VPC. Note the VPC does not have any NAT resources nor an Internet Gateway.

# Networking

## VPC

resource aws_vpc this {
  assign_generated_ipv6_cidr_block = false
  cidr_block                       = var.vpc_private_cidr
  enable_dns_hostnames             = true
  enable_dns_support               = true

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "vpc", random_string.this.result])
      Tech = "VPC"
      Srv  = "VPC"
    },
    var.tags
  )
}

## Route Table <-> Subnet associations

resource aws_route_table_association private_0 {
  subnet_id      = aws_subnet.private_0.id
  route_table_id = aws_route_table.private_0.id
}

## Route Tables

resource aws_route_table private_0 {
  vpc_id = aws_vpc.this.id

  depends_on = [
    aws_vpc.this
  ]

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "private-route", random_string.this.result])
      Tech = "Route"
      Srv  = "VPC"
    },
    var.tags
  )
}

## Subnets

resource aws_subnet private_0 {
  availability_zone               = var.availability_zone[0]
  vpc_id                          = aws_vpc.this.id
  cidr_block                      = var.vpc_private_cidr
  assign_ipv6_address_on_creation = false

  depends_on = [
    aws_vpc.this
  ]

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "subnet-a", random_string.this.result])
      Tech = "Subnet"
      Srv  = "VPC"
      Note = "Private"
    },
    var.tags
  )
}

 
resource aws_security_group private_lambda_0 {
  description = "Private Lambda SG"
  name        = join(var.delimiter, [var.name, var.stage, "private-subnet-lambda-0", random_string.this.id])
  vpc_id      = aws_vpc.this.id

  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [
      var.vpc_private_cidr
    ]
  }

  egress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [
      var.vpc_private_cidr
    ]
  }

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "private-subnet-lambda-0", random_string.this.id])
      Tech = "Security Group"
      Srv  = "EC2"
    },
    var.tags
  )
}

SQS Queue

The first resource after the base VPC resources I needed to create was the SQS queue. Like many other services offered by AWS the queues has a routable FQDNs. Leverage proper security and IAM configuration! Principle of least privilege to secure _all_ your resources. Remember: security first.

resource aws_sqs_queue dead_letter_queue {
  name = join(var.delimiter, [var.name, var.stage, "sqs-dead-letter", var.random_string.id])

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "sqs-dead-letter", var.random_string.id])
      Tech = "SQS"
      Srv  = "SQS"
    },
    var.tags
  )
}

resource aws_sqs_queue this {
  name                      = join(var.delimiter, [var.name, var.stage, "sqs", var.random_string.id])

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dead_letter_queue.arn
    maxReceiveCount     = 4
  })

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "sqs", var.random_string.id])
      Tech = "SQS"
      Srv  = "SQS"
    },
    var.tags
  )
}

Private Lambda

Next I created a Lambda function; assigning it to the private subnet and the security group that are contained inside the VPC. The Lambda code is Python based and as such I used Boto3 to handle creating the HTTPS request that will place the message in the queue. This will not work initially since we have not created the Service Endpoint.

## data

data archive_file this {
  type        = "zip"
  source_dir  = "${path.module}/src"
  output_path = "${path.module}/file.zip"
}

## resources

resource aws_lambda_function this {
  filename         = data.archive_file.this.output_path
  function_name    = join("-", [var.stage, var.name, "private-lambda", var.random_string.id])
  handler          = "index.lambda_handler"
  role             = aws_iam_role.this.arn
  runtime          = "python3.7"
  source_code_hash = data.archive_file.this.output_base64sha256

  # NOTE Need to pass the REGION and QUEUE_ARN to enable Boto3 to find the correct queue
  environment {
    variables = {
      AWS_ACCT_ID = var.aws_acct_id
      QUEUE_ARN   = var.aws_sqs_queue.arn
      REGION      = var.region
    }
  }

  # NOTE This places the Lambda inside a VPC into the subnet of choice
  vpc_config {
    security_group_ids = var.security_group_ids
    subnet_ids         = var.subnet_ids
  }

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "private-lambda", var.random_string.id])
      Tech = "Python_3_7"
      Srv  = "Lambda"
    },
    var.tags
  )
}

## IAM role, policies, and attachments

resource aws_iam_policy this {
  name   = join(var.delimiter, [var.name, var.stage, "private-lambda-policy", var.random_string.id])
  path   = "/"
  policy = file("${path.module}/iam/policy.json")
}

resource aws_iam_role this {
  assume_role_policy = file("${path.module}/iam/role.json")
  name               = join(var.delimiter, [var.name, var.stage, "private-lambda-role", var.random_string.id])
}

resource aws_iam_role_policy_attachment this {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

Public Lambda

The second Lambda I made will consume the SQS queue. Notice the configuration does not include a VPC or Subnet configuration?This means the Lambda will be public within my account.

## data

data archive_file this {
  type        = "zip"
  source_dir  = "${path.module}/src"
  output_path = "${path.module}/file.zip"
}

## resources

resource aws_lambda_function this {
  filename         = data.archive_file.this.output_path
  function_name    = join("-", [var.stage, var.name, "public-lambda", var.random_string.id])
  handler          = "index.lambda_handler"
  role             = aws_iam_role.this.arn
  runtime          = "python3.7"
  source_code_hash = data.archive_file.this.output_base64sha256

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "public-lambda", var.random_string.id])
      Tech = "Python_3_7"
      Srv  = "Lambda"
    },
    var.tags
  )
}

## IAM role, policies, and attachments

resource aws_iam_policy this {
  name   = join(var.delimiter, [var.name, var.stage, "public-lambda-policy", var.random_string.id])
  path   = "/"
  policy = file("${path.module}/iam/policy.json")
}

resource aws_iam_role this {
  assume_role_policy = file("${path.module}/iam/role.json")
  name               = join(var.delimiter, [var.name, var.stage, "public-lambda-role", var.random_string.id])
}

resource aws_iam_role_policy_attachment this {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

## Subscription to SQS queue

resource "aws_lambda_event_source_mapping" "example" {
  event_source_arn = var.aws_sqs_queue.arn
  function_name    = aws_lambda_function.this.arn
}

Service Endpoint

Here's the magic sauce! This Terraform resources connects a SQS Queue via an ENI into my VPC's private subnet. Now the VPC will be able to route the private Lambda's outbound HTTPS request to the SQS service. Even though the private Lambda has no apparent defined route to the public services.

resource aws_vpc_endpoint sqs {
  private_dns_enabled = true
  service_name        = join(".", ["com.amazonaws", var.region, "sqs"])
  vpc_endpoint_type   = "Interface"
  vpc_id              = aws_vpc.this.id

  security_group_ids = [
    aws_security_group.private_lambda_0.id
  ]

  # Interface types get this. It connects the Endpoint to a subnet
  subnet_ids = [
    aws_subnet.private_0.id
  ] 

  tags = merge(
    {
      Name = join(var.delimiter, [var.name, var.stage, "service-endpoint-for-sqs", random_string.this.id])
      Tech = "Service Endpoint"
      Srv  = "VPC"
    },
    var.tags
  )
}

resource aws_vpc_endpoint_subnet_association sqs_assoc {
  subnet_id       = aws_subnet.private_0.id
  vpc_endpoint_id = aws_vpc_endpoint.sqs.id
}

Demo / Proof

Executing the Private Lambda with a test payload. Watching the logs I can see the private Lambda executes successfully. Checking the public Lambda I also see the payload from the private Lambda. It works!

CloudWatch logs of the private Lambda invocation output. Notice the sqs.us-west-2.amazonaws.com:443 FQDN.

CloudWatch logs output after the public Lambda process the SQS queue message.

Conclusion

While it may seem a little weird at first Service Endpoints are a great way to attach supported AWS services into a VPC's private subnet(s). It's secure, fast, cheap, and best of all easy to manage.

Have you used Service Endpoints before? Do you have questions? Lets talk in the comments below.

Resources

Here is an the Example Terraform project on GiitHub.

How To: Database clustering with MariaDB and Galera.

David J Eddy — Wed, 27 Nov 2019 17:24:06 +0000

The Situation

MariaDB, a fork of MySQL, has had multi-master clustering support from the the initial version 10 release. However, the more recent releases have made it increasingly easy to setup a multi-master database cluster. By easy I mean it. But first, what is a multi-master cluster?

A multi-master cluster is one where each database instance is a master of course. The cluster contains no read-replicas, slave nodes, or 2nd class instances. Every instance is a master. The up side is no lag, the down side every instance has to confirm writes. So, the big caveat here is that the network and throughput between all the instances needs to be as good as possible. The cluster performance is limited by the slowest machine.

Preflight Requirements

Linux
CLI Terminal
(optional) Terraform 0.12.x
AWS account:
- API key and secret
- PEM key provisioned for EC2 access

(optional) I put together a small project that starts three EC2 instances. Feel free to use this to start up the example environment resources.

git clone https://github.com/davidjeddy/database_clustering_with_mariadb_and_galera.git
cd ./database_clustering_with_mariadb_and_galera

export AWS_ACCESS_KEY_ID=YOUR_API_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=YOUR_API_SECRET_KEY
export AWS_PEM_KEY_NAME=NAME_OF_YOUR_PEM_KEY

terraform init
terraform plan -out plan.out -var 'key_name='${AWS_PEM_KEY_NAME}
terraform apply plan.out

Once completed the output should look like this:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

db-a-key = maria_with_galera
db-a-ssh = ec2-3-84-95-153.compute-1.amazonaws.com
db-b-key = maria_with_galera
db-b-ssh = ec2-3-95-187-84.compute-1.amazonaws.com
db-c-key = maria_with_galera
db-c-ssh = ec2-54-89-180-243.compute-1.amazonaws.com

If that is what you get, we are ready to move on to the next part.

Photo by Taylor Vick on Unsplash

Setup

Now that we have three EC2 instances started up and running we can dig into the configuration for each database service. Open three new terminals; so in total we will have 4: localhost, DB-A, DB-B, DB-C. Using ssh log into the three database EC2 instances. After each login execution we should have something similar to the below.

ssh -i ~/.ssh/maria_with_galera.pem ubuntu@ec2-3-84-95-153.compute-1.amazonaws.com
The authenticity of host 'ec2-3-84-95-153.compute-1.amazonaws.com (3.84.95.153)' can't be established.
ECDSA key fingerprint is SHA256:rxmG0jtvI47tH3Yf3fAls9IsMPkho4DaRcSfA+NWNNs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-3-84-95-153.compute-1.amazonaws.com,3.84.95.153' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1054-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Nov 27 16:07:48 UTC 2019

  System load:  0.0               Processes:           85
  Usage of /:   13.6% of 7.69GB   Users logged in:     0
  Memory usage: 30%               IP address for eth0: 172.31.40.213
  Swap usage:   0%

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Take note of the IP address for eth0 on each instance. This is the private IP address that will be needed later. On each of the DB instances run the following commands to update the machine and install the MariaDB service and dependencies.

sudo apt-get update -y
sudo apt-get install -y mariadb-server rsync

The output this time is very long, but the ending should look like this.

...
Created symlink /etc/systemd/system/mysql.service → /lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service → /lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.
Setting up mariadb-server (1:10.1.43-0ubuntu0.18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.31) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...

To be extra sure we have everything installed, lets check the version of both MariaDB and rsync.

ubuntu@ip-172-31-40-213:~$ mysql --version && rsync --version
mysql  Ver 15.1 Distrib 10.1.43-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
rsync  version 3.1.2  protocol version 31

...

are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Since we need to configure the clustering go ahead and stop the MariaDB service on each instance using the standard stop command.

sudo systemctl stop mysql
sudo systemctl status mysql # never not always double check

You may have noticed that the command is mysql and not mariadb. This is because MariaDB is a fork of MySQL and the MariaDB team wants to keep binary compatibility with MySQL. This helps projects migrate with the least amount of headache.

Now do this same process on the DB-B and DB-C instances.

Photo by Charles 🇵🇭 on Unsplash

Configurations

Here is where the magic happens! We are going to create a new configuration file for each node at the location /etc/mysql/conf.d/galera.cnf. Open the file and add the following content. Where the configuration says [DB-A IP] replace with the PRIVATE IP address of that instance that we saw when we logged into each instance in the previous section. Also replace [DB-A NAME] with the name of the cluster node. DB-A, DB-B, or DB-C depending on what EC2 instance the file is located on.

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="test_cluster"
wsrep_cluster_address="gcomm://[DB-A IP],[DB-B IP],[DB-C IP]"

# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address="[DB-A IP]"
wsrep_node_name="[DB-A NAME]"

So, DB-A configuration should look like this:

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="test_cluster"
wsrep_cluster_address="gcomm://172.31.40.213,172.31.39.251,172.31.38.71"

# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address="172.31.40.213"
wsrep_node_name="DBA"

All three configurations should basically the same, minus the node_address and node_name being adjusted for each node.

Bringing It All Together

This is a very important step now; when starting the database on the first instance, aka DB-A, we have to bootstrap the cluster. Since no other instances are running the boot strap process tells the database hey, your the first one, chill out when it does not detect any other cluster members. After that though, DB-B and DB-C should join the cluster without an issue. So to start this first node use the following command on the DB-A instances.

ubuntu@ip-172-31-40-213:~$ sudo galera_new_cluster
ubuntu@ip-172-31-40-213:~$ sudo systemctl status mysql
● mariadb.service - MariaDB 10.1.43 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-27 16:32:03 UTC; 5s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
...
Nov 27 16:32:03 ip-172-31-40-213 /etc/mysql/debian-start[5129]: Checking for insecure root accounts.
Nov 27 16:32:03 ip-172-31-40-213 /etc/mysql/debian-start[5133]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

The important part here is the Active: active (running). Now, that we have the first cluster node running lets check the cluster status.

ubuntu@ip-172-31-40-213:~$ sudo mysql -u root -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster%';"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 1                                    |
| wsrep_cluster_size       | 1                                    |
| wsrep_cluster_state_uuid | 71780aba-1133-11ea-a814-beaa932daf25 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+

Hey; Check that out! We have a single instance cluster running. Awesome. Now we need to start DB-B and DB-C. Switch to each of those terminals and run the not bootstrapping command but instead the normal service start command.

ubuntu@ip-172-31-39-251:~$ sudo systemctl status mysql
● mariadb.service - MariaDB 10.1.43 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-27 16:36:45 UTC; 11s ago
...
Nov 27 16:36:45 ip-172-31-39-251 /etc/mysql/debian-start[15042]: Checking for insecure root accounts.
Nov 27 16:36:45 ip-172-31-39-251 /etc/mysql/debian-start[15046]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

Again the Active: active (running) is the important part. Switch back to DB-A and run the global status check command like we did after starting the DB-A services.

ubuntu@ip-172-31-40-213:~$ sudo mysql -u root -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster%';"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 3                                    |
| wsrep_cluster_size       | 3                                    |
| wsrep_cluster_state_uuid | 71780aba-1133-11ea-a814-beaa932daf25 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+

Yea buddy! A three node database cluster up and running!

Photo by Stephen Dawson on Unsplash

Confirmation

To be super sure everything is running and replicating as expected lets execute a few SQL commands to change the state of the database and then check the new state. On DB-A lets add a new schema and table with a data point.

sudo mysql -u root -e "CREATE DATABASE testing; CREATE TABLE testing.table1 (id int null);INSERT INTO testing.table1 SET id = 1;"

Now let's do a select statement on DB-C:

ubuntu@ip-172-31-38-71:~$ sudo mysql -u root -e "SELECT * FROM testing.table1;"
+------+
| id   |
+------+
|    1 |
+------+

YES! The new schema, table, and data replicated from DB-A to DB-C. We can run the select command on DB-B and see the same result! We can write to DB-C and see it replicated on DB-A and DB-B. Each node takes reads and writes then replicates the changes to all the other nodes!

Boom! A three node multi-master database cluster up and running! Log into one of the instances (does not matter since this is a multi-master) and create a new schema. Then exit and check the status of the cluster again. See the state value change? Yea, replication at work!

Conclusion

This is just the tip the functionality iceberg that is database clustering. I have had to skip over a very large number of topics like replication lag, placement geography, read-only replicas, bin_log format, and so much more. But this gives you a solid introduction to the concept of database clustering. Have fun!

Additional Resources

How to: Delete a stubborn Kubernetes namespaces.

David J Eddy — Wed, 25 Sep 2019 22:41:10 +0000

First posted on my blog.

The Situation

Recently I was introduced to a new project. The getting started documentation was decent and I was able to get the project started up. When it came time to clear the project out of my local machines Kubernetes cluster however, the same space for to the terminating life cycle phase and stayed there for days. Fast forward a couple of weeks and I notice it there, still terminating. Turns out the project had a finalizer that was not responding back. Think of remote service call. In this case the remote service never responded. So below is a little script I put together to force a namespace deletion.

The Code

#!/bin/bash
k8s_delete_ns=$1
echo "Provided namesapce: ${k8s_delete_ns}..."
echo "Exporting namespace configuration..."
kubectl get namespaces -o json | grep "${k8s_delete_ns}"
kubectl get namespace ${k8s_delete_ns} -o json > temp.json
echo "Opening editor..."
wait 3
vi temp.json
echo "Sending configuration to k8s master for processing..."
curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/${k8s_delete_ns}/finalize
echo "Waiting for namespace deletion to process..."
wait 12
kubectl get namespaces
echo "...done."

So what does this do? Lets remove the shell, echo and wait statements since we know the machine does not really do anything with those.

k8s_delete_ns=$1
kubectl get namespaces -o json | grep "${k8s_delete_ns}"
kubectl get namespace ${k8s_delete_ns} -o json > temp.json
vi temp.json
curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/${k8s_delete_ns}/finalize
kubectl get namespaces

So what does this do exactly?

Ok, so what do we have here. Line one takes the first argument from the command invocation and assigns it to a local variable. ./script.sh stuck_namespace. So stuck_namespace is the value that is assigned to k8s_delete.

The next two commands get's the current namespace configuration and writes it to temp.json.

The fourth command opens the temp.json configuration file using vi.

With the configuration open we want to remove any items listed in the finalizers array. This is the real magic moment here. Removing the finalizers with allow kubectl to remove the namespace.

Speaking of the next command uses curl to pass the newly edited configuration to kubectl. This updates the namespace configuration with the config that does not contain finalizers.

And finally, get namespaces will output the existing namespace after the update. If all went as expected the list will not contain the stuck namespace.

Conclusion

So there it is. There are a number of solutions out floating around the internet but the release cadence of Kubernetes renders some of then out-dated in a matter of months. So let me state this: this solution works for Kubernetes 1.13. Other than that, I hope you it helps you out as it helped me.

P.S.: If you would like to keep up with Kubernetes helpers I make, checkout the repo on GitHub.

Trial: 30 days with VueJs

David J Eddy — Wed, 14 Aug 2019 15:30:06 +0000

Given my roles over the last year plus writing code has be relegated to IaC, bash, or pipeline automation.s While nice that I still get to write logic the hunger to create something usable by people still nags in the back of my head. Given that the majority of the past decade or so has been server side; technologies like React, VueJS, Angular passed me by. Not that this is a problem, frontend never really interested me personally. Due mainly to the early 2000s when a dev had to write for IE AND Firefox, desperately. I hate repeating code just for one vendor.

As such, I have been listening / watching VueJS courses during my free time. As many of you know I am also studying for certifications, so the VueJS time has been limited. In the last 30 days I would ballpark 40 hours worth of effort has gone into studying VueJS.

If your interested, this is key usage of a keyboard.
Source http://chickart.de/portfolio-item/bachelorarbeit-visualisierung-der-tastaturnutzung-im-verlauf-der-bachelorarbeit/

Why VueJS? (the good)

VueJS has a number of points going for itself.

No JSX
Not owned by FAANG (Facebook, Amazon, Apple, Netflix, Google)
active and engaged community
Performance, small size, include only what is required in build
Minimal setup dev / prod environments
IE 11 not support, no added baggage for a dated and dead browser
Large UI library options

What, really!? (the bad)

While VueJS has it's good parts, nothing is perfect. One of the biggest pains out of the box is var data accessibility. Functions calling functions calling properties calling functions. Just to pass an atomic data to a sibling component. (To be fair many front end frameworks suffers this same access problem.)

Resources

Given the limited amount of time and attention I have been able to direct toward learning VueJS it was important for me to get the most bang for the attention minute buck. Here are some resources that really hit the spot.

Javascript, maybe not so bad...

Results

After maybe 40 hours of attention and effort some hands on practice and exposure to the community I think VueJS is worth looking at. It is flexible but not disorderly, powerful but not overwhelming complex, popular but not smothering. If can be included in nearly any standard web or native application as a part or whole, mobile native VueJS apps anyone? To round it out VueJS is performant and is on the adoption upswing.

Would I trade it for another option if the other option is in place and working? No, of course not. Would I pick VueJS for a new feature or project if given the chance? Yes, yes I would.

Intro: k3s, a less needy Kubernetes,

David J Eddy — Thu, 01 Aug 2019 13:08:39 +0000

What is all this?

The official documentation states "...k3s is a intended to be a fully compliant production-grade Kubernetes distribution with [the following] changes...". But what does that mean in layman's terms? It means Kubernetes has a lot of functionality that is not 100% required in the use case of IoT, Edge computing, or lower powered hardware. For example the kubeadm has some high hardware requirements. and runs very slowly on lower power / high latency hardware. Taking this further running k8s worker nodes on an ARM chip is a practice in frustration. Enter k3s, lower hardware is no longer a hard barrier for entry.

Up and Running

Installation is very straight forward. I would say almost as easy as apt-get install. Since k3s is aimed at Edge and IoT Debian is rarely the chosen OS. So instead it uses curl.

curl -sfL https://get.k3s.io | sh -

Execute the above in a terminal, the output should look similar to the following.

[INFO]  Finding latest release
[INFO]  Using v0.7.0 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v0.7.0/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v0.7.0/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kubectl
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Running ps aux | grep k3s returns our query with confirmation that k3s is indeed running.

Photo by Joey Kyber on Unsplash

Subsequent start ups of the master node is accomplished via an event shorter command.

echo 'Run the k3s server...'
sudo k3s server &
# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
sudo k3s kubectl get node

If the Rancher team was going for shortest commands possible, they win.

Now on to worker nodes. To join a worker node to a cluster it is also nearly as easy.

echo 'Run a k3s worker node...'
# On a worker node, run the following.
# NODE_TOKEN is on the server @ /var/lib/rancher/k3s/server/node-token
sudo k3s agent --server https://master_node:6443 --token ${NODE_TOKEN}

That is it. Three different commands to get a master and worker nodes installed, running, and joined together. Who said Kubernetes was confusing. (I am joking., k8s can be very confusing.)

Any Example Usage Case

As I wrote this posting an colleague of mine reached out to me asking how I would put together a system that could monitor the status of and receive data from a wide range of in-situ IoT devices for agriculture. The system needs to be able to tell what devices are online, the health of the system as well as receive data streams from the system. Once the data ingested and analyzed alerts and reports would be generated. The in-situ devices would be as lower-powered and very isolated, likely run off solar with a local battery pack. During the proof of concept phase Raspberry Pi 3's with custom enclosures would be the deployed IoT devices. Bam! Perfect case for k3s. When the device starts the startup script would instruct the device to join the cluster via a master node running on a ARM server (probably a AWS EKS master really). Then machine and sensor data could be feed into Kinesis Firehose to analysis. Bam, done.

Photo by Zan Ilic on Unsplash

Wrap Up

I can hear you saying 'so soon'? Well...yes. k3s really is that easy. One command to install, one command to start, one command to join. I would even dare to say it is easier to operate than Docker.

What do you think. Does Kubernetes have a place with Edge and IoT devices; or is it overkill like technology engineers so often tend to do? Let me know in the comments below.

Additional Reading

Intro: Hashicorp `Packer`

David J Eddy — Tue, 30 Jul 2019 23:28:34 +0000

The problem space...

In today's modern application development process an applications has many homes. Local development, (docker) containers, on premise Linux servers, maybe even Unix mainframes for production. Often an application is developed using a container then deployed to an cloud compute instance that is 'build like' the container. Then someone somewhere has to move the application to production and ensure it still works correctly. Boiling this problems down to the core we see it is a problem of creating machine images that match desired state. Docker container? machine image. On premise Linux integration server? machine image. Production cloud host? again, machine image. So exactly how are we suppose to create matching machine images over such a wide range of under lying systems?

Hashi-who pack-what-?

Hashicorp is a development and management tool publisher. Most famous there Infrastructure as Code (IaC) tool Terraform. Packer is one of the tools available from them. Created specifically to ease the creation of machine images, Packer is super easy to learn and has a very low bearer of entry. Container, Linux Machines and even Virtual Box and VMWare images can all be created with Packer. In one solid day and you can publishing machine images like a pro!

Image from Brett Jordan @ Unsplash: https://unsplash.com/@brett_jordan

Learning Curve

As stated above the learning curve for Packer takes one solid day. The packer help CLI command returns 6 options. 6! build, console, fix, inspect, validate, and version. Pretty self explanatory right?! Try that with any other CLI application and scrolling up and down becomes a way of life. Moving on to the configuration files, they are plan text JSON, not even HCL, just plain JSON. Inside the configuration files are what I like to call three 'top level concepts':

Builders: Who is building the machine image? Think of this similar to the build stage in a CI/CD pipeline; but much more versatile. AWS, Docker, GCP, 1&1, OpenStack, Oracle, VMWare, and even custom builds are available. Checkout the complete list to see over on the docs.
Provisioners: This part of the configuration determines what is used to install dependencies, update the core OS, create users, set file permissions, and other configuration processes INSIDE the image.
Post-Processors (optional): When these are run after the machine image is created additional commands can be executed. Upload the image for storage to an artifact repository, re-package a a VM from VMWare to VirtualBox, run build time reports, and other post event actions.

That is it; three main concepts, and one of them is optional!

Examples

Build a local Docker image and push to image repository.

{
    "builders": [{
        "commit": true,
        "image": "ubuntu:16.04",
        "type": "docker"
    }],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "apt-get update -y &amp;&amp; apt-get install -y python python-dev"
            ]
        }
    ],
    "post-processors": [
        {
            "repository": "example-ubuntu-16.04-updated",
            "tag": "latest",
            "type": "docker-tag"
        }
    ]
}

In the above Packer configuration Packer pulls the Ubuntu 16.04 Docker image from Docker Hub via the builders section. Followed by the shell provisioner to update the system and install the Python; now it is ready for you Flask or Django app! Finally, the last part, the post-processor adds a tag to the image and pushes to your image repository.

Photo by chuttersnap on Unsplash

Building an AWS AMI, Updating the OS, and saving the image as an AMI.

{
    "variables": {
      "aws_access_key": "",
      "aws_secret_key": ""
    },
    "builders": [{
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "us-east-1",
      "source_ami_filter": {
        "filters": {
          "virtualization-type": "hvm",
          "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",
          "root-device-type": "ebs"
        },
        "owners": ["099720109477"],
        "most_recent": true
      },
      "instance_type": "t2.micro",
      "ssh_username": "ubuntu",
      "ami_name": "packer-aws-ami-{{timestamp}}"
    }],
    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "apt-get install mysql-server libmysqlclient-dev"
        ]
    }]
  }

In the configuration above we use local AWS credentials to build an image based on Ubuntu 16.04. In the improvisers section the OS is updated and MySQL-server is installed using the shell provisioner. That is all. You now have a EC2 based MySQL server image.

This is just two simple examples of what Packer can do. Imagine it as part of your CI/CD pipeline! It is even possible to build different images for different targets with the same provisioner execution on each, at the same time! Parallel builds are an amazing advanced feature to look into as you dig into the full feature set.

Conclusion

All told Packer fits a nice niche in the build process: creating the underlying machine image. From there a provisioner like Shell scripts, Ansible, or PowerShell pick up and execute custom application specific commands. A fast, easy to understand, amazingly simple (and REPEATABLE) way to configure those sweet sweet golden images. Now there is no reason for base images to be out of date an unpatched.

So what do you think? Can we see what Packer fits into your daily build cycle, or even simplify an effort intensive process of creating images? Let me know your thoughts in the comments below.

Further Reading

What do you prefer: general all-in-one tools or focused single domain tools?

David J Eddy — Sun, 28 Jul 2019 15:54:54 +0000

While refactoring an application recently I was thinking about its build pipelines and the tools available to modern app dev teams. Docker was heralded as the world savor (like every new tech is, but that is a different conversation); however the reality is multi GB images, version sprawl, and a lack luster meta-data system, and a security model that has been the bane of security teams for the past 5 years. It leaves a person longer for the simpler times.

It got me thinking, is containerizing the applications run time worth it? If we applying the software concept of single responsibility to the build process; each step would have a tool best suited for one domain of responsibility. The counter argument being "yet another tools to learn". But, honestly, everything is YAML configuration these days so how hard would it really be.

So the question: Do you prefer small, separate, specific tooling or one tool that does it all to an acceptable level? What is the justification for you choice?

Cover image by https://unsplash.com/@soymeraki

Whats your daily tool chain these days?

David J Eddy — Mon, 15 Jul 2019 13:24:11 +0000

What are the tools you use daily today? Be as detailed or specific as you want.

Ansible
AWS
Containerization (Docker mainly)
GCP
Kubernetes
Linux (Debian / Ubuntu) & related CLI tools
Terraform
VS Code
Web browser (Chrome, Firefox)

Tagging with Terraform

David J Eddy — Tue, 25 Jun 2019 12:08:04 +0000

Sourced from my blog.

Using cloud resources can be accelerating to the business, liberating to the engineering teams, and expensive to bank account. If you have every left a large compute or database instance running over a weekend (or accidentally committed API keys to GitHub) it is easy to experience a large increase in operating costs for the month. Using the resource metadata tagging functionality responsible parties(1) can audit, track, and manage resources. It is even possible to enact automatic based on tag values (or lack thereof).

Basic Usage

The basic tagging structure in Terraform is straightforward enough. If you are familiar with the JSON data interchange format you should recognize this immediately. It is a Java inspired object declaration with quoted key values pairs using a colon as separator and a comma as a delimiter. Yeay, another DSL to learn. #welcometowebdev

...
"aws_resource_type" "aws_resource" {
    ...
    tags {
        "key": "value",
        "Name": "Value",
        "department": "engineering",
        "team": "core_api",
        "app": "name",
        "env": "dev"
        ...
    }
    ...
}

Nothing to difficult about that. Most (not all) resources in AWS support tagging in this manner. Straight forward JSON styled key/value pairs. Terraform even allows us to use variables in place of values, but not keys(2).

Weird..

...
"aws_resource_type" "aws_resource" {
    ...
    tags {
        "key": "value",
        "Name": "Value",
        "department": "${var.dept_name",
        "team": "${var.team_name",
        "app": "${var.app_name",
        "env": "${var.app_env}"
        ...
    }
    ...
}

"But David" you will say "application infrastructure can be very complicated. Will I have to copy / paste the 'tag' attribute all over the place?" Short answer; No. Long Answer: ...

Advanced Usage

Using some Terraform-fu we can assign the default sets of key/value pairs to a map type variable (local or imported) and that variable as the default tags data set. The format changes slightly between a JSON object to a Terraform (HCL) map data type; but not by much.

variable "default_tags" { 
    type = "map" 
    default = { 
        key: "value",
        Name: "Value",
        department: "${var.dept_name",
        team: "${var.team_name",
        app: "${var.app_name",
        env: "${var.app_env}"
  } 
}
...

With the tags abstracted as a map the tag attribute for the resource is linimized to a one liner.

Electrical circuits or IaC diagram? You decide!

...
"aws_resource_type" "aws_resource" {
    ...
    tags = var.default_tags
    ...
}
...

The power level up is to merging the default tags map variable with custom inline tags. We for this we get fancy and start using the TF merge(). Providing the default tag map variable as one function argument and the custom tags as a second map type function argument we get to use both the default provided tags AND custom inline tags! Boom, Magic!

...
"aws_resource_type" "aws_resource" {
    ...
    tags = "${merge(map( 
            "Special_Key", "some special value", 
            "Special_Key_2", "some other special value",
            ...
        ), var.default_tags)}"
    ...
}
...

Wrap Up

Hashicorp continues to improve Terraform with each release. Say what you will about HCL being nearly JSON but with additional functionality; it is a powerful and featurful tool set to manage a projects infrastructure. With an appropriate tagging strategy it also becomes a powerful way to track that infrastructure as well.

Notes

I did not say "managers" on purpose. In a true DevOps environment every developer, operator, SysAdmin, on every team within every technology value stream is should be aware of how and responsible for the system as a whole. Share the burden. No silos.
Using map() it is possible to use a variable as a key. This is because map() evaluated the variable "key" before returning.
This concept weirded me out at out first. But after thinking about the mechanics it makes sense. Why would the last operation in a function be on a variable NOT being returned?

Additional Reading

10 years of web development; 10 life lessons.

David J Eddy — Fri, 14 Jun 2019 14:37:52 +0000

Soured from my blog

Ten years ago (2009) the economic recession was in full swing. Every week another bank would collapse, millions of homes would go into foreclosure proceedings, and I started my career as a professional web developer. With a degree in hand and a little hobby experience, I set out into the cruel, cruel job market.

Anyone know what this is?

It took nearly 6 months to find my first role as a 'jr web developer'. Later finding out I was the only candidate that could create a working submit form. Yea, 4 year degree to show off something I knew _before_ university. Soon, I'll be a Infra-Engineer for a payment processor after spending a couple years in the SRE/DevOps consulting realm. Here are the 10 lessons I have learned over the last 10 years the in information technology.

Knowledge is a resource best shared.
The end user is always correct; conversely, the end user is never correct.
The end user will always find a way to use your program differently than intended.
If you know more about a subject than everyone else in the room, you are the expert.
You are your own best champion. No one knows your strengths and weaknesses better than you. Put yourself where you want to be.
Learn something new every day; the results are compounding.
Break complex systems into small single focus pieces. Systems are easier to understand, manipulate, and iterate as small pieces.
Ignore the imposter syndrome feelings, no one knows everything about everything.
Everything is an abstraction. Understand the low level means you automatically understand 1/2 the higher level abstractions.

So many things, so complicated.

So there you go. Successes, failures, hard won battles, and easily lost fights. Distilled down to 10 learned lessons. So how about you, any lessons you have learned in the past decade you can share?

Photo by Matthew Fournier on Unsplash.

FIX: Terraform + AWS: InvalidVPCNetworkStateFault

David J Eddy — Sun, 02 Jun 2019 15:30:00 +0000

While working with Terraform and AWS recently I ran into an error that did not seem to have much information about it. After about a day of research and troubleshooting I was able to solve it.

The Error

Error: Error applying plan:

1 error(s) occurred:

* module.web_app.aws_db_instance.rds: 1 error(s) occurred:

* aws_db_instance.rds: Error creating DB Instance: InvalidVPCNetworkStateFault: Cannot create a db.t2.micro database instance because no subnets exist in availability zones with sufficient capacity for VPC and storage type : gp2 for db.t2.micro. Please first create at least one new subnet; choose from these availability zones: us-west-1c, us-west-1b.

    status code: 400, request id: ea5f04be-8510-4cfc-9bb2-606c0e00d007

The key takeaways here are RDS, subnets, and availability. So I checked the VPC AZ's, the subnets assigned to them, CIDR ranges, etc. At one point I even compared the VPC configuration to a working zone. From what I could tell no differences existed.

The Cause

After some digging around I noticed the default VPC's subnets had been deleted. This causes the VPC and associated AZ subnets to be invalid in the default DB security group. The only way to recreate default subnets in a region is via the CLI, no web console ability for this action.

The Fix

The fix was to go into RDS subnet group configuration (https://us-west-1.console.aws.amazon.com/rds/home?region=us-west-1#db-subnet-groups) and re-assign the new two new default subnets to the RDS group. After that Terraform 'plan' and 'apply' returned to working as expected.