DEV Community: Ben Curtis

Bootstrap a consul cluster in AWS

Ben Curtis — Mon, 01 Nov 2021 17:39:28 +0000

I recently needed to stand up a test consul cluster, and I thought it might be useful to share my notes. Here's what you can do to quickly get a consul cluster going...

First, create an IAM role named consul and attach the AmazonEC2ReadOnlyAccess policy to the role.

Next, boot three Ubuntu instances, assigning the just-created role to them as the IAM role, and the following user data:

#!/bin/bash

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install consul

echo "datacenter = \"$(ec2metadata --public-hostname | cut -d. -f2)\"" >> /etc/consul.d/consul.hcl 
echo 'server = true' >> /etc/consul.d/consul.hcl
echo 'bootstrap_expect = 3' >> /etc/consul.d/consul.hcl
echo 'retry_join = ["provider=aws tag_key=Role tag_value=consul"]' >> /etc/consul.d/consul.hcl

systemctl start consul

This will install the latest consul on each of the instances, configure consul to operate in server mode, and bootstrap the cluster.

You also want to add the Role tag with the value of consul to the instances. This is how the instances will find each other to form a cluster.

After you boot the instances, edit the security group to allow inbound traffic from the security group (so the instances can connect to each other).

Once those steps are done, you can confirm the cluster is working by connecting to any of the instances and running consul members. You should see the three instances listed as servers.

Now you can use your consul cluster by pointing consul clients at the private IP of any of the instances.

Simple (and naïve) way to mark private URLs as invalid

Ben Curtis — Sat, 16 May 2020 18:19:38 +0000

I'm working on a new side project, and I needed some code to check that a user-supplied URL won't be accepted if it's a known private address... here's a gist with some quick code I wrote.

Managing PostgreSQL partitioned tables with Ruby

Ben Curtis — Thu, 31 Oct 2019 00:09:31 +0000

Introducing the pg_partition_manager gem: It helps you easily maintain PostgreSQL partitioned tables that need to be created and dropped over time as you add and expire time-based data in your application.

We use partitioned tables at Honeybadger in our primary PostgreSQL database to efficiently expire old data, as deleting a bunch of data from a huge table can cause database performance to suffer. Prior to version 10, PostgreSQL didn't have native support for partitioned tables, so we used the pg_partman extension to implement partitioning. It works by using PostgreSQL's table inheritance to create child tables of the table that is to be partitioned and triggers to insert data into the child tables rather than the parent table. That extension has worked well for us, but it has a downside -- it isn't an option when you are using Amazon RDS, as it isn't supported. Now that that PostgreSQL has support for native partitions, I figured now was the time to see about dropping that extension so we'd have the option of using RDS.

Our partitioning use-case is fairly simple: we partition tables based on time, creating a new partition for every day, week, or month, depending on how many rows we want to store across all partitions and on how long we want to retain the data. All of our partitioned tables have a created_at column that will be used to determine which partition stores each row. For example, we might have a table defined like this:

create table events (
  project_id integer,
  data jsonb,
  created_at timestamp
)
partition by range (created_at);

And if we wanted to have weekly partitions, they would look like this:

create table events_p2019_10_28 partition of events for values from ('2019-10-28') to ('2019-11-04');
create table events_p2019_11_04 partition of events for values from ('2019-11-04') to ('2019-11-11');

With a time-based partitioning scheme, deleting old data is as simple as dropping one of the partitions. The regular maintenance we need to do, then, is create new partitions for date ranges as we approach them and delete old partitions containing data we no longer want. To make that maintenance a little easier, I have created the pg_partition_manager gem. Naturally, it's inspired by my experience with the pg_partman extension, which has served us so well.

Let's take a look at how you'd use this gem, given the events table and partitioning scheme described above. You would create a script or rake task that looks like this:

require "pg_partition_manager"

PgPartitionManager::Time.process([{parent_table: "public.events", period: "week", premake: 1, retain: 3}])

The parent_table is defined as schema.table_name (public, the default schema, is often the only one Rails developers end up using). The period can be day, week, or month. You can choose how many tables you want created in advance (after the current period) with premake, and how many tables you want to keep (prior to the current period) with retain. The gem defaults to pre-creating 4 tables if you don't specify premake, and it defaults to retaining data for 7 days, 4 weeks, and 6 months if you don't specify retain.

Invoke that script/task with a daily cron job, and you're all set -- it will create and drop the tables as needed.

All ActiveRecord queries act just as they would with a non-partitioned table, so there's nothing you need to change in your code. That is, Event.create, Event.where, etc., will work just as they always have, with PostgreSQL putting the data in the right partition for you when you insert it. There is one change you may notice if you have lots of data, though... when you include created_at in your queries, PostgreSQL won't have to scan all the partitions -- just the ones the cover the range you specify in your where clause.

To recap, if you have a lot of time-based data that you want to delete as it expires, use PostgreSQL partitioned tables and the pg_partition_manager gem to make your app happy. :)

Configure Your App with SSM Parameter Store

Ben Curtis — Wed, 09 Oct 2019 16:13:33 +0000

Configuring your Rails app via environment variables works well, but sometimes you want to be able to update your configuration on the fly. Here's a way to update your app's environment using SSM Parameter Store.

Why would you want to do this? Well, say you deploy your Rails app to an EC2 instance that's part of an autoscaling group. To get the fastest boot times, you should create a custom AMI with your code already on it (a.k.a. a golden image) that the autoscaling group can use when it's time to boot a new instance. Unfortunately, if you store your configuration on the image (and use something like dotenv to load it), you'll need to create a new AMI every time you have a configuration change. You can work around this by using SSM Parameter Store parameters, and let your app fetch its configuration at boot time.

Putting data into Parameter Store is easy enough -- you can use the CLI or the AWS console to edit the variables. But how do you get the data back out for your app to use? One way to do it is to fetch these parameters into your ENV via an initializer, like so:

Aws::SSM::Client.new.get_parameters_by_path(path: "/honeybadger/#{Rails.env}/", recursive: true, with_decryption: true).parameters.each do |param|
  ENV[param["name"].split("/").last] = param["value"]
end

Assuming you have parameters named like honeybadger/production/MY_API_KEY, this snippet will result in ENV["MY_API_KEY"] having whatever value you supplied for that parameter.

But what if loading the environment variable values in an initializer is too late in the Rails boot up process? What if you need settings like DATABASE_URL to be set in SSM and you need to use those settings before your app loads? For that, you can save the variables to a .env file and let dotenv handle that. But first, let's set up the database and store our DATABASE_URL value.

Here's a terraform snippet that creates an RDS instance and stores the connection in SSM Parameter Store:

resource "aws_db_instance" "db" {
  allocated_storage      = 20
  storage_type           = "gp2"
  engine                 = "postgres"
  engine_version         = "11.4"
  password               = "${var.database_password}"
  name                   = "honeybadger"
  username               = "honeybadger"
}

resource "aws_ssm_parameter" "database_url" {
  name  = "/honeybadger/${var.environment}/DATABASE_URL"
  type  = "SecureString"
  value = "postgres://${aws_db_instance.db.username}:${var.database_password}@${aws_db_instance.db.endpoint}/${aws_db_instance.db.name}"
}

With the following shell command, you can grab all the parameters (just like we did with the Ruby snippet above) and get them ready for use as environment variables:

aws ssm get-parameters-by-path --path /honeybadger/production/ \
  --recursive --with-decryption --output text \
  --query "Parameters[].[Name,Value]" |
  sed -E 's#/honeybadger/production/([^[:space:]]*)[[:space:]]*#export \1=#' \
  > /home/honeybadger/shared/.env.production.local

Now you have a .env.production.local file that dotenv can load -- assuming that it's symlinked into your current path at deploy time, if you are using capistrano. As a bonus, you can also source that env file to have variables like $DATABASE_URL defined for you in any shell scripts you want to run.

We put that shell command in our deployment script (which is triggered by our CI/CD pipeline), so any new code that goes to production will pick up any changes to made to our parameters in SSM. Now we get to have our golden image and we don't have to build a new one for every configuration change. 😎