DEV Community: Hint

Deploy Infrastructure With CDKTF

Nate Vick — Tue, 07 Sep 2021 20:19:54 +0000

CDK or Cloud Development Kit came out of AWS in 2018 as a way to write Infrastructure as Code in software languages used day-to-day by developers (JavaScript/TypeScript, Python, Java, C#, and soon Go). Since its release, a community has built up around it, and new flavors have arrived. AWS CDK, CDKTF, and CDK8s are all based on the same core, but compile to different formats. AWS CDK compiles to CloudFormation, while CDKTF compiles to Terraform compatible JSON, and CDK8s compiles to Kubernetes config.

In this post, we will walk through how to use CDKTF with DigitalOcean's Terraform provider. A Terraform provider is a "plugin" for Terraform to interact with remote systems. In this case, the provider is created and maintained by DigitalOcean. We will share a few examples of creating a Digital Ocean Project, a VPC, a Postgres Database, and a Droplet.

Prereqs and Install

To use cdktf you will need the following packages installed:

Terraform ≥ 0.12
Node.js ≥ 12.16
Yarn ≥ 1.21

To install cdktf, we will use npm install

$ npm install --global cdktf-cli

You will also need to create a personal access token on Digital Ocean if you would like to deploy this config.

Create and Initialize the example project

First let's make a directory and change into it.

$ mkdir cdk-do-example && cd cdk-do-example

Initialize the project with the init command. In general, I use the --local flag, so all state is stored locally.

$ cdktf init --template=typescript --local

You will be prompted for Project Name and Description but defaults are fine.

The last step of setting up the project is to add Terraform providers to the cdktf.json. Open the project in your editor and let's add the DigitalOcean provider to it as follows.

{
  "language": "typescript",
  "app": "npm run --silent compile && node main.js",
  "terraformProviders": [
    "digitalocean/digitalocean@~> 2.9"
  ],
  "terraformModules": [],
  "context": {
    "excludeStackIdFromLogicalIds": "true",
    "allowSepCharsInLogicalIds": "true"
  }
}

Install Dependencies

Run cdktf get to download the dependencies for using DigitalOcean with Typescript.

$ cdktf get
Generated typescript constructs in the output directory: .gen

Let's Write Some TypeScript

Open main.ts in your editor and let's start by creating a DO Project.

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })

    new Project(this, 'example-project', {
      name: 'Example Rails Project'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

You will create a new DigitaloceanProvider and pass in the environment variable assigned to your DO personal access token. Next, we create the Project, which has a required key of name.

Optionally, as a test of the code above, run cdktf deploy to deploy your CDKTF project. The cli will ask if you want to make the changes listed under Resources. Type 'yes', then once it finishes, you will see a green check next to the Resources it successfully created.

cli prompt:

successful deployment:

Now we will add a VPC, a Postgres Database, and a Droplet. Once those examples are in, we will tie it all together before deploying it again.

Create a VPC

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    new Vpc(this, 'example-vpc', {
      name: 'example-vpc',
      region: 'sfo3'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

*Note: If you do not have VPC in your account already this will become the default VPC which will not be deleted when cleaning up the CDKTF Project.

Create a Postgres Database

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc, DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    const postgres = new DatabaseCluster(this, 'example-postgres', {
      name: 'example-postgres',
      engine: 'pg',
      version: '13',
      size: 'db-s-1vcpu-1gb',
      region: 'sfo3',
      nodeCount: 1
    })

    new DatabaseUser(this, 'example-postgres-user', {
      clusterId: `${postgres.id}`,
      name: 'example'
    })

    new DatabaseDb(this, 'example-postgres-db', {
      clusterId: `${postgres.id}`,
      name: 'example-db'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

Notice we assigned the DatabaseCluster to a variable const postgres. We then use the variable to create the DatabaseUser and DatabaseDb on that cluster.

Create a Droplet

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc, DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    new Droplet(this, 'example-droplet', {
      name: 'example-droplet',
      size: 's-1vcpu-1gb',
      region: 'sfo3',
      image: 'ubuntu-20-04-x64'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

Let's Put It All Together

If you run cdktf deploy now, it would create everything, but nothing created would be put into the Digital Ocean project or the VPC we create. Let's do that now.

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc} from './.gen/providers/digitalocean'
import { DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project, ProjectResources } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })

    const project = new Project(this, 'example-project', {
      name: 'Example Rails Project'
    })

    const vpc = new Vpc(this, 'example-vpc', {
      name: 'example-vpc',
      region: 'sfo3'
    })

    const postgres = new DatabaseCluster(this, 'example-postgres', {
      name: 'example-postgres',
      engine: 'pg',
      version: '13',
      size: 'db-s-1vcpu-1gb',
      region: 'sfo3',
      nodeCount: 1,
      privateNetworkUuid: vpc.id
    })

    new DatabaseUser(this, 'example-postgres-user', {
      clusterId: `${postgres.id}`,
      name: 'example'
    })

    new DatabaseDb(this, 'example-postgres-db', {
      clusterId: `${postgres.id}`,
      name: 'example-db'
    })

    const droplet = new Droplet(this, 'example-droplet', {
      name: 'example-droplet',
      size: 's-1vcpu-1gb',
      region: 'sfo3',
      image: 'ubuntu-20-04-x64',
      vpcUuid: vpc.id
    })

    new ProjectResources(this, 'example-project-resources', {
      project: project.id,
      resources: [
        postgres.urn,
        droplet.urn
      ],
      dependsOn: [ postgres, droplet ]
    })
  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

We start by assigning the project, VPC, and droplet to variables. In the DatabaseCluster definition, we add privateNetworkUuid: [vpc.id](http://vpc.id) to place the database in our newly created VPC. Similarly, on the Droplet definition, we place it in the VPC, by adding vpcUuid: vpc.id.

Lastly, create a new ProjectResource to assign other resources to the Digital Ocean project. In this small example, we will assign the database and droplet to the project using the urn for each resource. We will wait to assign those until both are created using a Terraform helper dependsOn.

With all of that in place you can deploy again. The database and droplet creation take a bit, so be patient. 🙂 Once it has finished, check your Digital Ocean Dashboard to see everything created and ready for use.

*Make sure you run cdktf destroy to remove these resources from your account or you will be charged by Digital Ocean.*

Things To Know

CDKTF is still a young project, so it is changing fast, has limited documentation, and you can run into unclear errors.

There are few things that help with the limited documentation. You can read the provider documentation on the Terraform registry site and use it as a guide. Also, using a language like Typescript with VSCode there are a lot of code hints, hover info, and signature information. The gif below is an example of what is shown in VSCode when you hover on the problem. Note the missing properties for DropletConfig.

When you run into unclear errors, prepend CDKTF_LOG_LEVEL=debug to the deploy and/or destroy commands to get very verbose output.

Wrapping Up

In this post, we used CDKTF to create some basic example resources on Digital Ocean which gives you a good primer to build more complex infrastructure in a language of your choice. You can find the code from this post in this repo. If you would like to chat more about CDK or infrastructure as code you can ping me on Twitter @natron99.

Active Record Encryption in Rails 7

Jason Dinsmore — Wed, 07 Jul 2021 04:47:04 +0000

Rails 7 will be introducing a very cool new feature for ActiveRecord - application-level encryption, summoned by the mighty encrypts declaration in a model. This new feature provides a layer of encryption that sits between our application code and the database. In essence, when our data using Active Record Encryption has been loaded into an AR object, it will be unencrypted, and when it is sitting in the database it will be encrypted.

In this post, we will take a high-level look at how to use this functionality, discuss some cool things it can do, and acknowledge some of its limitations.

Before we dive in, I would be remiss if I didn't point to the excellent documentation in the Rails guide.

For the duration of the post, I will refer to the feature as encrypts for brevity (and also because I am not sure what else to call it 😉).

Backstory

encrypts was merged into Rails in PR 41659 by @jorgemanrubia. In the PR description he states that the functionality is an extraction from HEY, which had the implementation reviewed by a security firm. If you are interested in hearing more of the story behind the feature, I'd recommend Jorge's blog post on the subject - it's an interesting read.

Deterministic Sidebar

Let's take a quick second to discuss the difference between deterministic and non-deterministic encryption. It's really pretty simple, but is a central tenet in understanding how to use encrypts.

Think of encryption as a function that is applied to some input (text), resulting in some output (encrypted text).

If the function is deterministic, then any time the function is applied to the same text, it will output the same result.

If the encryption function is non-deterministic, then we can't predict what the output will be the second time we encrypt a given value. In theory, it is possible we would get the same result we got the first time, but the odds of that happening are very, very low. In the context of encrypts with its default non-deterministic configurations, subsequent encryptions of the same plaintext are almost certain to produce different ciphertexts.

If we are using deterministic encryption with encrypts for a model attribute, then any two rows in the database that would have the same plaintext value will also have the same stored encrypted value. If we using non-deterministic encryption, then two rows that have the same plaintext value will generally have different encrypted values. As we'll see in a bit, this has implications as to whether we can query the encrypted data.

Setup

There's not a ton of configuration required to get going with encrypts, but there are some things to be aware of.

Keys

The main requisite is that we will need to generate a set of keys and add them to our credentials file(s). We can generate the keys to add by running: bin/rails db:encryption:init, which will output something like:

Add this entry to the credentials of the target environment:

active_record_encryption:
  primary_key: zxMXS0hBbpa5BzRKPv9HOSF9etBySiHQ
  deterministic_key: 0pM2UHHBQr1kf1irO6JgakcSOXu0r1Vn
  key_derivation_salt: I5AkViD0UJhSqK3NY49Zvsls3ZoifyXx

The primary key is used to derive the root encryption key for non-deterministic encryption. Note that the primary_key value in the credentials file can also be a list of keys.

The deterministic_key is used for deterministic encryption. If you recall from the section on determinism above, we'll get the same result if we encrypt the same data with this key multiple times. Currently, encrypts does not support using a list of keys for deterministic encryption. If we want to completely disable deterministic encryption, not providing the key is a sure-fire off switch.

The key_derivation_salt is used to derive encryption keys.

App Configuration

The encrypts API provides several configuration options. All of the options are defined in the config.active_record.encryption namespace and I'd encourage reading through them if you are going to use this feature. I believe you'll find that most of the options have pretty reasonable defaults.

I will mention the config.active_record.encryption.extend_queries option as it defaults to false, but enabling it has several implications:

enables querying unencrypted data in an encrypted column (also need to enable config.active_record.encryption.support_unencrypted_data for this)
allows supporting multiple encryption schemes
enables support for uniqueness validations

Database

When an encrypted string or text attribute is stored in the database, it isn't stored as an ordinary string or text - it is stored as a more complex data structure that is serialized when written and deserialized when read. This data structure allows some meta information to be stored along with the encrypted text, which gives the app some clues about how the text was encrypted.

This extra meta-info introduces some storage overhead - up to 250 bytes.

The guide recommends increasing the size of a 255 byte string field to 512 bytes if using encryption on a column. It also says that for a text field, the overhead is generally negligible.

Invocation

Finally, we can talk about actually using the thing!

In the most basic use case, to encrypt a single column, we simply add an encrypts declaration to our model for the attribute we want to encrypt. For example, if we had a Dog model with a toy_location field (dogs like to hide their toys, you know) that needs encryption, our model would look like:

  class Dog < ApplicationRecord
    encrypts :toy_location

Pretty simple, eh?

Writing

Writing an encrypted attribute is completely transparent. We just do what we would normally do in Rails:

> dog = Dog.create!(name: 'Bruno', toy_location: 'top secret')

If we were to look at the content sitting in the database directly, we would see something like:

> result = Dog.connection.execute('SELECT toy_location FROM dogs LIMIT 1').first
   (1.4ms)  SELECT toy_location FROM dogs LIMIT 1
=> {"toy_location"=>"{\"p\":\"oVgEJvRaX6DJvA==\",\"h\":{\"iv\":\"WYypcKysgBY05Tum\",\"at\":\"OaBswq+wyriuRQO8yCVD3w==\"}}"}

The value here is just serialized JSON, let's go ahead and parse it:

> JSON.parse(result['toy_location'])
=> {"p"=>"oVgEJvRaX6DJvA==", "h"=>{"iv"=>"WYypcKysgBY05Tum", "at"=>"OaBswq+wyriuRQO8yCVD3w=="}}

That gave us a Hash. Most of the keys in this Hash are defined in the ActiveRecord::Encryption::Properties::DEFAULT_PROPERTIES constant. p is the payload, aka the encrypted plaintext. h is a Hash of headers that contain information relating to the encryption operation. Here, iv is the initialization vector the plaintext was encrypted with - more about this in the next section on searching, and at is an auth_tag that will be used during the decryption process to verify that the encrypted text hasn't been altered. You may notice other headers from the DEFAULT_PROPERTIES Hash above depending on how your encryption is set up and being used.

Reading

When we load a model with an encrypted attribute, Rails will seamlessly decrypt the encrypted value. Let's find the Dog we created above by his name:

> Dog.find_by!(name: 'Bruno').toy_location
=> <Dog id: 1, name: "Bruno", toy_location: "top secret", created_at: "2021-05-28 22:41:23.142635000 +0000", updated_at: "2021-05-28 22:41:23.142635000 +0000">

As you can see, the encrypted value was automatically translated to a readable attribute on our model instance - pretty slick.

Searching

What if we wanted to find Bruno by his toy_location instead of his name? We can do that just like we would if the field were not encrypted:

> dog = Dog.find_by!(toy_location: 'top secret')
  Dog Load (2.1ms)  SELECT "dogs".* FROM "dogs" WHERE "dogs"."toy_location" = ? LIMIT ?  [["toy_location", "{\"p\":\"oVgEJvRaX6DJvA==\",\"h\":{\"iv\":\"WYypcKysgBY05Tum\",\"at\":\"OaBswq+wyriuRQO8yCVD3w==\"}}"], ["LIMIT", 1]]
=> #<Dog id: 1, name: "Bruno", toy_location: "top secret", created_at: "2021-05-28 22:41:23.142635000 +0000", updated_at: "2021-05-28 22:41:23.142635000 +0000">

Notice that our query string was automatically converted into the encrypted JSON string we saw when we looked in the database.

Initialization Vector/Determinism

When using deterministic encryption, all records with the same plaintext value will use the same initialization vector to encrypt. This is so ActiveRecord will generate the same ciphertext for the same input, which is a prerequisite for being able to search the encrypted data. Under the hood, Rails uses the plaintext to generate the initialization vector for deterministically encrypted data - otherwise the IV is randomly generated.

If two rows with the same plaintext were to use different initialization vectors to perform the encryption, the serialized JSON that ends up in the database would be completely different.

In order to be able to perform searches on the encrypted data, the stored values need to be exactly the same.

This means that all of the stored values in the serialized hash need to be identical for two rows that have the same text value, AND that Rails can re-compute the exact same hash on the fly to find rows that are matches for the search string.

Determinism at its finest.

Searching Plaintext

What if we did not have the luxury of starting with pristine data? For example, if our Dog table already existed and had a pre-existing toy_location column on it that was not encrypted?

Well, as we can see by the query generated above, if we had a Dog record with top secret (unencrypted) as its toy_location, that query aint gonna find it. Also, it seems pretty likely that if we try to load a Dog record with stored plaintext into memory, Rails is going to have problems when it attempts to decrypt the plaintext.

One option we have would be to convert our plaintext data to encrypted data, which seems ideal to me. We may have reasons to want to avoid doing a data migration like that, however.

In that case, encrypts will allow us to keep storing the plaintext values as plaintext, and will encrypt any new or updated data. To opt-in to supporting a mix of encrypted/unencrypted data, enable the config.active_record.encryption.support_unencrypted_data configuration option.

Enabling this behavior will prevent the errors we would get when it tries to decrypt the plaintext and will also allow us to perform searches when a column contains a mismatch of plaintext and encrypted data.

If we enable the setting and re-run our search query above, we'll see:

Dog Load (0.3ms)  SELECT "dogs".* FROM "dogs" WHERE "dogs"."toy_location" IN (?, ?) LIMIT ?  [["toy_location", "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}"], ["toy_location", "top secret"], ["LIMIT", 1]]

Now, it is looking for records that have the encrypted content or have the plaintext version of that content. Perfect!

Case Insensitive Searches

By default, searching is case sensitive. If we need to ignore case when searching for some reason, we have a few options:

Option 1:

We can query for all of the case variations we need to match - eg. Dog.where(toy_location: ['Top secret', 'top secret']).

Option 2:

We can specify downcase: true on our encrypts declaration. This will cause the text to be downcased before it is stored. ActiveRecord will automatically downcase our search text when performing queries. The downside here is that all case information is lost when it is downcased. Sorry to be a downer.

Option 3:

We can specify ignore_case: true on the encrypts declaration and add an original_column_name column to our database (eg. original_toy_location). With this in place, if we created a dog with an uppercase letter in the encrypted field:

Dog.create!(name: 'Max', toy_location: 'Top secret')

the toy_location column would be populated with the encrypted form of 'top secret' (the value downcased), and the original_toy_location column will have the encrypted form of 'Top secret' (the value we set).

Any searches would be done against the toy_location column, and the model's toy_location attribute would be populated from the original_toy_location column when it is loaded into memory.

One thing to note here - while the toy_location column is encrypted deterministically in this situation (so it can be searched), the original_toy_location column appears to be encrypted non-deterministically. This makes sense, since that column does not need to support searching. This can be confirmed by comparing the toy_location and original_toy_location values for two records with the same plaintext value. As you can see below, they have the same stored values (initialization vector, payload, etc) for the toy_location column (searchable and downcased), and different stored values for the original_toy_location column (not searchable, case preserved):

{ "toy_location"          => "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}",
  "original_toy_location" => "{\"p\":\"5syLqDK6GCbBDw==\",\"h\":{\"iv\":\"KBGp4FrI7oL4/a3p\",\"at\":\"JnH6hxLX35cAwroImk2XqQ==\"}}" },
  "toy_location"          => "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}",
  "original_toy_location" => "{\"p\":\"0246w4+SSqqlJw==\",\"h\":{\"iv\":\"1uEnjlCNot9sYNgR\",\"at\":\"UhkhK6YlOTxJg75juqIMGA==\"}}" }

Other Cool things

encrypts does even more than we have looked at so far. In the interest of not writing a book, I'm not going to go into much detail here, but would like to mention some of its other capabilities.

We have only looked at encrypting simple strings, but encrypts can encrypt rich text attributes too.

It also provides support for previous encryption schemes. This means that we can start out using non-deterministic encryption on a column and change to using deterministic encryption later on. I would definitly recommend reading the fine print before using this feature.

We can rotate our (non-deterministic) keys. This is pretty cool, just note that it is not currently supported for deterministic encryption.

Related to the rotating keys, we can configure encrypts to store a reference to the key used to encrypt in the encrypted data itself.

If using deterministic encryption, encrypts supports using unique constraints. If we need to ensure uniqueness in any of our encrypted columns, there are some things to be aware of. Be sure to read up on it first.

Encrypted columns are automatically filtered from Rails logs by default. encrypts provides a way to disable this functionality.

This might be a good place to mention that the implementation is modular, and allows quite a bit of customization. Many of encrypts options can be set on a per-attribute basis, or at more global levels.

Limitations

Even with all of its glory, encrypts does have limitations. We'll take a look at a few that stood out to me. Given the variety of applicable use cases and the breadth of the functionality, I am sure other folks will have their own list.

Fuzzy searching - The search functionality encrypts provides requires an exact match on search text. This means doing a LIKE query, for example, won't work. It also means that any queries done on encrypted columns will need to go through Rails and ActiveRecord vs being manually crafted in SQL.
Rich text search - While it is rad that encrypts can encrypt rich text, it can only do so non-deterministically right now. This means that we won't be able to search rich text.
Deterministic searching does not support multiple keys - Something good to be aware of going in - if using deterministic encryption/searching, we won't have the ability to use more than one key at a time. If we need to change keys, we'll likely need to do something fancy.
Rails console exposes data - This may seem obvious, but if a malicious person gets access to our Rails console, they can load encrypted data into objects and view the plaintext all day long. In Jorge's post, he mentioned that HEY is using a console extension that sits on top of the encryption feature that protects and audits console access. Unfortunately, that is a private gem (named console1984) and not available in Rails (at this time).
Deterministic encryption reduces security - I don't think this is a fault of the implementation per se, but if we use deterministic encryption, then any two rows that have the same value for an encrypted attribute will have the same value stored in the database. While we can't necessarily reverse engineer how it was encrypted, if we know what one of the row's plaintext value is, then we know what the other row's plaintext value is as well. Non-deterministic encryption doesn't have this same weakness.

Summary

TLDR: I am pretty excited and intrigued by this feature. I think it will be cool to see how people use it and how it evolves over time. My hunch is that some of the current limitations will go away (like not supporting multiple keys for deterministically encrypted attributes) as more people begin using it and digging through the code.

WSL for Rails developers

Joel Hayhurst — Wed, 19 May 2021 17:43:58 +0000

It is now possible, thankfully, to develop in Rails on Windows. For many years, it seemed one needed a Mac for this purpose; and I have nothing against Macs, and enjoy the MacBook Pro, which I have used throughout my career. But I also have a Windows machine. I like to play games sometimes, and now that I work from home, along with an increasing number of people, I have access to it. I like the idea that developers should be able to enjoy their preference in environment, where reasonable. Some prefer to boot directly into Linux, which is also a perfectly fine choice. On a whim, I decided to try Rails development on Windows, and it has worked out nicely.

With WSL, the Windows Subsystem for Linux, we can now run a Linux distribution within Windows 10. You can use it similarly to how you might use Homebrew on a Mac, and the whole thing is surprisingly seamless, especially after some recent improvements. All of the Linux development and server tools you're used to are present, and work the same as you are familiar with.

Installation

It's relatively easy to install; first you have to enable WSL as a "Windows feature", and then you go to the Microsoft Store and install your preferred distro. I know they have Debian and Ubuntu available, as well as some others. I chose Debian.

Terminals

As for terminals, there's a few options. The default terminal that comes with WSL is named after the distribution you installed, so for me it is simply called "Debian". It has the Debian swirl logo, which I like, and the default color scheme and font it comes with I find pleasant. On the downside, I've found it will always scroll down if new content comes up (say you are tailing a test log), so I got in the habit of waiting for the tests to finish before examining logs. It also doesn't have tabbing, but I found I don't need tabbing as much as I thought; I'm not on a laptop screen, so I have enough space to keep separate terminals open. It is possible to use PowerShell to access WSL, by running the wsl command, but I find the color scheme is a bit off. And there's other options, like the new Windows Terminal, which does have tabbing and lots of customization options, but requires editing a text file to change settings (you can do it, you're a developer!), and still feels kind of in beta. In general though, I'm fine with the default WSL terminal.

VSCode

WSL also integrates perfectly with VSCode, which I use for most of my code editing. You can edit files directly within WSL and have them immediately affect the app. I mention this because there used to be an issue with this in WSL 1, but it has been resolved since WSL 2.

Docker

Docker Desktop also integrates very nicely with WSL since version 2. Before WSL 2, Docker had to use Hyper-V to emulate Linux, but now we don't even need Hyper-V enabled. Docker now uses the WSL 2 backend, and it seems quite a bit faster to start as well, in my experience.

Filesystem

When you use WSL, you will want to store all of your relevant code within the WSL filesystem, not on the Windows filesystem. In WSL 1 there was more crossover, but not in WSL 2; everything goes on the Linux virtual drive. It is still possible to browse these files using Windows Explorer, if you are so inclined, but in general probably best to stick with Linux tools when using this drive. Still I do open one of my WSL directories in Explorer regularly for the purpose of easily opening screenshots produced by Capybara.

Copy-and-paste

One other note about transitioning from Mac, you'll probably want to enable the Ctrl-Shift-C and Ctrl-Shift-V copy-paste functionality in the terminal. On Mac, we have that handy Command key, but on Windows we are stuck with just Ctrl, which is used plenty by Linux itself; so Ctrl-Shift is a necessary compromise. It's not that hard to get accustomed to, and besides that, we can also use right-click to copy and paste text.

Caveats

I've been developing with this system for over a year now and haven't run into any serious problems. Probably the biggest issue I had is that WSL, combined with the code repositories of some large apps I'm working on, used up a lot of disk space, and my C: drive only has 256 GB. I did however find a way to move WSL to a separate drive, a nice 1 TB SSD, and that solved my space issues. This can be accomplished using the wsl --export and wsl --import commands without much fuss.

Another issue I've faced is that when working on a larger application, a lot of memory is used up by WSL. Running two operating systems simultaneously uses more resources. Even after quitting the applications, and quitting Docker, this memory would still be reserved by WSL. The solution is that when I'm done with using WSL for the day, I run wsl --shutdown in PowerShell and that frees up all the memory again. I went ahead and upgraded to 32 GB of RAM to avoid these paging annoyances. However, this upgrade is only necessary because of the size of this application. Every other application I've worked in was comfortable with 16 GB.

Conclusion

These are all of the thoughts I have on WSL at this time. I hope you find it useful to be able to program in Rails on Windows, if that is your desire. If you give it a shot, you might find it's more comfortable than you expected. Thanks for reading.

What's Cooking in Rails 7?

Jason Dinsmore — Wed, 21 Apr 2021 16:16:09 +0000

A new version of Rails always brings new and exciting features. This writeup takes a look at some of the things Rails 7 has in store for us.

To find these examples, I dug through the Changelogs and selected some of the features that have me most excited so far, primarily based on how I currently use Rails.

I encourage you to do the same - you may find something awesome that impacts a feature you use regulary or provides functionality you've been lying awake at night fantasizing about.

For each feature, I have included the PR that introduced it and the GitHub user who submitted the PR.

Note that several of these features have followup PRs from people making essential contributions by updating documentation, adding changelog entries, tweaking implementations, etc. If you are one of those folks, please know that while you're not given explicit credit here, we are very grateful for all you do! 🙏

Railties

The main Branch

PR 40254 by @prateekkish

When generating a new Rails project or plugin without the --skip-git flag, the default branch of the created repository will now be main instead of master. If you've got a different branch set in your git config's init.defaultBranch, that default will be used instead.

Rails began initializing the Git repository in 5.1 via PR 27632 by @dixpac.

benchmark Anywhere

PR 40734 by @semaperepelitsa

This PR adds the ActiveSupport::Benchmarkable.benchmark method to the Rails namespace, which allows you to easily benchmark a block of code anywhere without having to declare a logger or extend/include the ActiveSupport::Benchmarkable module.

To use the new functionality, you'd do something like this:

  Rails.benchmark('Print bar') do
    # code to benchmark
    100.times { puts 'bar' }
  end

After your code executes, the message you provided and the time it took the code to execute in milliseconds will be passed along to the Rails logger, ie: Print bar (0.6ms). The benchmark call will return the return value of the block, 100 in this case.

Support Stats For Stylesheets and ERB Views

PR 40597 by @joelhawksley

If you've run rails stats, you know that it provides various statistics about your app's codebase. This new functionality adds Views and Stylesheets to the output, providing stats for files in app/views and app/assets/stylesheets.

Note that stylesheets need to have a .css or .scss extension, and view files need to have the .erb extension in order to be processed.

ActionPack

Redirect Back or To

PR 40671 by @dhh

Adds redirect_back_or_to(fallback, **) as a nice shorthand for redirect_back(fallback_location:, **).

This avoids having to specify the fallback_location kwarg (keyword argument) to redirect_back. redirect_back_or_to will redirect the user back to whence they came if it can, and will redirect to the fallback location if it can't.

The old syntax still works and hasn't been explicitly deprecated, but now redirect_back calls redirect_back_or_to under the hood.

Unpermitted Parameter Context

PR 41809 by @bbuchalter

This PR provides controller and request context when the parameters object is initialized so that when unpermitted attributes are logged, the controller, action, request, and filtered parameters will be logged as well.

ActionView

Lazy Load Images

PR 38452 by @jonathanhefner

Currently, apps can take advantage of the HTML standard's lazy loading of images by manually specifying the loading attribute as lazy in calls to image_tag. This new configuration allows you to set the default to "lazy" app-wide, so that the attribute will only need to be specified for places you want an image to be eager loaded.

Tag Attributes From Hash

PR 40657 by @seanpdoyle

This cool addition provides a slick way to tranform a hash of attributes into attributes on an ERB tag.

For example:

<div <%= tag.attributes({ id: 'percent-loaded', role: 'progressbar', aria: { valuenow: '75' }}) %>>

Would render:

<div id="percent-loaded" role="progressbar" aria-valuenow="75">

Perhaps not the greatest example, but if the hash were generated programmatically and required logic, this could be pretty useful.

ActiveModel

Add Numericality In Range Validation

PR 41022 by @mpapis

When validating numericality of a model attribute, this new syntax lets you specify a range for the numericality validator versus having to sandwich between the greater_than_or_equal_to and less_than_or_equal_to options.

Example usage:

validates :latitude, numericality: { in: -90..90 }

ActiveRecord

Invert a where Clause

PR 40249 by @kddeisz

This addition provides a handy way to get at the inverse of a where clause by via invert_where.

For example:

good_students = Student.where(grade: 80..100)
# SELECT \"students\".* FROM \"students\" WHERE \"students\".\"grade\" BETWEEN 80 AND 100

bad_students = good_students.invert_where
# SELECT \"students\".* FROM \"students\" WHERE NOT (\"students\".\"grade\" BETWEEN 80 AND 100)

Exclude a Record From Results

PR 41439 by @GlenCrawford

Ever need a way to get all records matching some condition except a record you already have? excluding may be your jam.

Instead of:

other_users = User.where(rating: 80..).where.not(id: primary_user)

You can:

other_users = User.where(rating: 80..).excluding(primary_user)

Build or Create Association on Has One Through

PR 40007 by @perezperret

Enables the build_association and create_association functionality for has_one through: relations. Previously these were not available on through associations.

For example, if you had:

class Dog
  has_many :toys
  has_one :toy_box
end

class Toy
  belongs_to :dog
  has_one :toy_box, through: :dog
end

class ToyBox
  belongs_to :dog
end

You'll now be able to:

toy.build_toy_box
# <ToyBox:0x00007f572007e170 id: nil, dog_id: 3>

Or:

toy.create_toy_box
# <ToyBox:0x00005601f2ac09a0 id: 5, dog_id: 3>

Respect Column Type When Calculating Average

PR 40351 by @schmijos

In prior versions of Rails, calling ActiveRecord::Caculations#calculate with :average would result in a BigDecimal, even if a column was of a Float or Integer type.

For example:

Coordinate.last.longitude.class
# Float
Coordinate.calculate(:average, :longitude).class
# BigDecimal
Coordinate.calculate(:average, :longitude)
# 0.13002356e3

With the new behavior, it honors the type of the database column:

Coordinate.calculate(:average, :longitude).class
# Float
Coordinate.calculate(:average, :longitude)
# 130.02356

One of the motivators for this change was that JSON conversions of BigDecimal result in a string value, whereas Float results in a numeric value.

JSON.parse({ avg_longitude: 130.02356.to_d }.to_json)
# { "avg_longitude" => "130.02356" }
JSON.parse({ avg_longitude: 130.02356 }.to_json)
# { "avg_longitude" => 130.02356 }

Automatically Encrypt/Decrypt a Model Attribute

PR 41659 by @jorgemanrubia

Finally, we get to encrypts.

This is deserving of a post all its own. The TLDR; version is that this feature adds a mechanism to auto-encrypt/decrypt an ActiveRecord attribute by declaring encrypts attr_name in the model. When a record is written to the database, it will automatically be encrypted, and when it is loaded from the database, the attribute will automatically be decrypted.

encrypts also provides a workflow for changing the encryption scheme being used, without needing to re-encrypt all previously encrypted records under the new scheme.

Data can be encrypted in a non-deterministic manner (encrypting the same text twice will result in different ciphertexts), or can be encrypted deterministically - which allows querying the encrypted attribute.

Other niceties are the capability to encrypt Action Text and to filter encrypted attributes from the application log files.

It really does quite a lot and provides several ways to customize behavior, depending on your needs.

What Else?

I touched on the things I am most excited about so far, but there is other great work underway. ActionPack, ActionView, ActiveRecord, and ActiveSupport in particular, all have rapidly growing lists of changes.

Thanks again to all the contributors who are making this next release possible. It's shaping up nicely!

Ruby on Lambda: Integrating with ActiveStorage

Nate Shoemaker — Tue, 11 Aug 2020 20:00:51 +0000

This series of articles will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talk about creating a AWS Lambda Layer with Docker. In part two, we chatted about architecting a serverless app. In part three, we went through some strategies surrounding deployment. Here in part four, we'll investigate what's needed to integrate a Rails app that uses Active Storage with AWS Lambda. Check out the other posts in the series:

Note: there are some assumptions made in this article regarding architecture and reading Part Two of this series, Planning & Architecting, may be useful before diving in here.

Performance was our main reason for moving from Sidekiq to AWS Lambda. We used Active Storage on a few models and wanted to continue using it after the migration. Specifically, we wanted to take advantage of Active Storage variants via a GraphQL API and React frontend. The app had multiple views that requested all of the variants at once, and because the variants are created at the time of the request, the user experience under Sidekiq was very poor. What if we could create all of those variants on Lambda as we processed the document? It sounded plausible to us, so we spiked on it.

Note that we'll be touching on specific pieces of code that directly relate to the integration; this won't be a tutorial on how to use Active Storage or setup AWS Lambda. We are using S3 as our storage service, and will be referencing it as a standalone entity. Let's dive in!

Uploading

Instead of passing form params to a model, we’ll take more of a manual approach. In the controller:

@document.file.upload(file)

We have a Document model with has_one_attached :file. This allows the file to be uploaded to S3 without creating the backing Active Storage record. Delaying that allows us to process the document on Lambda, create the needed variants, and respond back with data that will be used to create those records.

Lambda has no knowledge of the Rails app or Active Storage, but if we can point it to the file we just uploaded, it can reference and process it. After a Document has been created, we send document.file.key to the vent function (LINK TO PLANNING AND ARCHITECTURE POST) that starts the processing work. The document’s key is the name of the file on S3. On Lambda, the real fun begins.

Variant Creation

Essentially, we needed to trick Active Storage. When requesting a variant, Active Storage will look for variant.jpg in a specific location on S3:

"variants/#{key}/#{variant_key}"

key is the original file's key (document.file.key), and variant_key is a hash derived from the variant's transformation options (e.g. {auto_orient: true, resize: '1000x1000'}) and the Rails app's ENV['SECRET_KEY_BASE']. Let's look at the Lambda code:

def create_variant(image, key, transformation)
 # create message verifier and variant key
 verifier_key = verifier.generate(transformation, purpose: :variation)
 variant_key = Digest::SHA256.hexdigest(verifier_key)

 tmp = Tempfile.new('variant.jpg')

 MiniMagick::Tool::Convert.new do |convert|
   convert << image.path
   # apply transformations here, like:
   # convert.auto_orient
   # convert.resize transformation[:resize]
   convert << tmp.path
 end

 s3 = Aws::S3::Client.new
 s3.put_object(
   bucket:       "some-cool-bucket",
   key:          "variants/#{key}/#{variant_key}",
   body:         tmp,
   content_type: 'image/jpeg'
 )
end

#create_variant takes a File, document.file.key, and the transformation options as arguments. The variant_key is created (we'll look at what verifier is soon), and the actual image conversion is handled with ImageMagick. Note the actual ImageMagick conversions depend on the transformation options being passed in. For an example I used the options that were previously mentioned, {auto_orient: true, resize: '1000x1000'}. After the image is processed, we then place it on S3, using the specific path Active Storage expects.

Figuring out how to create the verifier_key was quite difficult. It took a late night fueled by coffee and spelunking into the bowels of Active Storage, but it proved fruitful. Let's take a look at #verifier:

def verifier
 key_generator = ActiveSupport::CachingKeyGenerator.new(
   ActiveSupport::KeyGenerator.new(ENV['SECRET_KEY_BASE'], iterations: 1000)
 )
 key = key_generator.generate_key('ActiveStorage')
 ActiveSupport::MessageVerifier.new(key)
end

Active Storage was created by people much smarter than I, and I'll be the first to admit that I did not look at all the implementation details surrounding ActiveSupport::CachingKeyGenerator, than I, and I'll be the first to admit that I did not look at all the implementation details surrounding ActiveSupport::KeyGenerator, ActiveSupport::MessageVerifier. However, their names are descriptive and give us a good bird's-eye view. A key is generated from your Rails app's SECRET_KEY_BASE, and that key is then turned into a digest that is used to create the path to the variant. A few notes:

The iterations option when instantiating ActiveSupport::KeyGenerator does not have to be 1,000. I, of course, didn't test every number, but the natural numbers I tested all had the same result. Active Storage uses 1,000, so I thought I would too.
The argument passed to #generate_key is the salt, and for our intended purposes on Lambda it can be any string.

Data for the `active_storage_attachments` record

Early on in the spike, we hit a lot of ActiveStorage::IntegrityErrors. While you can find yourself raising this error on accident in many scenarios, the reason why is always the same: the hashed contents of the file don't match what's stored in the database. Since we uploaded the file straight to S3, we purposefully didn't create a record in active_storage_attachments, and don't have the reference to the file anymore without downloading it (which would be a waste). We had to create the digest on Lambda while we still had a reference to the file.

Here's the data we collect for each page in a document (note that file is a File object):

{
 content_md5:  Digest::MD5.file(file).base64digest,
 byte_size:    File.size(file).to_s,
 key:          key,
 content_type: content_type
}

After every page is processed, our sink function collects all the data and sends a JSON response back to the Rails app.

All together now

When the Rails app receives the JSON, Sidekiq workers are started that process the response. Part of the workers' responsibility is to create a record in active_storage_attachments:

def create_blob(data)
 params = {
   key:          data['key'],
   filename:     data['key'],
   checksum:     data['content_md5'],
   byte_size:    data['byte_size'],
   content_type: data['content_type']
 }

 ActiveStorage::Blob.create_before_direct_upload!(params)
end

#create_before_direct_upload! allows us to create the record and not upload the file to S3 as it already exists there. Tricking Active Storage, and all.

And with that, Active Storage works just as it normally would. With our implementation, however, we process all variants in parallel, improving load times and user experience for everyone!

Ruby on AWS Lambda: Package & Ship It

Nate Shoemaker — Tue, 11 Aug 2020 20:00:47 +0000

This article is part of our Ruby on AWS Lambda series. A recent project had us migrating an existing pdf document processing system from Rails/Sidekiq to AWS Lambda. The processing includes OCR, creating preview images, splicing the pdf, and more. Moving to Lambda reduced processing time by 300% in some cases; parallelization for the win!

This series will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talked about creating an AWS Lambda Layer with Docker. In part two, we chatted about architecture. Here in part three, we'll go through some strategies surrounding deployment. Check out the other posts in the series:

Deploying to Lambda gets more complicated as the application grows. There is a big difference between deploying a single hello world function with zero dependencies, and deploying multiple functions in multiple languages along with dependencies in a Lambda Layer. We discovered some useful things along the way while making that transition. Let's dive in.

Function naming conventions and environment variables

We followed a straightforward convention for differentiating between function names and their environments:

{function_name}_{environment}

Given our three Rails environments (development, staging, and production) and a function name of pdf_processor, we'd have three separate functions: pdf_processor_development, pdf_processor_staging, and pdf_processor_production. Adding more functions may seem a bit overwhelming in terms of function count, but the clear distinctions between functions and their environments allow for isolated development, testing, and debugging.

Along with other normal environment variables that the functions use, we added a RUBY_ENV environment variable that was either development, staging, or production. This is primarily used for name-spacing S3 buckets, DynamoDB tables, and calling other lambda functions. This decision played out well because we now have isolation in how we use other systems and services. This also makes reading logs and tracking performance metrics a lot easier.

Instrumentation

Instrumentation refers to the measure of a product's performance, to diagnose errors, and to write trace information. Early on in development, we added instrumentation to gain insight into how the bulk of the processing work performs.

Here’s our implementation:

# instrument.rb

module Instrument
 def instrument(tag)
   start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
   -> { puts("--> #{tag} took #{((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round}ms") }
 end
end

Then it is used like:

require 'instrument'

i = instrument('doing some work')
do_the_work()
i.call

The log would look like:

--> doing some work took 300ms

The start variable gets caught in a closure and is available for the Proc that is returned. The semi-cryptic looking Process.clock_gettime is used to read from the OS clock, and is more accurate than Ruby's Time class.

Serverless

When we first started building on Lambda, we deployed the AWS way: zip the code up and ship it via the AWS CLI. This was fine in the beginning, but it left a lot to be desired. It puts the responsibility on the developer to know about the deployment process, something that should be handled by configuration. That added responsibility also detracted from a deployment being deterministic. If you missed a small detail, a deployment, and thus the function itself, could be botched. We don't zip and deploy our Rails apps up manually, and we shouldn't have to do that with our serverless functions either!

In comes Serverless Framework. Everything we love about configuration and deterministic deployments are handled by Serverless. All configuration is written in YAML and is concise, intuitive, and incredibly powerful.. Environment variables, runtime specification, memory configuration, AWS roles, and layer versioning can all be done in a single YAML file. These features became even more valuable when we rewrote our vent function in JavaScript. We now have a separate serverless config for each lambda function that allows us to achieve true language isolation. If you'd like to learn more about using Serverless, take a look at their AWS quickstart guide.

Shipping a Lambda application doesn’t have to be difficult. I hope what we’ve learned has helped you!

Ruby on AWS Lambda: Layer Dependencies

Joel Hayhurst — Tue, 11 Aug 2020 20:00:41 +0000

This article is part of our Ruby on AWS Lambda blog series. A recent project had us migrating an existing PDF document processing system from Rails Sidekiq to AWS Lambda. The processing includes OCR, creating preview images, splicing the PDF, and more. Moving to Lambda reduced processing time by 300 times in some cases.

This series of articles will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talk about creating a AWS Lambda Layer with Docker. Check out the other posts in the series:

Building dependencies for Lambda can be confusing. They need to be compiled, zipped up, and then made into a layer. We also have to keep the file size for the dependencies under Lambda's limit. Here is how we did that using a Dockerfile.

You can check out the full Dockerfile here.

Dockerfile details

One question when building the dependencies was which Docker image to use. We first tried using the amazonlinux image, but this actually resulted in some build problems with one of our dependencies. We later found the LambCI images and ended up using lambci/lambda:build-ruby2.7, because we are using Ruby. This worked perfectly for us, and it has the benefit of already having build tools installed, making for faster Docker builds.

# Use AWS Lambda ruby2.7 build environment
FROM lambci/lambda:build-ruby2.7

AWS Lambda currently has a limit of 250 MB for its dependencies. If you are building significantly larger dependencies, as we were, then you are likely to reach this limit. It is therefore very important to compile your dependencies in a way that reduces file size. By using the -Os compile time option, we were able to reduce the size of our binaries by over 90%.

# Optimize compilation for size to try and stay below Lambda's 250 MB limit
# This reduces filesize by over 90% (!) compared to the default -O2
ENV CFLAGS "-Os"
ENV CXXFLAGS $CFLAGS

While slightly less performant than the default -O2, the massively reduced file size is worth it in this situation.

Next up is building Leptonica.

WORKDIR /root

# Leptonica image-reading dependencies
RUN yum install -y libjpeg-devel libpng-devel libtiff-devel

RUN curl -O http://www.leptonica.org/source/leptonica-1.79.0.tar.gz
RUN tar zxvf leptonica-1.79.0.tar.gz

WORKDIR leptonica-1.79.0
RUN ./configure --prefix=/opt
RUN make install

You will need to install the -devel packages when compiling, but won't need the -devel variant when providing the dependencies for runtime.

Then Tesseract:

WORKDIR /root

# Optional Tesseract foreign language training dependencies
# libicu-devel on Yum is of insufficient version (50, 52 is required)
# These are also not really necessary for our usage.
#RUN yum install -y libicu-devel pango-devel cairo-devel

RUN curl -Lo tesseract-4.1.1.tar.gz \
  https://github.com/tesseract-ocr/tesseract/archive/4.1.1.tar.gz
RUN tar zxvf tesseract-4.1.1.tar.gz

WORKDIR tesseract-4.1.1
RUN ./autogen.sh --prefix=/opt
# These ENV vars have to be set or it will not build
ENV LEPTONICA_CFLAGS -I/opt/include/leptonica
ENV LEPTONICA_LIBS -L/opt/lib -lleptonica
RUN ./configure --prefix=/opt
RUN make install

# English training data
WORKDIR /opt/share/tessdata
RUN curl -LO https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata

GhostScript was technically installable via RPM, but the amount of dependencies was too great. More on that later. We decided to just compile it with a minimal dependency set.

WORKDIR /root

RUN curl -LO \
  https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs952/ghostscript-9.52.tar.gz
RUN tar zxvf ghostscript-9.52.tar.gz

WORKDIR ghostscript-9.52
RUN ./configure --prefix=/opt
RUN make install

Ironically, we end up installing ghostscript-devel so that ImageMagick can be built. It might be possible to use the prior GhostScript installation here, but this was simple enough for build purposes.

WORKDIR /root

RUN yum install -y ghostscript-devel

RUN curl -Lo ImageMagick-7.0.10-6.tar.gz \
  https://github.com/ImageMagick/ImageMagick/archive/7.0.10-6.tar.gz
RUN tar zxvf ImageMagick-7.0.10-6.tar.gz

WORKDIR ImageMagick-7.0.10-6
RUN ./configure --prefix=/opt
RUN make install

We considered using libvips instead of ImageMagick, but it would have added scope to the project. Nonetheless I have some commented out code in the Dockerfile for building libvips in case we decide to switch to it in the future.

For Ruby gems, a Gemfile in the same directory that contains all of the gems is all we need.

source 'https://rubygems.org'

# PDF processing gems
gem 'simhash2'
gem 'phashion'
gem 'rtesseract'
gem 'mini_magick'
gem 'pdf-reader'
gem 'hexapdf'

# Other gems used in these files
gem 'procto'
gem 'adamantium'
gem 'concord'
gem 'activesupport', '~> 6.0.2'
gem 'chronic'
gem 'activestorage'

Note that a little bit of trickery is necessary to modify the gem paths for loading in Lambda.

WORKDIR /root

# Phashion dependencies
# Can skip this step because they are already installed above for Leptonica
#RUN yum install -y libjpeg-devel libpng-devel

# Copy Gemfile from host into container's current directory
COPY Gemfile .

RUN bundle config set path vendor/bundle
RUN bundle

# Modify directory structure for Lambda load path
WORKDIR vendor/bundle
RUN mkdir ruby/gems
RUN mv ruby/2.* ruby/gems
RUN mv ruby /opt
WORKDIR /root

Now the RPM packages. I left in the installation for GhostScript and its dependencies from RPM as a comment, mainly so you can see how many packages I had to specify manually.

Note that you will have to specify the x86_64 variants of these packages when using these tools.

WORKDIR /root

# Install yumdownloader and rpmdev-extract
RUN yum install -y yum-utils rpmdevtools

RUN mkdir rpms
WORKDIR rpms

# Download dependency RPMs
RUN yumdownloader libjpeg-turbo.x86_64 libpng.x86_64 libtiff.x86_64 \
  libgomp.x86_64 libwebp.x86_64 jbigkit-libs.x86_64
# GhostScript and dependencies
# To reduce dependencies, we are compiling GhostScript from source instead
# RUN yumdownloader ghostscript.x86_64 cups-libs.x86_64 fontconfig.x86_64 \
#   fontpackages-filesystem freetype.x86_64 ghostscript-fonts jasper-libs.x86_64 \
#   lcms2.x86_64 libICE.x86_64 libSM.x86_64 libX11.x86_64 libX11-common \
#   libXau.x86_64 libXext.x86_64 libXt.x86_64 libfontenc.x86_64 libxcb.x86_64 \
#   poppler-data stix-fonts urw-fonts xorg-x11-font-utils.x86_64 avahi-libs.x86_64 \
#   acl.x86_64 audit-libs.x86_64 cracklib.x86_64 cracklib-dicts.x86_64 cryptsetup-libs.x86_64 \
#   dbus.x86_64 dbus-libs.x86_64 device-mapper.x86_64 device-mapper-libs.x86_64 \
#   elfutils-default-yama-scope elfutils-libs.x86_64 gzip.x86_64 kmod.x86_64 kmod-libs.x86_64 \
#   libcap-ng.x86_64 libfdisk.x86_64 libpwquality.x86_64 libsemanage.x86_64 \
#   libsmartcols.x86_64 libutempter.x86_64 lz4.x86_64 pam.x86_64 qrencode-libs.x86_64 \
#   shadow-utils.x86_64 systemd.x86_64 systemd-libs.x86_64 ustr.x86_64 util-linux.x86_64 \
#   expat.x86_64 xz-libs.x86_64 libgcrypt.x86_64 libgpg-error.x86_64 elfutils-libelf.x86_64 \
#   bzip2-libs.x86_64

# Extract RPMs
RUN rpmdev-extract *.rpm
RUN rm *.rpm

# Copy all package files into /opt/rpms
RUN cp -vR */usr/* /opt

# The x86_64 packages extract as lib64, we need to move these files to lib
RUN yum install -y rsync
RUN rsync -av /opt/lib64/ /opt/lib/
RUN rm -r /opt/lib64

Notice some more path management. We used rsync for copying because cp gave us some problems.

Now we just need to zip up the dependencies.

WORKDIR /opt
RUN zip -r /root/ProcessDocumentLayer.zip *

And lastly, the entrypoint for the Dockerfile, which copies the zip file to an output directory.

ENTRYPOINT ["/bin/cp", "/root/ProcessDocumentLayer.zip", "/output"]

Now we just need the Docker commands to build this. I put them at the very top of the file under a "Usage" section.

# Usage:
# docker build -t lambda .
# docker run -v $(pwd):/output lambda
# ./publish_layer.sh.

The publish_layer.sh script is a small one we wrote that uses awscli to upload and publish the layer. You'll have to authenticate with AWS for it to work. I used aws configure for this purpose but you can check out this article for more info.

#!/bin/sh
aws s3 cp ProcessDocumentLayer.zip s3://process-document-layers
aws lambda publish-layer-version --layer-name ProcessDocumentLayer --description "Process Document dependencies" \
  --content S3Bucket=process-document-layers,S3Key=ProcessDocumentLayer.zip --compatible-runtimes ruby2.7

And that's it. With this Dockerfile, we are able to easily build and publish a dependency layer for our OCR system on Lambda. We hope this was useful for you!

Ruby on AWS Lambda: Planning & Architecting

Nate Shoemaker — Tue, 11 Aug 2020 17:58:39 +0000

This article is part of our Ruby on AWS Lambda blog series. A recent project had us migrating an existing pdf document processing system from Rails Sidekiq to AWS Lambda. The processing includes OCR, creating preview images, splicing the pdf, and more. Moving to Lambda reduced processing time by 300% in some cases; parallelization for the win!

This series will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talked about creating an AWS Lambda Layer with Docker. In part two, we'll chat about architecting a serverless app. Check out the other posts in the series:

The Problem

Rails devs, tell me if you've been here before: you have small units of work that should be processed concurrently, so you reach for Sidekiq. Solved! Well, usually solved. Sidekiq is an amazing tool that we use at Hint, and it solves most concurrency problems.

However, a bottleneck is always a possibility. In our case, the document processor we engineered could process a PDF with one page, or 500. But since each page has to be processed and there is no page limit when uploading, it was common to process hundreds of pages. This work took much too time with Sidekiq, even with tens of workers. We wanted to see the same performance whether the PDF a user uploads has 50 pages, or 500.

The Solution

AWS Lambda was a good solution for our performance problem. It allows us to run our workers in parallel independent of the number of pages we are processing. However, Lambda doesn't give you any guidance on how to design a serverless application. You are provided the basic building blocks: a function that calls other functions, and the ability to use any other AWS service. Quite open-ended! Luckily, there are some standard architecture practices that have emerged in the serverless community. We'll be focusing on the most popular strategy: fan-out, fan-in.

"What do you mean, software architecture?"

Glad you asked! Martin Fowler said it best:

When people in the software industry talk about “architecture”, they refer to a hazily defined notion of the most important aspects of the internal design of a software system.

When we talk about architecture in the context of Lambda, we'll be touching on function composition, utilizing other AWS services, and how the application communicates with the real world. If you haven't, check out Martin Fowler's thoughts on software architecture.

Fan-out, Fan-in

The fan-out, fan-in architecture is simple: one function starts n number of functions, and when those finish, another function does something with the results. The first function is referred to as the ventilator (or vent). The vent function calls worker functions. Once the worker functions finish, the sink function is called. So, vent -> workers -> sink.

For example, let's use a hypothetical 100-page pdf.. The vent function is triggered from an outside application and 100 worker functions are called. Each worker process a single page. The sink function is then invoked when all 100 are done, and we do things with the result. In theory, 1,000,000 pages should take as long as 100, since all the processing is happening in parallel. Let's go into a little bit more detail about the whole process:

User uploads PDF from Rails app
PDF is uploaded to S3 via Active Storage
Vent function is triggered with the file key as an argument
Vent function calls n number of worker functions, passing file key as an argument so workers can find the file on S3
Worker function processes page, stores preview image, collects information about page and stores that in DynamoDB, and a record in DynamoDB that holds the keeps track of the pages left is decreased by one
If the worker is the final worker in the queue, the sink function is called
Sink function collects all page information from DynamoDB and sends results as JSON back to the Rails app

DynamoDB is a NoSQL database offered by AWS. When using the fan-out, fan-in pattern you must keep track of progress so the sink function can be called when all workers are completed. The nature of serverless means that no state can be held, thus an external, stateful service must be used. In our case, we are using it as a glorified counter. When a worker function finishes, we decrease the records count column (pages_left) by one:

  def decrement_pages_left(job_id)
    db = Aws::DynamoDB::Client.new
    resp = db.update_item(
      table_name:                  "documents-#{ENV['RUBY_ENV']}",
      key:                         { job_id: job_id },
      update_expression:           'set pages_left = pages_left - :val',
      expression_attribute_values: { ':val' => 1 },
      return_values:               'UPDATED_NEW'
    )

    invoke_sink_function(job_id) if resp[:attributes]['pages_left'] == 0
  end

Note that the sink function is then called if there are no pages left to process. Easy!

For a more in-depth look into the fan-out, fan-in pattern take a look at this fantastic blog post by Yan Cui.

Knowing what language is better suited for a certain task

We initially used Ruby for our vent, worker, and sink functions. However, we hit a bottleneck in the vent function. Invoking hundreds of concurrent network requests (which are IO bound) is not one of Ruby's strong suits. Using any kind of library that takes advantage of CPU cores doesn't gain any advantage because of Lambda's CPU limitations (more on that in the next section). So, what language does Lambda support that has asynchronous IO bound operations out of the box? JavaScript!

Now, this isn't a bash on Ruby. All programming languages have their strengths and weaknesses, and a rewrite in a different language should be well researched and thought out beforehand. Luckily for us, we have lots of Ruby and JS experience. Also, the vent function encapsulates very little business logic, so a rewrite would be a good fit if an initial spike proved fruitful. And it did! When processing a 186-page document, the vent function took 30 seconds in Ruby, and 2 seconds in JS. Nailed it!

If you experience performance issues on Lambda, make sure to research the problem thoroughly. The cool thing about Lambda is that you can use different languages throughout the application process. If part of the process would perform much better with a different language, try it out!

CPU/RAM limitations

When assigning resources to your Lambda functions, memory is the only configurable option. Why? Well, there is no obvious answer in the UI, and even worse, it's hidden deep in a FAQ. When you choose the amount of memory you want to allocate for a function, you are given proportional CPU power and other resources. What the docs don't tell you, however, is that if you allocate enough memory, you'll be given two cores instead of one.

This Stack Overflow comment has more info. TL;DR: if you are using 1.8GB or more of memory, you get to use two CPU cores, instead of one. Even two cores is not a lot, and that's why asynchronous operations in programming languages that utilize multiple cores don't perform well on Lambda. When architecting a serverless application, it's better to split larger tasks into smaller subtasks, when possible.

I hope you learned something new today! Architecting serverless applications come with their own unique set of challenges, but the community has great solutions that have been production-tested for quite some time.

5 Ways to Splat in Ruby

Jason Dinsmore — Tue, 05 May 2020 15:14:28 +0000

Originally posted on Hint's blog.

In honor of it being 5/5 today (aka Cinco de Mayo), I thought we'd continue the pattern and take a quick look at 5 different ways to use Ruby's infamous splat (*) operator.

Let's hop right into it!

1. Interpolate Into an Array

Splat can be used to expand the contents of one array into another:

middle = %i(bar baz)
[:foo, *middle, :qux]

# [:foo, :bar, :baz, :qux]

2. Capture an Argument List

You can use splat to capture a list of arguments into an Array. Below, we're capturing the entire list of arguments being passed to foo into an Array named bar:

def foo(*bar)
  "#{bar.class.name}: #{bar}"
end

foo(:a, :b, :c)

# Array: [:a, :b, :c]

3. Capture First/Last or First/Rest Values

Splat can also be leveraged to split up an array into parts. Kinda like pattern matching:

arr = [1, 2, 3, 4, 5]
first, *, last = arr
puts first # 1
puts last # 5

first, *rest = arr
puts first # 1
puts rest # [2, 3, 4, 5]

4. Coerce a Value Into an Array

If you want to ensure the thing you are dealing with is an Array, you can splat it.

foo = 1
bar = *foo
puts bar # [1]

foo = [1, 2, 3]
bar = *foo
puts bar # [1, 2, 3]

5. Convert an Array Into a Hash

Lastly, splat can convert an Array into a Hash. Your Array needs to contain an even number of elements for this to work. If the Array were grouped into pairs, the first element in each pair will become a key in your Hash, and the second element of the pair will be the corresponding value.

arr = [:foo, 1, :bar, 2, :baz, 3]
Hash[*arr]
# { foo: 1, bar: 2, baz: 3 }

Rails System Tests In Docker

Nate Vick — Tue, 28 Apr 2020 17:04:35 +0000

Originally posted on Hint's blog.

At Hint, we use Docker extensively. It is our development environment for all of our projects. On a recent greenfield project, we wanted to use Rails System Tests to validate the system from end to end.

In order to comfortably use System Tests inside of Docker we had to ask ourselves a few questions:

How do we use RSpec for System Tests?
How do we run the tests in headless mode in a modern browser?
Can we run the test in a non-headless browser for building and debugging efficiently?

Context

In the context of answering these questions, we are going to first need to profile the Rails project to which these answers apply. Our Rails app:

Uses Docker and Docker Compose.
Uses RSpec for general testing.
Has an entrypoint or startup script.

If some or none of the above apply to your app and you would like to learn more about our approach to Docker, take a look at this post: Dockerizing a Rails Project.

Prep Work

If your project was started before Rails 5.1, some codebase preparation is necessary. Beginning in Rails 5.1, the Rails core team integrated Capybara meaning Rails now properly handles all the painful parts of full system tests, e.g. database rollback. This means tools like database_cleaner are no longer needed. If it is present, remove database_cleaner from your Gemfile and remove any related config (typically found in spec/support/database_cleaner.rb , spec/spec_helper.rb , or spec/rails_helper.rb).

Dependency Installation

After that codebase prep has been completed, verify that RSpec ≥ 3.7 and the required system tests helper gems are installed.

group :development, :test do
  gem 'rspec-rails', '>= 3.7'
end

group :test do
  gem 'capybara', '>= 2.15'
  gem 'selenium-webdriver'
  # gem 'chromedriver-helper' don't leave this cruft in your Gemfile.:D
end

These are the default system test helper gems installed with rails new after Rails 5.1. We will not need chromedriver-helper since we will be using a separate container for headless Chrome with chromedriver.

Note: The configuration below has been tested on Mac and Linux, but not Windows.

Docker

Speaking of containers, let's add that service to the docker-compose.yml configuration file.

services:
  # other services...
  selenium:
    image: selenium/standalone-chrome

We have added the selenium service, which pulls down the latest selenium/standalone-chrome image. You may notice I have not mapped a port to the host. This service is for inter-container communication, so there is no reason to map a port. We will need to add an environment variable to the service (app in this case) that Rails/RSpec will be running on for setting a portion of the Selenium URL.

services:
  app:
    build: .
    command: bundle exec rails server -p 3000 -b '0.0.0.0'
    # ... more config ...
    ports:
      - "3000:3000"
      - "43447:43447"
    # ... more config ...
    environment:
      - SELENIUM_REMOTE_HOST=selenium

Capybara

We added a port mapping for Capybara as well: 43447:43447. Now let's add the Capybara config at spec/support/capybara.rb.

# frozen_string_literal: true

RSpec.configure do |config|
  headless = ENV.fetch('HEADLESS', true) != 'false'

  config.before(:each, type: :system) do
    driven_by :rack_test
  end

  config.before :each, type: :system, js: true do
    url = if headless
            "http://#{ENV['SELENIUM_REMOTE_HOST']}:4444/wd/hub"
          else
            'http://host.docker.internal:9515'
          end

    driven_by :selenium, using: :chrome, options: {
      browser:              :remote,
      url:                  url,
      desired_capabilities: :chrome
    }

    # Find Docker IP address
    Capybara.server_host = if headless
                             `/sbin/ip route|awk '/scope/ { print $9 }'`.strip
                           else
                             '0.0.0.0'
                           end
    Capybara.server_port = '43447'
    session_server       = Capybara.current_session.server
    Capybara.app_host    = "http://#{session_server.host}:#{session_server.port}"
  end

  config.after :each, type: :system, js: true do
    page.driver.browser.manage.logs.get(:browser).each do |log|
      case log.message
        when /This page includes a password or credit card input in a non-secure context/
          # Ignore this warning in tests
          next
        else
          message = "[#{log.level}] #{log.message}"
          raise message
      end
    end
  end
end

Let's break it down section by section.

headless = ENV.fetch('HEADLESS', true) != 'false'

We are using the headless variable to make some decisions later in the file. This variable allows us to run bundle exec rspec normally and run system tests against headless Chrome in the selenium container. Or we run HEADLESS=false bundle exec rspec and when a system test will attempt to connect to chromedriver running on the host machine.

config.before :each, type: :system do
  driven_by :rack_test
end

Our default driver for system tests will be rack_test. It is the fastest driver available because it does not involve starting up a browser. It also means we cannot test JavaScript while using it, which brings us to the next section.

config.before :each, type: :system, js: true do
  url = if headless
          "http://#{ENV['SELENIUM_REMOTE_HOST']}:4444/wd/hub"
        else
          'http://host.docker.internal:9515'
        end

  driven_by :selenium, using: :chrome, options: {
    browser:              :remote,
    url:                  url,
    desired_capabilities: :chrome
  }

  # ...more config...       
end

Any specs with js: true set will use this config. We set the url for Selenium to use depending on if we are running headless or not. Notice the special Docker domain we are setting the non-headless url to; it is a URL that points to the host machine. The special domain is currently only available on Mac and Windows, so we will need to handle that for Linux later.

We set our driver to :selenium with config options for browser, url, desired_capabilities .

config.before :each, type: :system, js: true do

  # ...selenium config...

  Capybara.server_host = if headless
                           `/sbin/ip route|awk '/scope/ { print $9 }'`.strip
                         else
                           '0.0.0.0'
                         end
  Capybara.server_port = '43447'
  session_server       = Capybara.current_session.server
  Capybara.app_host    = "http://#{session_server.host}:#{session_server.port}"
end

Here we set the Capybara.server_host address to the app container IP address if headless or 0.0.0.0 if not.

The last part of RSpec configuration is to require this config in spec/rails_helper.rb .

# frozen_string_literal: true

# This file is copied to spec/ when you run 'rails generate rspec:install'
require 'spec_helper'
ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
# Prevent database truncation if the environment is production
abort("The Rails environment is running in production mode!") if Rails.env.production?
require 'rspec/rails'
require 'support/capybara'

Next, we need to install ChromeDriver on the host machine. You will need to place it in a location in your $PATH. Once it is there, when you want to run non-headless system tests, you will need to start ChromeDriver chromedriver --whitelisted-ips in a new terminal session. Now on a Mac, you should be able to run headless or non-headless system tests. Those commands again are:

#HEADLESS
bundle exec rspec

#NON-HEADLESS
HEADLESS=false bundle exec rspec

Special Linux Config

There is one last step for Linux users because of the special host.docker.internal URL is not available. We need to add some config to the entrypoint or startup script to solve that issue.

: ${HOST_DOMAIN:="host.docker.internal"}
function check_host { ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1; }

# check if the docker host is running on mac or windows
if ! check_host; then
  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
  echo "$HOST_IP $HOST_DOMAIN" >> /etc/hosts
fi

We set an environment variable to the special Docker URL. We then create a function to check if the host responds to that URL. If it responds, we move on assuming we are running on Mac or Windows. If it does not respond, we assign the container's IP to an environment variable, then append a record to /etc/hosts. We are now all set to run system tests on Linux as well.

Bonus: CI Setup

Let's wrap this up with config to run system tests on Circle CI. We need to add SELENIUM_REMOTE_HOST and the Selenium Docker image to .circleci/config.yml

version: 2
jobs:
  build:
    parallelism: 1
    docker:
      - image: circleci/ruby:2.6.0-node
        environment:
          SELENIUM_REMOTE_HOST: localhost
      - image: selenium/standalone-chrome

Connect with me on Twitter(@natron99) to continue the conversation about Rails and Docker!

Macro See, Macro Do

Nate Shoemaker — Fri, 17 Apr 2020 14:38:51 +0000

Originally posted on Hint's blog.

This post assumes you have Erlang/Elixir installed and that you can spin up a Phoenix project.

I've been diving into Elixir and Phoenix as of late. I have thoroughly enjoyed my time spent with both and could gush about them for far, far too long. Elixir has some features whose underlying implementation wasn't obvious at first glance, namely the macro quote/2 (that slash and number indicates the arity).

I first came across quote/2 in a fresh Phoenix project. Let's create one now! Run mix phx.new foo, open up lib/foo_web/foo_web.ex and see quote/2 being used:

# lib/foo_web/foo_web.ex

defmodule FooWeb do
  def controller do
    quote do
      use Phoenix.Controller, namespace: FooWeb

      import Plug.Conn
      import FooWeb.Gettext
      alias FooWeb.Router.Helpers, as: Routes
    end
  end

  # ...
end

controller/0 is then used in controllers like this:

# lib/foo_web/controllers/page_controller.ex

defmodule FooWeb.PageController do
  use FooWeb, :controller

  # ...
end

Going through Programming Phoenix I saw this macro being used again and again. I understood what it meant: the use, import, and alias macros are being injected into PageController, so dependencies can be shared across modules. But why not just include them in the function definition? What is going behind the scenes? Why quote/2? Being a Rails developer accustomed to magic, I accepted it and moved on.

One of Phoenix's (and Elixir) strengths is that nothing is hidden from the developer. Everything is gloriously defined, displayed, and explicitly composed right in front of you. There really isn't any magic. Thus my acceptance of it bothered me, so let's dive in and learn about quote together!

Copy and (almost) paste

The best way to learn is by doing, so why don't we create some modules and reproduce what we've seen. Here's a very simple example I came up with:

# bar.exs

defmodule Bar.Math do
  def sum(x, y), do: x + y
end

defmodule Bar.AllTheThings do
  def things do
    quote do
      alias Bar.Math
    end
  end
end

defmodule Bar.Work do
  use Bar.AllTheThings, :things

  def print_sum(x, y) do
    IO.puts("the sum of #{x} and #{y} is #{sum(x, y)}")
  end
end

Bar.Work.print_sum(2, 2)

Again, I don't really know what quote/2 is doing and why, but we mimicked what we saw in Phoenix pretty close. I think we're ready to try this out, let's run elixir bar.exs and see what happens:

> elixir bar.exs
** (UndefinedFunctionError) function Bar.AllTheThings.__using__/1 is undefined or private
    Bar.AllTheThings.__using__(:things)
    bar.exs:18: (module)
    bar.exs:17: (file)

🤔

It seems that we're missing a function that Elixir assumes we have implemented. I'll be honest - I've never written a module that was consumed by use, so let's double back to FooWeb in our Phoenix App to see if we missed anything. At the bottom of the file, you'll see:

  defmacro __using__(which) when is_atom(which) do
    apply(__MODULE__, which, [])
  end

Ah! Elixir was looking for that function, so let's slap that it in Bar.AllTheThings:

defmodule Bar.AllTheThings do
  def things do
    quote do
      alias Bar.Math
    end
  end

  defmacro __using__(which) when is_atom(which) do
    apply(__MODULE__, which, [])
  end
end

Diving into defmacro is outside the scope of this post, but we can acknowledge it as a requirement of a module that's consumed by use. The use of apply/3 is straightforward: take a module, an atom that represents the function name, and call it with some arguments.

apply(Bar.AllTheThings, :things, [])
# is equivalent to
Bar.AllTheThings.things([])

And then:

> elixir bar.exs
the sum of 2 and 2 is 4

Great, our dependency injection works. Now that we understand the structure let's dig into what's happening under the hood.

I heard you like Elixir, so let's represent some Elixir with Elixir

From the docs:

> quote(opts, block)

Gets the representation of any expression.

Let's try it out:

> iex
iex(1)> quote do sum(2, 2) end
{:sum, [], [2, 2]}

That's right! We are representing Elixir with Elixir. Elixir's AST (abstract syntax tree) is... Elixir! Pretty cool, huh? Macros, such as quote/2, are represented by a tuple of three elements. The first element is (usually) an atom, the second is for metadata, and the third is the argument list.

I wonder what our import Bar.Math looks like as an AST? Let's find out! Comment out everything in bar.exs except for the Bar.Math module. Rename the file to bar.ex so Elixir can compile it, and run iex:

> iex
iex(1)> c "bar.ex"
[Bar.Math]
iex(2)> quote do
...(2)>   import Bar.Math
...(2)> end
{:import, [context: Elixir], [{:__aliases__, [alias: false], [:Bar, :Math]}]}

There it is! We can see the AST as a three element tuple. It holds all the information that Elixir needs to know to import a module. quote/2 gives us some fantastic syntax sugar; could you imagine writing these tuples everywhere? Just for fun, let's see how deep into the rabbit hole we can go. Rename bar.ex back to bar.exs, uncomment all the code, and change the import Bar.Math to the AST representation without quote/2:

# bar.exs

# ...
defmodule Bar.AllTheThings do
  def things do
    {:import, [context: Elixir], [{:__aliases__, [alias: false], [:Bar, :Math]}]}
  end

  defmacro __using__(which) when is_atom(which) do
    apply(__MODULE__, which, [])
  end
end
# ...

And:

> elixir bar.exs
the sum of 2 and 2 is 4

It works! Let's go another level in by removing things/0 and placing our AST directly in __using/1:

# bar.exs

# ...
defmodule Bar.AllTheThings do
  defmacro __using__(which) when is_atom(which) do
    {:import, [context: Elixir], [{:__aliases__, [alias: false], [:Bar, :Math]}]}
  end
end
# ...

You know the drill:

> elixir bar.exs
the sum of 2 and 2 is 4

Nice! Is it possible to inline the AST we have in our Bar.Work module? Sadly, we can't. The use/2 macro changes this:

use Bar.AllTheThings, :things

to:

Bar.AllTheThings.__using__(:things)

We've come to the end of this Elixir in Elixir train! There are no other stops on this line.

Wrapping up

So, what did we learn? The quote macro transforms Elixir code to an AST. We can then leverage that to achieve real dependency injection in a functional language. How cool is that?

Strong Opinions About Strong Parameters

Benjamin Wood — Fri, 10 Apr 2020 17:11:47 +0000

Rails announced strong_parameters as a replacement for protected_attributes nearly eight years ago. In 2020 many Rails apps have not completed the migration. Others have made the migration, but are worse off than before. Was strong_parameters a bad idea? I don't think so. But like many things, it depends on how you use it.

Instances of `ActionController:Parameters` leak outside of the controller

This is undoubtedly the most common mistake I've seen. The reason that strong_parameters is superior to protected_attributes is that it narrows the concern of mass-assignment protection to the controller. With protected_attributes, the entire app had to consider which attributes could mass-assigned, and which could not. This led to anti-patterns like the use of without_protection which allowed developers to skip mass-assignment protection under certain circumstances.

The benefit of strong_parameters is lost when ActionController::Parameters are passed outside of a controller and suddenly you have to be concerned about mass-assignment protection everywhere, again.

  # FILE: app/lib/payment_processor.rb
  class PaymentProcessor
    def initialize(params)
      @params = params
    end

    def process
      # ... payment processing logic, etc
      Payment.create(payment_params)
    end

    private

    def payment_params
      @params.require(:payment).permit(:some, :payment, :attrs)
    end
  end

  # FILE: app/controllers/payments_controller.rb
  class PaymentsController < ApplicationController
    def create
      payment = PaymentProcessor.new(params)

      if payment.process
        ...
      else
        ...
      end
    end
  end

Strong Opinion: Parameters should be permitted at the controller level. If you pass some params on to another object (like a service class), first permit the params that are allowed for mass-assignment, then call to_h on it to convert the object to a good ol' ActiveSupport::HashWithIndifferentAccess.

  # FILE: app/lib/payment_processor.rb
  class PaymentProcessor
    def initialize(payment_params)
      @payment_params = payment_params
    end

    def process
      # ... payment processing logic, etc
      Payment.create(@payment_params)
    end
  end

  # FILE: app/controllers/payments_controller.rb
  class PaymentsController < ApplicationController
    def create
      payment = PaymentProcessor.new(payment_params.to_h)

      if payment.process
        ...
      else
        ...
      end
    end

    private

    def payment_params
      params.require(:payment).permit(:some, :payment, :attrs)
    end
  end

Use of strong parameters where mass assignment is not being performed

The Rails community is pretty split on this one. Should you always permit parameters? Or only permit parameters when mass-assignment is being performed? Here's an example:

  # FILE: app/controllers/users_controller.rb
  class UsersController < ApplicationController
    def special_update_action
      user = User.find(user_params[:id])

      user.special_attribute = user_params[:special_attribute]

      if user.save
        ...
      else
        ...
      end
    end

    private

    def user_params
      params.require(:user).permit(
        :id,
        :special_attribute,
        :some,
        :user,
        :attrs,
        :including
      )
    end
  end

Strong Opinion: Don't complicate your controllers by permitting params when mass assignment is not being performed.

  # FILE: app/controllers/users_controller.rb
  class UsersController < ApplicationController
    def special_update_action
      user = User.find(params[:id])

      # Notice that special_attribute is pulled off of `params` directly
      user.special_attribute = params[:special_attribute]

      if user.save
        ...
      else
        ...
      end
    end
  end

Order of operations when mutating a params object

Remember, strong_parameters is meant to sanitize externally-provided parameters. If you add a key to the params object, then call permit, you'll end up having to permit the parameter you just added.

  # FILE: app/controllers/credit_cards_controller.rb
  class CreditCardsController < ApplicationController
    def update
      credit_card = CreditCard.find(params[:id])

      params[:top_secret_token] = TopSecretToken.new(args)

      if credit_card.update(credit_card_params)
        ...
      else
        ...
      end
    end

    private

    def credit_card_params
      # Notice the addition of top_secret_token is included in this
      # scenario so that it can be mass-assigned above
      params.require(:credit_card).permit(
        :top_secret_token,
        :some,
        :credit_card,
        :attrs
      )
    end
  end

Strong Opinion: Permit your params before mutating the object. This allows you to permit only the externally-provided parameters, then modify the object as you see fit.

  # FILE: app/controllers/credit_cards_controller.rb
  class CreditCardsController < ApplicationController
    def update
      credit_card = CreditCard.find(params[:id])

      assignable_params = credit_card_params
      assignable_params[:top_secret_token] = TopSecretToken.new(args)

      if credit_card.update(assignable_params)
        ...
      else
        ...
      end
    end

    private

    def credit_card_params
      # Notice that top_secret_token does *not* need to be included
      # here because it is assigned to the (permitted) return value
      # of credit_card_params
      params.require(:credit_card).permit(:some, :credit_card, :attrs)
    end
  end

Defining permitted attributes multiple (sometimes many) times

With protected_attributes, the model provided a central place to specify which attributes were permitted for mass-assignment. With strong_parameters, permitted parameters are often defined multiple times for the same resource in different controllers. This is not DRY and it is not maintainable. Add in consideration for accepts_nested_attributes_for and you're maintaining countless lists of the same parameters.

Note: In the previous examples parameters were permitted directly in the controller as seen below. This was done for simplicity's sake, and also on the basis that no other controllers duplicated the same set of permitted parameters.

  # FILE: app/controllers/departments_controller.rb
  class DepartmentsController < ApplicationController
    def create
      department = Department.new(department_params)

      if department.save
        ...
      else
        ...
      end
    end

    private

    def department_params
      params.require(:department).permit(
        :some, :department, :attrs,
        employees_attributes: [
          :some, :employee, :attrs
        ]
      )
    end
  end

  # FILE: app/controllers/special/departments_controller.rb
  module Special
    class DepartmentsController < ApplicationController
      def create
        department = Department.new(department_params)

        if department.save
          ...
        else
          ...
        end
      end

      private

      # Copy pasta
      def department_params
        params.require(:department).permit(
          :some, :department, :attrs,
          employees_attributes: [
            :some, :employee, :attrs
          ]
        )
      end
    end
  end

Strong Opinion: Duplicated parameters should be permitted in a module that can be included wherever it is needed. Furthermore, nested attributes should not be redefined multiple times. Here's an example:


  # FILE: app/controllers/concerns/strong_parameters/employee.rb
  module Concerns
    module StrongParameters
      module Employee
        def employee_params
          params.require(:employee).permit(*self.permitted_attrs)
        end

        def self.permitted_attrs
          %i(some employee attrs)
        end
      end
    end
  end

  # FILE: app/controllers/concerns/strong_parameters/department.rb
  module Concerns
    module StrongParameters
      module Department
        def department_params
          params.require(:department).permit(*self.permitted_attrs)
        end

        def self.permitted_attrs
          [
            :some, :department, :attrs,
            { employees_attributes: [
              :id, :_destroy, *Concerns::StrongParameters::Employee.permitted_attrs] }
          ]
        end
      end
    end
  end

  # FILE: app/controllers/departments_controller.rb
  class DepartmentsController < ApplicationController
    include Concerns::StrongParameters::Department

    def create
      department = Department.new(department_params)

      if department.save
        ...
      else
        ...
      end
    end
  end

  # FILE: app/controllers/special/departments_controller.rb
  module Special
    class DepartmentsController < ApplicationController
      include Concerns::StrongParameters::Department

      def create
        department = Department.new(department_params)

        if department.save
          ...
        else
          ...
        end
      end
    end
  end

  # FILE: app/controllers/employees_controller.rb
  class EmployeesController < ApplicationController
    include Concerns::StrongParameters::Employee

    def create
      employee = employee.new(employee_params)

      if employee.save
        ...
      else
        ...
      end
    end
  end

Summary

We follow these practices at Hint, and it's helped us (and our clients) a lot. Does your organization need help wrangling strong parameters? We can help. Ping me on twitter: @benjaminwood .

Originally published on the hint.io blog here: https://hint.io/blog/strong-parameters-strong-opinions

DEV Community: Hint

Deploy Infrastructure With CDKTF

Prereqs and Install

Create and Initialize the example project

Install Dependencies

Let's Write Some TypeScript

Create a VPC

Create a Postgres Database

Create a Droplet

Let's Put It All Together

Things To Know

Wrapping Up

Active Record Encryption in Rails 7

Backstory

Deterministic Sidebar

Setup

Keys

App Configuration

Database

Invocation

Writing

Reading

Searching

Initialization Vector/Determinism

Searching Plaintext

Case Insensitive Searches

Other Cool things

Limitations

Summary

WSL for Rails developers

Installation

Terminals

VSCode

Docker

Filesystem

Copy-and-paste

Caveats

Conclusion

What's Cooking in Rails 7?

Railties

The main Branch

PR 40254 by @prateekkish

benchmark Anywhere

PR 40734 by @semaperepelitsa

Support Stats For Stylesheets and ERB Views

PR 40597 by @joelhawksley

ActionPack

Redirect Back or To

PR 40671 by @dhh

Unpermitted Parameter Context

PR 41809 by @bbuchalter

ActionView

Lazy Load Images

PR 38452 by @jonathanhefner

Tag Attributes From Hash

PR 40657 by @seanpdoyle

ActiveModel

Add Numericality In Range Validation

PR 41022 by @mpapis

ActiveRecord

Invert a where Clause

PR 40249 by @kddeisz

Exclude a Record From Results

PR 41439 by @GlenCrawford

Build or Create Association on Has One Through

PR 40007 by @perezperret

Respect Column Type When Calculating Average

PR 40351 by @schmijos

Automatically Encrypt/Decrypt a Model Attribute

PR 41659 by @jorgemanrubia

What Else?

Ruby on Lambda: Integrating with ActiveStorage

Uploading

Variant Creation

Data for the active_storage_attachments record

All together now

Ruby on AWS Lambda: Package & Ship It

Function naming conventions and environment variables

Instrumentation

Serverless

Data for the `active_storage_attachments` record

Instances of `ActionController:Parameters` leak outside of the controller