Terraform Modules: Atomic Design

Intro

Following The Pragmatic Programmer mantra, I do my best to ...

Learn at least one new language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut.

Not necessarily to show it off or to be capable of talking about random technologies, but to expand and train my problem-solving skills, to get new perspectives when approaching a challenge.

We might not notice it but when we learn (or have learned) to code we aren't just learning to type some characters that a compiler/interpreter can understand, it is a new way of thinking, a new way of breaking down solutions (into sequential steps).

It doesn't matter whether you ever use any of these technologies on a project, or even whether you put them on your resume. The process of learning will expand your thinking, opening you to new possibilities and new ways of doing things.
The cross-pollination of ideas is important;

As someone who works intensively with infrastructure components (servers, databases, Kubernetes, CI/CD, etc) I aimed for something completely different this year. Something that stands on a whole different spectrum of the system, this year I decided to learn Flutter.

In-a-nutshell, Flutter is a better React Native. A framework that enables implementation of GUI applications for multiple platforms with a single code base.

Then it reminded me a discussion I had with a friend in the past about React components and the Atomic Design methodology, which helps to structure web components into modules.

In the Atomic Design methodology, the granularity of modules is distinguished by using chemistry inspired names: atoms, molecules and organisms.

Then the connection of the ideas from

• Pragmatic Programmer's cross-pollination to
• Atomic Design (on Flutter components) to
• Terraform modules

came almost like a thunderbolt, striking me with this insight when I was working with a huge legacy Terraform code base refactoring with lots of code duplication (read: copy+paste, "we fix it later", then the author quits the company and
never fix anything).

Although initially proposed as a Web UI methodology, Infrastructure as Code tools such as Terraform that makes heavy usage of modules can benefit from Atomic Design to improve its code reusability and massively reduce duplication.

Details

The Atomic Design methodology proposes five distinct levels, listed from the finest to the thickest:

1. Atom;
2. Molecules;
3. Organisms;
4. Templates;
5. Pages.

However, to extract the gist, we'll only be focusing on Atoms, Molecules, and Organisms (from 1. to 3.). Templates and Pages are too domain-specific focused on Web UI development.

Atoms

Atoms represent the finest grain in terms of granularity in the design. When referring specifically to its implementation in Terraform a resource and a small scoped single-purpose module could be used interchangeably.

Sometimes the idea of turning a simple resource into a module makes sense to ease parameterization and reusability, especially when it is necessary to parse inputs. Although, due to its extreme limited scope it might not look attractive
to convert the resource into a module at first sight, on the long run it pays off to do so in order to achieve scalability and reproducibility.

e.g.:

data "aws_route53_zone" "default" {
  zone_id = var.zone_id
  name    = var.zone_name
}

resource "aws_route53_record" "default" {
  zone_id = data.aws_route53_zone.default.zone_id
  name    = var.name

  ttl  = var.ttl
  type = var.record_type

  records = var.records

  dynamic "alias" {
    for_each = [var.alias]

    content {
      name = each.value.name
      zone_id = try(each.value.zone_id, data.aws_route53_zone.default.zone_id)

      evaluate_target_health = lookup(
        each.value,
        "evaluate_target_health",
        false,
      )
    }
  }
}

In this case, even though aws_route53_record is a simple resource that might feel too narrow in scope to write a module, the implementation of the module allows to bundle the AWS Route53 Zone data source together, which helps to:

1. provide a simpler contract by allowing the usage of zone_name alone;
2. validate the zone_name input, ensuring that a given zone_name corresponds to an actual existing and valid AWS resource;
3. same goes to zone_id, which will feel (and oftentimes, be) redundant, when specified as an input Terraform will read the data from AWS API ensuring consistency.

e.g.:

module "awesome_dns_fqdn" {
  source = "path/to/modules/atoms/aws_route53_record"
  version = "~> 1.0"

  name      = "record.example.com"
  zone_name = "example.com."

  record_type = "A"
  records     = ["1.2.3.4"]
}

Hence, resources and modules are sometimes interchangeable as they deliver the same outcome for the finest resources' granularity.

Molecules

When groups of atoms are bounded together, they create a molecule which is the smallest fundamental unit of a compound.

Contrary to the original Atomic Design for Web UI, in Terraform, Atoms are useful on their own. However, the usage of atoms comes with a high price on scalability: code duplication. Actually, duplication is an understatement, it is more like code exponentiation (more on this later).

Implementation example

Suppose we are creating a public facing API Gateway that needs a DNS record.

Let's compose it with the previous example:

data "aws_route53_zone" "default" {
  name = var.zone_name
}

module "awesome_api_gateway_certificate" {
  source  = "terraform-aws-modules/acm/aws"
  version = "~> v3.0"

  domain_name = var.domain_name
  zone_id     = data.aws_route53_zone.default.zone_id

  wait_for_validation = true
}

module "awesome_api_gateway" {
  source = "terraform-aws-modules/apigateway-v2/aws"
  version = "~> 1.0"

  name          = var.api_gateway_name
  description   = var.api_gateway_description
  protocol_type = "HTTP"

  cors_configuration = {
    allow_headers = [
      "content-type",
      "x-amz-date",
      "authorization",
      "x-api-key",
      "x-amz-security-token",
      "x-amz-user-agent",
    ]
    allow_methods = ["*"]
    allow_origins = ["*"]
  }

  # Custom domain
  domain_name                 = var.domain_name
  domain_name_certificate_arn = module.awesome_api_gateway_certificate.acm_certificate_arn

  # Routes and integrations
  integrations = var.api_gateway_integrations
}

module "awesome_dns_fqdn" {
  source  = "path/to/modules/atoms/aws_route53_record"
  version = "~> 1.0"

  name    = var.domain_name
  zone_id = data.aws_route53_zone.default.zone_id

  record_type = "CNAME"
  alias     = {
    name    = module.awesome_api_gateway.apigatewayv2_domain_name_configuration[0].target_domain_name
    zone_id = module.awesome_api_gateway.apigatewayv2_domain_name_configuration[0].hosted_zone_id
  }
}

This helps illustrating an example in which the aws_route53_record atom could be easily replaced with its equivalent resource and it would still provide the same outcome.

Commonly it is possible to use module and resource interchangeably as Atoms, the decision of whether or not to implement a module is ultimately defined by the need of parsing and/or validating the inputs (variables).

Usage example

module "awesome_lambda" {
  source  = "path/to/modules/molecules/aws_lambda_function"
  version = "~> 1.0"

  function_name = "awesome"
  description   = "An Awesome lambda function for the Awesome API Gateway"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  # Incomplete implementation, don't use this on production
}

module "another_awesome_lambda" {
  source  = "path/to/modules/molecules/aws_lambda_function"
  version = "~> 1.0"

  function_name = "awesome"
  description   = "An Awesome lambda function for the Awesome API Gateway"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  # Incomplete implementation, don't use this on production
}

module "awesome_api_gateway" {
  source = "path/to/modules/molecules/aws_api_gateway"
  version = "~> 1.0"

  domain_name = "record.example.com"
  zone_name   = "example.com."

  api_gateway_name = "awesome-api-gateway"
  api_gateway_description = "An Awesome API Gateway"

  api_gateway_integrations = {
    "POST /" = {
      lambda_arn             = module.awesome_lambda.function_arn
      payload_format_version = "2.0"
    }

    "$default" = {
      lambda_arn = module.another_awesome_lambda.function_arn
    }
  }
}

As you probably have already realized, when the level of abstraction goes up (e.g. from atom to molecule) the module implementation is in itself a good implementation example (i.e. as in community modules examples).

They help to self-document the usage and implementation of a given module and through generic implementations it allows us to have multiple molecules implementing multiple distinct use-cases. e.g.:

1. Public API Gateway with DNS record + TLS certificate;
2. Public API Gateway v1, no DNS record;
3. Private API Gateway.

Why would we chose to implement multiple times the Atom modules in order to create multiple distinct use-cases? We are getting closer to the code exponentiation problem and solution proposal. Can you feel it?

Organisms

Going further, the example of composition for molecules can have its hard-coded values turned into variables in order to compose an Organism, which can facilitate the implementation of the same definition across different environments. Thus, achieving reproducibility as well as the Factor X. of the Twelve Factor App.

However, it is important to note that the level of abstraction between Organisms and Molecules can be easily confused or misunderstood. Generally speaking, as a
rule of thumb an Organism is the composition of Molecules that allow parameterization for business or domain-specific logic (e.g. the actual awesome_api configuration).
Therefore, in comparison with the previous, Organisms (usually) have a lower level of generalization since they are business-specialized modules.

Iterating over our implementation example, the Organism would implement the awesome_api, creating the following resources:

• AWS Lambda function;
• AWS API Gateway;
• TLS Certificate on AWS ACM;
• DNS record on AWS Route53.

By implementing the previous examples as organisms we:

1. reduce the amount of boilerplate code;
2. foster reusability of modules;
3. provide a simple interface for non-operators to manage TF code.

When you sum it all up, you will notice that it is all about autonomy and "DevOps" through encouragement of self-service Ops. One wouldn't need to know a lot about Terraform to grab a module and pass some parameters to it, followed by
a code review process Operators and Software Developers can manage the Infrastructure in harmony, together. (:

Code Exponentiation? What?

Read that as a dramatization of the "code duplication" term.

When it comes to Infrastructure as Code, there is no easy way around the jungle of resources that grows over time. Fast pacing tech companies are "moving fast and breaking things", oftentimes the Operators are worried about a massive
amount of challenges at once: keep the servers up and running, with a consistent response time, low error rate, and all that playbook from Google's SRE wisdom.

All things considered, a good Infrastructure as Code design is generally a first-world problem. However, as the time passes it evolves into a real issue that slows down the implementation of resources as code. Either that or there
will be a huge ton of copy+paste to keep up with the pace, followed by a routine of find+replace when changes are applied, then harder to track pull requests and slower code reviews.

Lets take our awesome_api example and scale it up to multiple environments followed by a second awesome_api:

.
├── development
│   ├── an-awesome-api
│   │   └── main.tf
│   └── another-awesome-api
│       └── main.tf
├── staging
│   ├── an-awesome-api
│   │   └── main.tf
│   └── another-awesome-api
│       └── main.tf
└── production
    ├── an-awesome-api
    │   └── main.tf
    └── another-awesome-api
        └── main.tf

In order to replicate the configuration and ensure consistency, the following is way simpler to implement (and review) than copy+paste huge chunks of Terraform definitions

module "awesome_api" {
  source = "path/to/modules/organisms/aws_lambda_with_api_gateway"
  version = "~> 1.0"

  domain_name = "record.example.com"
  zone_name   = "example.com."

  lambda_functions = [
    # Index 0 -- An Awesome Lambda Function, used for POST
    {
      name        = "an-awesome"
      description = "An Awesome lambda function for the Awesome API Gateway"
      handler     = "an_awesome.lambda_handler"
      runtime     = "python3.8"
    },
    # Index 1 -- Another Awesome Lambda Function, used as $default
    {
      name        = "another-awesome"
      description = "Another Awesome lambda function for the Awesome API Gateway"
      handler     = "another_awesome.lambda_handler"
      runtime     = "python3.8"
    },
  ]

  api_gateway_name = "awesome-api-gateway"
  api_gateway_description = "An Awesome API Gateway"

  api_gateway_integrations = {
    "POST /" = {
      lambda_function_index  = 0
      payload_format_version = "2.0"
    }

    "$default" = {
      lambda_function_index = 1
    }
  }
}

Conclusion

At the end of the day we get an ugly Terraform state containing many

module.something.module.something_else.module.yet_another_thing...

But the productivity boost gained by merging modules based on context is a worth investment. Especially for huge Terraform repositories with multiple teams collaborating and managing a lot of resources.

Cross-team collaboration is fostered by applying the Atomic Design methodology for Terraform modules, code reusability becomes an important factor over copy+paste and the repository gravitates towards the DRY principle.

Same post, different places

Reddit r/Terraform post: Terraform Modules: Atomic Design