CloudFormation vs Terraform

Why would someone choose to use a third-party tool when AWS provides a service that offers similar functionality?
While it could be argued that there are now fewer reasons to choose a third-party tool over AWS services, considering the significant improvements in CloudFormation in recent years, if I were to make the choice today, I would still prefer to use Terraform. Here are a few reasons to support this decision.

Language

Terraform utilizes HCL (HashiCorp Configuration Language), which is designed to be both human-readable and machine-friendly. On the other hand, CloudFormation relies on either JSON or YAML. In terms of readability and authoring, YAML generally surpasses JSON, but it still requires careful handling of nested scopes and can cause issues if indentation is incorrect. In contrast, HCL typically involves fewer scopes (enclosed in curly braces) and enforces formatting conventions inspired by Go, resulting in a more visually appealing code.
Terraform offers a comprehensive range of string interpolations and built-in functions, including conditionals and soon-to-be-supported loops. This enables the modeling of complex logic within the DSL itself, without the need for a full-fledged programming language (although external data sources can be integrated seamlessly). On the other hand, CloudFormation’s intrinsic functions are noticeably limited in comparison.
Terraform has an advantage in terms of code reusability through modules and provides flexibility in structuring projects according to individual preferences. While CloudFormation supports nested stacks, it generally requires all infrastructure code to be contained within a single large file or multiple files within a single project repository. In contrast, Terraform allows for granular slicing and organizing of code files, making it more manageable for developers. Modules can be stored on GitHub or the public Terraform Module Registry, allowing easy versioning and sharing across multiple projects.

State

Terraform stores its state in a file, which can be stored on disk, committed to source control (though not recommended), or kept in systems like S3 or a configuration management tool. Regardless of the chosen storage method, it is crucial to ensure the integrity and availability of the state file. Accidentally using the production environment’s state while deploying to a testing environment or losing/corrupting the state file can be problematic. Although the introduction of workspaces (previously known as “environments”) has made it less prone to mistakes, working with Terraform state in the early stages did result in a few challenges.
In contrast, CloudFormation operates on AWS infrastructure and handles the state management for you, relieving you of the need to worry about its underlying mechanisms. Additionally, CloudFormation attempts to roll back changes if they cannot be successfully applied (unless it gets stuck in the UPDATE_ROLLBACK_FAILED state). Furthermore, CloudFormation’s backend can receive signals from your resources, allowing you to configure rolling updates for an Auto Scaling group, which is highly beneficial. Unlike Terraform, CloudFormation is not affected if the machine triggering the deployment unexpectedly goes down (as can happen with interchangeable CI/CD servers), while Terraform would need to recover from a partial update.

Lets consider a scenario

• Terraform provides the capability to manage configuration drift. By running terraform plan, it compares the desired state defined in configuration files with the actual state of the infrastructure fetched during the execution. If any inconsistencies are detected (usually caused by external changes), Terraform computes the difference and suggests reverting the manual changes to align with the desired state. Performing periodic Terraform plans in idempotent deployments allows for easy detection and reversal of configuration drift, making it a routine part of infrastructure provisioning pipelines.
• Meanwhile, CloudFormation is unaware of any changes made outside its state.
• To conduct a quick experiment, begin by deploying a new CloudFormation stack to create a fresh VPC and a security group devoid of any ingress rules.

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  VPC:
    Type: "AWS::EC2::VPC"
    Properties:
      CidrBlock: 10.0.0.0/16

  SecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupName: TestSG
      GroupDescription: Test security group
      VpcId: !Ref VPC

• Then, go to the web console and manually add a new ingress rule. For example, let’s open our SSH port to the world to do some real quick debugging. Let's try out .

• When attempting to update your CloudFormation stack, you would encounter an error indicating the following: Template validation error: No updates are to be performed.

Configuration

Parameters in CloudFormation enable the reuse of templates across different environments. For instance, you can specify cheaper EC2 instances for the test environment and more powerful ones for production.
CloudFormation supports up to 60 parameters that need to be provided during runtime. You have several options to accomplish this. The simplest approach is to pass parameters individually and set default values that make sense. Alternatively, you can supply parameters in bulk from a file (ensuring the correct file is passed), import values from another stack’s outputs, or retrieve them from Parameter Store. Additionally, you can utilize a Mapping section to look up parameters using a specific key, such as environment.

Whereas for terraform

Terraform offers greater flexibility compared to CloudFormation. It introduces the concept of data sources, allowing read-only access to resource properties without impacting their state. This enables easy integration of related stacks and avoids hard-coding parameters. With data sources, you can access externally managed resource properties (e.g., finding the latest AMI), consume outputs from CloudFormation stacks, reuse attributes from other Terraform stacks (such as database endpoints or DNS records), and even run custom scripts to provide required values. The current infrastructure state serves as the single source of truth, ensuring changes propagate seamlessly to dependent resources during the next Terraform run, avoiding the risk of forgetting duplicate values. This approach maintains clear separation of concerns. Additionally, using the same CI/CD pipeline for all Terraform projects becomes feasible, eliminating the need to manage multiple argument sets for each resource. Being cloud-agnostic, Terraform allows composition of infrastructure from different cloud providers and third-party services like PagerDuty or New Relic. In summary, Terraform simplifies infrastructure management as code, eliminating the need to learn separate tools and syntax for different platforms and services.

Conclusion

Does this imply that Terraform is always superior to CloudFormation? Not necessarily. There are specific use cases where employing CloudFormation is more appropriate and advantageous.
In certain scenarios, CloudFormation can be a preferable choice over Terraform, such as when managing the S3 bucket used for storing Terraform state files. Here are some key points to consider:

1. Manual Creation: Manually creating the S3 bucket for storing Terraform state files is not recommended. It would make the bucket difficult to manage and audit effectively.
2. Chicken-and-Egg Problem: Provisioning the S3 bucket in Terraform initially presents a challenge. Where do you store the state file for the bucket used to store the state files? This creates a circular dependency dilemma.
3. Source Control: One option is to commit the state file to a source control system. While this may work if changes to the resource are infrequent, it is generally not the optimal approach.
4. CloudFormation Solution: CloudFormation can be an appealing alternative to address this challenge. By using CloudFormation, you eliminate the need to manage the state file separately, as CloudFormation handles it automatically.
5. Terraform Migration: Another approach is to create the S3 bucket in Terraform initially using local state. Subsequently, you can add the S3 backend configuration pointing to the same bucket. After reinitializing your Terraform project with ‘terraform init’, Terraform will migrate the state file to the S3 bucket.

Note

• By considering these points, you can decide between CloudFormation and Terraform for managing the S3 bucket used for storing Terraform state files.